The leading technology for finding the main text on a web page does so at a rate of about 50-100 web pages per second. That’s simply too slow.

At first, ontolo attempted similar methodologies as exist today. But we weren’t happy with the inconsistent quality, incomplete data, and glacial speed. This is a hard problem. Very smart people have worked on this for a very long time. And yet, 50-100 URLs analyzed per second seems to be the upper limit for performance.

But we began noticing some other ideas that might work… By re-imagining the problem, we hand-crafted a new way of parsing web pages that you won’t find in public code libraries.

Today, we’re not only able to extract the main article of a web page, but we’re able to split it into more than 100 separate sections. Additionally, within those sections, we identify many different things like text, links, email addresses, ads, multimedia, and much more.

Having access to this kinda of data lets you answer very specific questions such as:

“What blog posts were written about topic X, where any of these, let’s say, 25 of my top competitors are linked to in the main post, but where our company is mentioned or linked to from the comments?”

And also…

“And also make special note of pages with ads, affiliate links, or other sponsorship opportunities (events, donations, etc) so that I can ensure when people read about my competitors, I can be right there, too.”

And one more thing…

“And one more thing. I’d like any email addresses, contact pages, advertising pages, and social media accounts found on the page. I’d like all of them so I can start building more and better relationships with folks in this industry. But I’d also like for them to be separated out so I can see which social media accounts are linked to from the article vs. which are probably for that website owner or business.”

With that level of detail in your data, the amount of detail only available from ontolo, questions like these become very easy to answer.

But how quickly? What about the speed? If it takes days to get these answers, it’s just not worth it. And even if it were to take 15 minutes or so, as is the case with most other large-scale marketing research tools, it interrupts the flow of your creativity and productivity.

But you don’t have to wait even a single minute. ontolo is able to completely anlayze web pages faster than they can be downloaded, without breaking a sweat.

If you want 100,000 URLs analyzed, right now, you will have them in under 30 seconds.

When you set out to design high-speed software, you begin to see clearly that other parts of the system aren’t quite up to the task.

Neither the hardware or the operating systems of today were designed for this kind of specialized software or speed. To process this much data, this quickly, we began running into very real, physical limitations outside of the code we designed.

Nearly half of our time spent developing ontolo has been invested in working through bottlenecks. We learned much more intimately how the operating system, processors, memory, network cards, and even the motherboard, can handle that much data, faster.

As a result, not only is our software finely tuned to the task of extracting copious amounts of data from web pages, but our technology has also been designed to work with the physical limitations of inexpensive, readily-available computer hardware.

In fact, our our software has been refined to the degree that our primary limitations at this point are the speed of our internet connection and the speed of other websites.

The common solution to large-scale technology problems of today is to “throw hardware at it.” For a number of reasons, we don’t like that idea.

At times bordering on pathological conviction, we chose to make the code right, later vs. making it sufficient, now. One machine is easier to manage than ten. Good code is easier to debug than hacked code. Elegant code is easier to improve upon than rushed code. As you tell us more of the features you want, we want to implement them quickly and well. You simply can’t do that with rushed code.

Throughout the design process, we were repeatedly advised by the entire range of folks involved in technology, that we should launch fast, throw hardware at it, and refine it later. Yet, if you talk to anyone who’s developed high-performance technology, one of the first things they’ll tell you is that you never go back and improve code that “already works.”

So we decided to double the development time to do it right. If you’re going to commit to learning a new technology for your market research, regardless of how valuable it is to you, you deserve to know that what you’re relying upon will continue to be reliable, regardless of the growth challenges faced by ontolo. By solving scalability with good code before more hardware, the entire system becomes much more reliable, further into the future.

(Folks interested in this sort of counter-intuitive design decision, feel free to reach out and ask. It’s something to which we gave a lot of consideration. It wasn’t always certain, but we feel now that we made the right decision for the standards we set for ontolo.)

From data to people, from hearts to minds, and from bodies to souls, in the end, relationships define both our experience and success.

From Machiavelli to Karla McLaren’s Language of Emotions, it’s clear that relationships are simply not a thing that can be measured, created, or cultivated by data. …

The most fundamental pieces of marketing all come down to relationships. But relationship are tough. It’s hard to find the right people, approach them in the right context, and also to understand who a person is and where they’re coming from.

ontolo will never be able to replace relationships. But it can certainly help get you the right information, faster, and help you begin the process of building new relationships, faster than ever before.