ontolo Blog

400,000,000+ Prospects a Day: Behind the Scenes of ontolo (+ Last Day for Discount)

ontolo’s crawler/parser/indexer/whathaveyou might be the fastest that I know of in the marketing research space. Last I heard, moz’s machines crawl, parse and analyze at a rate of about 25,000,000 URLs per day. ontolo can do more than 400,000,000 per day and is limited not by the hardware or code, but by the maxed out gigabit network connection. And if you’ve seen any of the demos this week, you also know that ontolo collects orders of magnitude more data other research tools. And by the end of this, you’ll understand why that is.


In the final post for the week, I’m going to walk you through how ontolo works, behind the scenes.

I’m writing this for you in plain language and a way that is easily understood, even if you don’t have a technical background. I’m also going to include technical details for folks who are interested, and those will be in sidebars with grey backgrounds.

So let’s get started.


C

Fundamentally, there are two parts to ontolo: Prospecting and Querying. Since you can’t query data that doesn’t exist, we’ll go through the ontolo prospect lifecycle, starting from a Prospecting Term or CSV Upload, all the way to a result from one of your queries.

Before I get into the process, there’s one critical piece here that’s important to understand, and that’s the programming language that ontolo is written in.

That language is: C.

Most operating systems that run the internet are written in C. And most of the popular programming languages are also written in C.

The reason that this is important is because C is fast. Very, very, very fast. It’s also dangerous. Where other programming languages have lots of things in place to keep you from Doing Bad Things, C doesn’t. C is a sawed-off shotgun, and if you don’t know how to fire it, it’s going to hurt. But if you do know how to fire it, it’ll get the job done very, very quickly.

Or, if you prefer a different kind of analogy, C is like riding a motorcycle, while other programming languages are like driving a truck. The motorcycle will be much faster, but offers far less protection in the event that Bad Things happen.

This extra protection that is offered by other programming languages comes at a cost of speed, resources (e.g. “gas”), etc.

To give a concrete example, the fastest I was ever able to get a crawler and very basic parser to run in PHP was at a rate of about 4,000,000 URLs a day. When I wrote one in Perl, I was barely able to hit 50,000,000 URLs per day analyzing far less than what ontolo now analyzes.

The reason these aren’t running at 400,000,000 URLs a day like ontolo is because they add in a lot of extra steps so that you can program things faster and have to worry about fewer things.

With C, you worry about everything. And in exchange, you can make things very, very fast.

I never thought I’d be a programmer, absolutely never a C programmer. But I trudged through the pain and agony and absurdity to learn it because if I was going to make this thing that was in my head, I was going to make in the absolute best, fastest way possible.

And, 20 months later, here we are.

Technical Note:

For those who are interested, there is one C++ library being used for the search index, and everything else except the front-end is in C. Redis is used for queuing, and the MySQL connector is used in one spot, then XXHash64 is used for another piece. Every other piece of ontolo has been hand-written, from scratch, in pure C. I think you can C that I have fallen in love.


Prospecting

When you decide you want to find a new marketing prospect, you can either upload a URL list you might have from moz, AHrefs, Majestic, etc. or you can tell ontolo to find new prospects for you. I’ll come back to the upload option when discussing the crawler.

When you tell ontolo “I want to find blogs about rock climbing,” you just need to tell ontolo you want stuff about “rock climbing” when you’re on the Blogs tool.

ontolo takes your term, “rock climbing,” and then combines it with dozens of “prospecting templates” behind the scenes. So ontolo might expand that into “rock climbing blogs,” “rock climbing comments,” “rock climbing bloggers,” etc. In total, across all of the tools, ontolo has 765 different prospecting terms it uses, and this number continues to grow on an almost-weekly basis.

Once ontolo has expanded your Prospecting Term into a full group of terms, those then go into the Prospecting Queue, one per Prospect Source. (e.g. Bing)

Technical Note:

The prospecting queue is currently a straightforward Redis instance with a queue per prospecting source. Redis announced this week the ability to build custom modules in C (this was only sort-of possible in Lua). And until that announcement, I’d begun working on my own rate-limited queuing system with some features that Redis doesn’t have. So we’ll see if I end up sticking with Redis, or continue working on a replacement.

Once a Prospecting Term is in the Prospecting Queue for a Prospect Source, there are dozens of machines – Prospectors – around the world that monitor the queues for each Prospect Source (e.g. Bing), then “pop” a prospecting term off the queue. That Prospector queries the Source, extracts the prospect URLs from the results, then adds those results to the Crawl Queue.


Rate Limiting and Prospecting

One thing that’s incredibly important to me is that we’re considerate of the technical resources from which we’re both Prospecting and Crawling. In other words, if we were to send 4,000 requests a second to Bing, we’d not only get blocked, but we’re also kind of assholes at that point. The same goes for the actual URLs we crawl, which I’ll get to in a bit.

Most folks “get around” being blocked by different sites by using a bunch of IPs or proxies so it’s more difficult to track that it’s actually one person making a bunch of requests. While ontolo does something similar, we also try to be very conservative and considerate in the rates with which we prospect from sites.

This is why the total number of Prospect Sources is important: more sources let us get you more prospects, faster, while still being considerate in terms of how frequently we visit a Prospecting Source each minute. In other words, if we decide “Across our entire prospector network, we’re going to only visit a site X time per minute,” more Prospecting Sources let us be equally considerate to all of our Prospecting Sources, but get you’re able to get more prospects, faster.


Uploading Prospects

This is why uploading prospects is so much faster and can really begin to push the limits of the ontolo crawler’s network connection: There’s no waiting, the prospects are immediately added to the Crawl Queue.


Crawling

Similarly to how the Prospectors are constantly monitoring the queues for each Prospecting Source, the Crawler is constantly monitoring the Crawl Queue.

As soon as a new Prospect is added to the Crawl Queue, the Crawler immediately runs a rate-limiting check, then downloads that Prospect URL.


Rate Limiting the Crawler

Right now, the Crawler is rate-limited in a very basic way: if a Prospect pulled from the Crawl Queue, and that site has been crawled in the previous 1 second, the prospect is eliminated. Forever.

It’s a pretty crude form of rate-limiting, and it actually leaves a lot of prospects uncrawled.

For that reason, the new rate-limited queue will likely be released in the next two weeks, *possibly* this weekend. And that queue will allow 32 different web pages per site to be in the queue.

Technical Note:

It turns out, rate-limiting algorithms aren’t as straightforward as the idea itself, and there are a few completely-reasonable ways to do it. I ended up designing several different algorithms and data structures, but none seemed to be the sort of elegant solution I look for. Until it hit me the other day.

The new algorithm and structure will take less than 5mb of memory, will hold data for more than 16k hostnames, and will allow the queue to store up to 32 URLs per hostname. Obviously, all of that is tunable, and the 16k may need to be extended to 32k or 64k. But the point is that the data stays small, increasing the odds of it sitting more often in the CPU’s L3 cache. At the same time, we’ll begin rate limiting per two seconds, rather than one second.

Some of you might have noticed that I’ve mentioned that the rate is limited by hostname rather than IP. I might go into this decision at some point in another post, but the important point is this: rate limiting by IP is a better way to do so, all around, BUT, it jacks with another part of the crawl process enough that I’m putting that off until later, as a very different solution would be needed, and major pieces of code would need to be rewritten. The solution I mention in the previous paragraph is an easy drop-in to the existing process.

Once a prospect makes it through the rate limiter, it then goes through the process of being downloaded. Once it’s downloaded, it then goes to the Parser.

Technical Note:

The download process, even here, sticks with the idea of only using external libraries when absolutely needed: DNS lookups are handed sending requests via UDP. The DNS results are then parsed by the crawler before sending the request to the hostname’s IP. This entire process of parsing DNS packets is handled, byte by byte, in the crawler, not via syscalls like gethostbyname() or getaddrinfo(), etc. This lets us much more efficiently create asynchronous DNS requests.

There is, however, an increase in the number of syscalls, as getaddrinfo() is only a single call, while a UDP request requires, at a minimum, calls for socket(), sendmsg(), read()/recv(), and close(). But, clearly, the benefit of async calls far outweighs the cost of the syscalls.


The Parser

Once we’ve downloaded the HTML for the Prospect, it’s time to parse it.

If you’re unfamiliar with the idea of “parsing,” the simple idea is this: “On this we page, lets extract the Title and links.” Parsing is that process that pulls that information out of the HTML code.

But ontolo’s parser is far more detailed than that.

ontolo’s parser looks at a web page much more like a human does, rather than a computer. It’s able to “figure out” where the header, side navigation, bottom navigation, comments, and article are all located. This lets it separate out things like “the outbound links in the sidebar” because they might be blogroll links.

But it’s even more detailed than that. ontolo identifies seven “Sections” and more than a dozen “Contexts” per section, and several different “Elements” (links, text, etc) per Context. This let’s the ontolo parser identify things like “the author’s name of the second comment” or “the publish time of the actual article” or “the RSS feed URL” or “the social media links in the footer.”

This kind of granularity, as far as I know, exists nowhere else. And because of how ontolo is so obsessively and meticulously designed, it’s able to do all of this at a rate of more than 4,000 web pages per second, while most “common” solutions like this one by moz is able to identify only the main content block at a rate of around 100 web pages per second.

To be clear, this isn’t a knock against moz. They’re solving a very different problem in a very different way. But this does demonstrate both the level of difficulty of the problem, along with speed improvement you get with ontolo vs. other marketing tools.


The Index

Once the Prospect has made it through the parser, and is now in tens of thousands of little pieces, neatly categorized, it’s time to send it to the search index.

This is pretty straightforward, but, for competitive reasons, I’m going to avoid discussing the details of how ontolo stores this data.

But I will say this: this is the piece of the puzzle that, if you get it wrong, the entire thing will never work. It took me almost 12 months to figure this piece out. And I’ve never seen the principle that allowed me to figure this out, discussed anywhere else. But it’s a tough one.

That said, the ontolo index is fast. Not only can you add a lot of documents to it very quickly and have them ready for you in a split second, but when you search on the entire set, it’s still very, very fast, almost always getting results back in under a quarter second, even when there are millions of documents in an index.


Querying

Now that you’ve added Prospects to your index, it’s time to let the querying tools filter them out. I’ve made the analogy before that ontolo is like casting a wide net AND going through it instantly with a fine-toothed comb. The querying process is the fine-toothed comb. Or, perhaps more precisely: each query is like lots of fine-toothed combs.

When you run a search on Google, you usually type in a few words, then get your results. You do the same with ontolo, but you’re able to specify with ontolo that you want results with Facebook users in links, or ads on the page, or even specific ads like Google AdSense, or contact pages, etc.

So ontolo gives you not only the ability to search by topic/term, but ALSO by marketing opportunity. And this is where the querying process comes into play.

I mentioned that there are 765 Prospecting Templates that ontolo uses. Well, the file containing all of the Query Templates for ontolo is over 4,000 lines long. And, even faster than the Prospecting Templates, this is rapidly growing in size as even more precision is folded into ontolo.


Query Results

This part of the process is why the most recent release was delayed an extra three weeks. And this is also another great example of where C simply outperforms other programming languages, flat out.

As I was getting ready to release this most recent version a month ago, I was on the last step: generating the results and reports.

The front-end of ontolo is written in PHP (I’d like to make a joke here about writing my own web server this year to replace Nginx and WordPress to get even more speed out of it, but that wouldn’t be a joke.) and I was having PHP handle the final step of results analysis.

I was about halfway through the report generation, but the processing was taking about six seconds. At that rate, it would have very possibly taken more than 10 seconds to fun the final analysis.

So, two weeks later, I had a post-processing server written from scratch that sits between the web server and the index. When you run a query, that query goes to an application I refer to as IDX, and IDX talks to the actual Index. IDX gets the results, then runs the analysis that gets you things like the Overview report. It then sends that data to the web server, which displays the results you see.

But instead of taking 10 seconds to do it, it averages completing the entire analysis in 0.15 seconds. And I have a new version I’m wrapping up that I expect could get it to 0.05 seconds or faster.


Today is the Last Day for the Discount on All Plans

If this is something you might be interested in, for this week only, we’re offering 10% off all plans, including yearly plans, which are already discounted 10%.

Click Here to Sign Up Now


This Week’s Tutorials

I’m walking through various use cases this week. I’m guessing at least one, if not all, would be pretty useful for you. So be sure to read these each day this week.

Here’s the lineup:


And that’s that.

I’m looking forward to helping you dramatically cut your research time so you can focus on the more creative parts of your campaigns.

I’ll see you on the other side.

– Ben

Influencing the Influencers: Donald Trump and Reputation Management with ontolo

I want to start off by saying that this post is politically agnostic/secular. I realize that’s not always easy, so we’ll see if I can pull it off here or if it turns into a disaster.

I’m also going to show you a couple things that are almost completely backwards from what I’ve shown you so far with ontolo. There are some counterintuitive ways to use ontolo, and to interpret data from ontolo and what it might mean.


Trump: A Reputation Management Machine

Whatever your feelings are about Trump and his politics, two things might be agreed upon by anyone:

  1. He’s masterful at gaining large amounts of media attention.
  2. That attention is not always positive.

For that reason, he’s a clear choice to demonstrate how you can use ontolo for *some* parts of reputation management research.

If you do find that you, an executive at your company, or some other public situation would be best met with some action toward reputation management, I can, unequivocally, recommend Andy Beal and his consulting company, Reputation Refinery.

Now that I think about it, I’ve probably worked more hours with Andy than I have with anyone else in my life. He has strong ethics, knows how to enjoy himself (with frequent trips to Hawaii with his wonderful wife, Sheila, who runs the top Hawaiian travel blog, Go Visit Hawaii), and does wonderful work with the homeless through the Raleigh Rescue Mission.

On top of all of that, he’s:

In addition to being the easiest person in the word to intro, he really, really knows his stuff when it comes to reputation management. So let’s hope I get this right.


Throwing a Bomb into the Bomb Factory

There’s an ethos that rose from the hacker culture of the 80s and 90s. And when I say “hacker culture,” I don’t mean it like it’s used today. I mean it as: gaining illegal entry into computers and networks.

That ethos is: Knowledge is Power.

While that’s still true today, as the cost of communication has come down – ie: it takes milliseconds, not months to get a message to the other side of the world – you could say that the ethos could be amended to:

Knowledge is power, and that power is amplified by your network.

Donald Trump understands this and has managed to influence the influencers to gain the attention he has acquired over the last several months. In his understanding that what gets published by the media is that which is most shocking, he’s also thrown a few bombs into the bomb factory, which he’s later had to defuse.

If I ever found myself in his situation, this is how I would use ontolo.


The Goal? Influence the Influencers

If I’ve just set off a firestorm in the media, and I need to calm that storm, what I need to do is go a level deeper, and influence the influencers.

In other words, the folks that are publishing about something that I just did, I need to go straight to them. And on the internet, that’s bloggers and social media.

Over the last few days, I’ve shown you how ontolo views a web page more like a human than a computer. It’s able to identify the main article, navigation, and comments. And, so far, I’ve shown you that links, social accounts, etc, that show up in the main article are the most important.

That’s usually true, but not today.


ontolo; Viewing Pages More Like a Human than a Computer

The benefit of splitting out a page into those three sections (ontolo actually splits it into several sections. Some of that will be discussed in Friday’s post, Behind the Scenes of ontolo’s Technology), is that you get a deeper insight into the intent of the publisher. If social media links in the main article can be considered “vouching” for the person, a social media link in the navigation is highly likely to be the account for publisher; be it a team of writers or a single blogger.

For reputation research, this is where ontolo can do what no one else can do: ontolo can find the publishers writing about a certain topic. Let’s see what that looks like.


Getting Started with Some Keywords

If you’re running a reputation management campaign, you’ll likely have specific occurrences or events that you’re working on directly. Due to polarizing nature of politics, I’m going to deliberately stay away from specific instances here.

What I *will* choose are broad terms that reflect language used by someone who might have a problem with someone else, and also general situations any candidate might find themselves in during an election:

  • Donald Trump
  • Donald Trump 2016
  • Donald Trump beliefs
  • Donald Trump Bernie Sanders
  • Donald Trump candidate
  • Donald Trump comments
  • Donald Trump detractors
  • Donald Trump disagree
  • Donald Trump dislike
  • Donald Trump election
  • Donald Trump GOP
  • Donald Trump Hillary Clinton
  • Donald Trump ideology
  • Donald Trump John Kasich
  • Donald Trump nominee
  • Donald Trump offensive
  • Donald Trump opinion
  • Donald Trump polarizing
  • Donald Trump problems
  • Donald Trump Ted Cruz
  • Donald Trump wrong

I’m them going to add those keywords to these sections of the ontolo tool, to “Gather New Prospects.”

  • Broad Search
  • Content Marketing : Blogs
  • Content Marketing : Reviews
  • Content Marketing : Research
  • Competition : Testimonials
  • Experts : Authorities
  • Experts : Interviews
  • Experts : Question & Answer
  • Organizations : Non-Profit
  • Organizations : Conferences
  • Organizations : Publications
  • Organizations : Associations
  • And, for kicks, Link Building : Submit Content

A couple minutes later, while it’s still running the last bits of prospecting, we’re at 26,239 prospects.

And when we search the term “Donald Trump”, we can see that 24,544 of our 27,459 documents (which have increased since the last screenshot) mention the words “Donald” and “Trump.”

And by the time we search the phrase, “Donald Trump,” we’ve now got 30,740 prospects, 26,083 of which have the phrase “Donald Trump.”

Let’s see what we’ve got in here.


Finding Bloggers and Social Media Accounts

Each of the 12 screenshots below is from the Content Marketing : Blogs search form. The idea here is to find bloggers and publications that are writing about Trump and using negative language in the article.

For each one, we’ll show the overview of Social accounts, Email addresses and contact forms, website counts in the Results set, then, finally, showing a small slice of the actual results, and how easy ontolo makes it to find contacts and who to reach out to.


Query: Content Marketing: Blogs: [Donald Trump offensive | offend | offends | offending | offended]

Social Media Accounts. Note how accounts appearing in the “Navigation” column often line up with the image two down for “Result Links.” This is how ontolo can help you make more insightful decisions about the meaning of links, accounts, etc.

Social Media Accounts:

Email Accounts and Contact Forms:

Search Result Sites and URLs:

Actual Result URLs:


Query: Content Marketing: Blogs: [Donald Trump wrong]

Social Media Accounts. Note how accounts appearing in the “Navigation” column often line up with the image two down for “Result Links.” This is how ontolo can help you make more insightful decisions about the meaning of links, accounts, etc.

Social Media Accounts:

Email Accounts and Contact Forms:

Search Result Sites and URLs:

Actual Result URLs:


Query: Content Marketing: Blogs: [Donald Trump detractors]

Social Media Accounts. Note how accounts appearing in the “Navigation” column often line up with the image two down for “Result Links.” This is how ontolo can help you make more insightful decisions about the meaning of links, accounts, etc.

Social Media Accounts:

Email Accounts and Contact Forms:

Search Result Sites and URLs:

Actual Result URLs:


Wrapping Up

As you can see, the nature of insights from ontolo lets you discover intents and insights with a very high level of precision.

If this is something you might be interested in, for this week only, we’re offering 10% off all plans, including yearly plans, which are already discounted 10%.

Click Here to Sign Up Now


This Week’s Tutorials

I’m walking through various use cases this week. I’m guessing at least one, if not all, would be pretty useful for you. So be sure to read these each day this week.

Here’s the lineup:


And that’s that.

– Ben


p.s. If you feel like it might help to get some additional help with your reputation monitoring, I highly recommend Andy Beal’s Trackur for reputation monitoring. And if you’d like to bring in a consultant or agency, his reputation management agency, Reputation Refinery is among the best.

6,194 Queries and 41,932 Prospects in 15 Minutes: The New ontolo for Link Building

“I hate launching. All the stresses of putting yourself out there and bugging people.”
– Stephenie Zamora

I’m guessing a few of you also know that feeling.

Today, I’m showing you how to use the new ontolo for good, old-fashioned link building, using lots of screenshots

By the end, you’ll see how ontolo ran 6,194 prospecting queries across more than 100 sources, and discovered 41,932 link prospects in under 15 minutes.

No only that, but you’ll also see ontolo’s new exports and how ontolo’s search functionality helps you sift through huge lists of link prospects with a fine-toothed comb, all in a split second.

Let’s get started.


Quick note: this week only, to celebrate the launch of this major release of ontolo, I’m offering 10% off all account levels. Details are at the end.


Prospecting for Business Coaching: Stephenie Zamora

For this walkthrough, I chose to use as an example Stephenie Zamora and her newest business that’s set to launch next week, on May 16th, YourPassionBasedBusiness.com, which helps folks connect more deeply into their most meaningful work and contributions.

I’ve known Stephenie for a couple years now and can personally vouch for her authenticity, compassion, depth of caring, her ability to connect deeply with others, and her ability to help others connect with what’s most important to them.

She’s been a tremendous friend and mentor, and today, we’re going to walk through doing some low-level link building to get her new business going. She’s also a prolific writer, writing and publishing almost every day. I touch on guest post opportunities, but this article focuses exclusively on link building.

So let’s get started.


Building Our Initial Keyword List

We’re going to begin with these keywords. Again, these are very broad, and that’s completely fine. We’re going to let ontolo do the work of figuring out how to narrow things down. And the more and broader data ontolo has, the better it will be able to narrow in on opportunities, in the early stage. It’s a bit counter-intuitive, but that’s one of the things that sets ontolo apart.

  • self help
  • personal development
  • self improvement
  • personal growth
  • self development
  • self growth
  • personal improvement
  • starting a business
  • passionate business
  • solopreneur
  • coaching business
  • personal coaching business
  • life coaching business
  • business coach
  • business coaching business
  • life coach
  • personal coach
  • business coaching
  • life coaching

I’m going to add these to the broad search prospecting. This *won’t* modify these terms before sending them to the sources. This is a much faster prospecting step, but only because there are fewer total queries behind the scenes. As a result, you get fewer prospects.

With fewer prospects, and because Stephenie writes so much great content, so consistently, I’m also going to add these terms to:

  • Content Marketing > Blogs
  • Content Marketing > Submit Content
  • Content Marketing > Ebooks
  • Content Marketing > Guest Posts

Then, for a pure link building and SEO play, I’m going to also prospect under:

  • Link Building > Directories
  • Link Building > Link Lists
  • Link Building > Blogrolls
  • Link Building > Submit Links

I’m also going to prospect for other expert and authorities, as they would be outlets for her to offer interviews.

  • Experts > Authorities
  • Experts > Interviews
  • Experts > Questions & Answers
  • Experts > Tips & Advice


Initial Prospecting: 12,217 in About a Minute.

By the time I was done adding Prospecting Terms for each prospecting type, there were already 8,694 prospects in her index. By the time I’m done writing this sentence, there are now 12,217 prospects in her index.

I’m going to wait a couple minutes for the sake of this walkthrough and letting the prospecting wrap up. Remember that, on Friday, I’ll be going over more of the behind-the-scenes technical details of how ontolo works, including more about how the prospecting works, what makes it faster, what we do to be considerate of other web servers, etc.


An Important Note on the Speed of ontolo

It’s now less than five minutes from when I began the prospecting, and the prospecting queue has finished up. I want to talk about the speed of ontolo here, because I think this is something that’s important to understand.

When I set out to build this version of ontolo in September of 2014, there were three things I committed to with an unreasonable and deeply-stubborn level of conviction:

  • ontolo must gather an enormous amount of data
  • ontolo must analyze that data with an absurd level of precision unavailable anywhere else
  • ontolo must feel impossibly fast.

ontolo’s crawler is the fastest I know of. I’ve seen moz reports that discuss crawling at about a rate of 25,000,000 URLs per day. ontolo is able to crawl more than 400,000,000 URLs per day, and has even pushed at a rate of more than 500,000,000 per day, with the limit not being the server it runs on, but its gigabit connection to the internet holding it back, as the server is using less than 25% of its resources when running at that rate.

So when you’re waiting on ontolo, it’s not the crawler, it’s the prospector. One of the things that’s important to me is that we’re considerate of the resources of other servers we use for querying. And this is why the *number* of prospecting sources is so important: with more sources, we can get more prospects, faster, while still being considerate of the our prospecting sources. So when I say we have more than 100 sources, it’s not as much so that you can find *more* prospects, it’s that you can find more, *faster*.

I wanted to explain that a little bit. Because behind the scenes, while you’re seeing only the prospect number tick up some amount, it’s actually doing an insane amount of work behind the scenes.


Total Queries Run

So, if I look at how these 19 keywords are expanded, each prospecting type is expanded into this many different *queries*:

  • 22 queries: Content Marketing > Blogs
  • 48 queries: Content Marketing > Submit Content
  • 6 queries: Content Marketing > Ebooks
  • 65 queries: Content Marketing > Guest Posts
  • 11 queries: Link Building > Directories
  • 21 queries: Link Building > Link Lists
  • 16 queries: Link Building > Blogrolls
  • 21 queries: Link Building > Submit Links
  • 32 queries: Experts > Authorities
  • 33 queries: Experts > Interviews
  • 34 queries: Experts > Questions & Answers
  • 17 queries: Experts > Tips & Advice
  • Total queries per keyword: 326
  • x19 keywords = 6,194 queries.

So that’s 6,194 queries that ontolo is performing across a bit over 100 different sources. So that’s over 60 queries per source. And that takes a bit of time to be considerate of the technical resources of sites we prospect from.

So that’s the wait time you’re experiencing, it’s not at all from the technical capacity of ontolo.

The reason this is important to me is because, like you, I’m sure, I don’t like waiting. But I can be pretty ornery and cantankerous, so maybe waiting doesn’t bother you as much as it does me, but I’ll do everything I can to keep you waiting as little as possible.

And with that, back to our regularly scheduled program…


Narrowing in On Prospects

There are now 41,932 prospects in Stephenie’s index, and it’s wrapping up the last bits of prospecting.

To see how much it helps to have your prospects in a fully-searchable index (as opposed to simply having backlink data or csv exports), here’s a screenshot of the social overview *without* a search term:

And now, *with* a search term:

coach | coaches | coaching | “self help” | “personal development” | “personal growth”

To put it another way, when you use a prospecting or research tool that doesn’t have the ability to query the index, you will spend far less time with ontolo in order to find the same number and quality of prospects.

This increase in prospect relevance is a result of the ontolo philosophy that could be said something like, “cast a huge net, then get the fine-toothed comb.”


Finding Link-Building-Specific Prospects

Since this post is about link building and SEO, I’m going to go straight to the “pure” link building and outreach tactics. Then, because Stephenie is such a prolific writer, I’m also going to look a bit into writing opportunities.

Since we’re looking directly for prospects, we’ll skip the overviews and head straight to the results for Directories, Link Lists, Blogrolls, and Link Submissions. I’ll include screenshots for each *without* a query, then *with* a query.


Finding Directories

Prospects: Link Building > Directories


Finding Blogrolls

Prospects: Link Building > Blogrolls



Finding Link Submissions

Prospects: Link Building > Link Submissions



Exporting and Reviewing

Now that we have our lists, we’ll export them and take a look at the exports to get contact information and URLs. I went through the top ten results on each, and spend no more than 30 seconds on each page to see if there were link opportunities. Highlighted pages below are ones where I found opportunities for niche directory submissions, link submissions, and/or blogrolls:

Directories:

Link Submissions:

Blogrolls:

You can see that, especially when you’re able to filter prospects so narrowly, you’re able to much more quickly find highly-targeted prospects.

Likewise, if you prefer to work in a spreadsheet, ontolo exports all the data from the web interface, and does so in a way that makes it easy to filter down more precisely to what you’re looking for.

And, yes, contact emails and contact pages are also included. Here’s a list of the column headers in your CSV exports:

  • Rank
  • Hostname
  • Path
  • URL
  • Title
  • # OBLs: Body: Followed
  • # OBLs: Body: Nofollowed
  • # OBLs: Nav: Followed
  • # OBLs: Nav: Nofollowed
  • # OBLs: Comments: Followed
  • # OBLs: Comments: Nofollowed
  • # Contact: Emails
  • # Contact: URLs
  • Contact: Emails
  • Contact: URLs
  • # Social: Twitter
  • # Social: Facebook
  • # Social: LinkedIn
  • # Social: Pinterest
  • # Social: Youtube
  • # Social: Google Plus
  • # Social: Instagram
  • # Social: Tumblr
  • # Social: Github
  • Social: Twitter
  • Social: Facebook
  • Social: LinkedIn
  • Social: Pinterest
  • Social: Youtube
  • Social: Google Plus
  • Social: Instagram
  • Social: Tumblr
  • Social: Github
  • RSS Feeds
  • # Affiliate Links
  • # Files: PDF
  • # Files: Spreadsheet
  • # Files: Archive (e.g. zip)
  • # Files: Audio
  • # Files: Word Docs
  • # Files: Videos
  • Files: PDF
  • Files: Spreadsheet
  • Files: Archive (e.g. zip)
  • Files: Audio
  • Files: Word Docs
  • Files: Videos


Possible Next Steps

She could then check back at the overview reports, curate the top sites, then head out and get the backlinks from the top sites.

With these lists from moz, AHrefs, or Majestic, she could then upload them and have even more prospects, but now with the ability to quickly and precisely narrow in on specific opportunities.

Because she writes and publishes *DAILY*, she could also research to find more guest post opportunities, which she already publishes on a regular basis. ontolo would let her find new opportunities, and more specific audiences, as well as broader audiences.

Since there are specific needs and goals she’s looking to help folks with, she could research to find sites that have published guest posts about those specific topics. Then, during her outreach, she could reference those other guest posts that have already been published, further increasing her placement rate.

This idea – that a site which has published something about a topic is more likely to do it again – functions on the idea of Commitment and Consistency, from the book Influence, The Psychology of Persuasion, by Robert Cialdini…a book I’d recommend to every person in or near the world of marketing.


Sign Up Now; 10% Off, This Week Only

Remember that this week is 10% off all plans, that’s for both Pro and Agency, including an *additional* 10% off yearly plans, which are already discounted by 10%.

Click Here to Sign Up Now


This Week’s Tutorials

I’m walking through various use cases this week. I’m guessing at least one, if not all, would be pretty useful for you. So be sure to read these each day this week.

Here’s the lineup:


And that’s that.

– Ben


p.s. If you feel like it might help to have someone on your side as you’re going through the challenges of growing a business that aligns deeply with your values, I can personally vouch for and recommend Stephenie very highly. She’s not only been a great friend, but also an amazing mentor to me over the last couple of years.

Her newest coaching program is set to launch next week, on May 16th, which you can view at YourPassionBasedBusiness.com.

ontolo v7 Demo: Content Marketing Research and Identifying Influencers

Today, I’m showing you how to use ontolo for content marketing research and identifying influencers in a couple of different ways. More specifically, to identify guest post opportunities.

I’m going to walk you through the new reports and data, show you (lots of screenshots) how to identify top influencers on social media and also the top-influencing websites, and then I show you how to find guest posts related to specific topics, all in a split second.

By the end, you’ll see more than 60,000 new marketing prospects readily available for searching, exporting, etc.

I wrap it all up by showing you how quickly ontolo can crawl prospects, and how you can make those old backlink exports orders of magnitude more useful than they have ever been.

NB: For the folks who missed the 25% off on Friday, I’m offering 10% off this week. The link is at the end of this post.

I’ll also be walking through various use cases this week. I’m guessing at least one, if not all, would be pretty useful for you. So be sure to read these each day this week.

Here’s the lineup:

  • Monday: Content Marketing Research and Identifying Influencers
  • Tuesday: SEO and Link Building
  • Wednesday: Social Media and Community Building
  • Thursday: Competitive Analysis and In-Depeth Influcer Research
  • Friday: Behind the Scenes of ontolo’s Technology

Let’s get started.


Prospecting: John Doherty and His New Venture: Credo

For this walkthrough, I chose to use as an example John Doherty and his new business, Credo, that connects folks to other folks looking for marketing consultants.

Most of you probably know John from around the industry; he’s a great guy with a great combination of sharp and humble. He’s consistently produced fantastic results as both an in-house (Zillow) and agency (Distilled) marketer, and his newest venture, Credo helps folks (maybe you) find high-quality, vouched-for marketing consultants and agencies.

I didn’t run through this process first with him in mind to see what the results would be like. I just went through it and logged the process. And yet, unsurprisingly, he shows up over and over and over in the results below. It’s a real testament to how prolific he’s become as a marketer.

John doesn’t know I’m doing this, but I have a feeling he might have more of a hesitation with the compliments than in using his new business as an example…

Let’s go.


Getting Some Pretty Good Keywords

The first step of any marketing campaign – and ontolo is no different – is to get your keywords right. I’m sure you understand this already, but what I wish that more people understood, is that it’s not about the keyword: it’s about what those words mean to the person who chose those words, and not one of millions of others. It’s what the words represent; their intent, their meaning, their desires, and desired outcomes.

With John, I went to his personal site, his new business site, and his Facebook and Twitter accounts. I looked at his bios, what he linked to, and what he wrote on his blogs. From there, I pulled out a range of terms that stood out:

  • marketing
  • seo
  • digital marketing
  • online marketing
  • internet marketing
  • search engine optimization
  • growth marketer
  • growth hacker
  • entrepreneurship
  • entrepreneur
  • solopreneur
  • growing a business
  • small business

I also made a list of a sort of “template” that you could expand using the terms above. That list, using “seo” as the keyword you might insert, is:

  • seo webinars
  • seo conferences
  • seo agency
  • seo consultant
  • seo advice
  • seo tips
  • seo strategies
  • seo consulting


Using ontolo to Find Blogs

I then went to the Blog section in Content Marketing. The first set of keywords I used were:

  • seo webinars
  • seo conferences
  • seo agency
  • seo consultant
  • seo advice
  • seo tips
  • seo strategies
  • seo consulting

After you submit them, it confirms they were added to the prospecting queue:

This then found 5,661 new prospects (Noted in the bottom left). Most were found in about 30 seconds. For the sake of finding out how many prospects were found in total, I let the prospecting run for a minute-and-a-half or so before continuing on to the next step.

I’ll explain more about the prospecting process in Friday’s piece – Behind the Scenes of the ontolo’s Technology – but ontolo has more than 100 sources from where it finds marketing prospects. While the *crawler* can process more than 4,000 web pages per second, ontolo does its best to be considerate of technical resources of web servers.

For that reason, ontolo has fairly restrictive and conservative rate limiting for the frequency with which it will visit a website. What this means is that the prospecting will be slower than the crawling. If waiting 60 or 90 seconds is too long, the fastest crawler option is to upload URL lists (e.g. from moz, AHrefs, Majestic, etc). I’ll show you that here in a bit.

We now see the 5,661 prospects in our ontolo index, and that we’ve used 8 of our 500 Prospecting Terms today.

At this point, I accidentally reloaded the page while taking a screenshot, which means that it went out and prospected those terms again. So you’ll see that there are now 16 Prospecting Terms that have been used, and there are now 8,105 prospects in the index. Why even more prospects were found with the same Prospecting Terms will be explained in Friday’s behind-the-scenes article.


Finding Experts and Influencers

You can see here a few things.

First, all 8,105 prospects were searched in 0.021 seconds. If you have to wait for your research, you’re less likely do to it, or do it as thoroughly. So every aspect of ontolo was designed to be as fast as possible, and it’s even this fast when you have hundreds of thousands, or even millions of prospects in your index.

The second thing you’ll see is a sort of meta-analysis of the search results, in the section labeled “Overview.” This is a unique feature to ontolo, where it analyzes the top results that were returned by your search (in this case, every prospect in your index). As you search different terms, or use different search pages, this overview is calculated in realtime and will change for each set of search results.

The default tab for the overview is social media accounts. You’ll likely notice that this gives, perhaps, the best overview of the influencers, experts, and authorities in a market.

Because there are only seo-related keywords here, you’ll likely recognize many of these accounts.

At the same time, another unique feature of ontolo is the ability to search in different “sections” of a page. ontolo looks at and analyzes web pages more like a human than a computer. And this lets it identify accounts that are linked to from the main article vs. in the comments vs. in the navigation. The benefit here is that an analysis of the most frequently-linked social media accounts doesn’t get watered down by accounts linked to in the navigation or comments. The data is much more precise, much more relevant, and much more valuable.

As you scroll down, you’ll see even more social networks from which accounts are identified. At this point, ontolo identifies accounts from:

  • Twitter
  • Facebook
  • LinkedIn
  • Pinterest
  • YouTube
  • Google Plus
  • Instagram
  • Tumblr
  • GitHub

If you have other social sites you think ontolo should consider, email us at support@ and let us know.

I’m going to skip most of the rest of the overview, as you can see the details of that in last week’s post, Ontolo v7 is Out of Beta.

What I *am* interested in, though, is the Results tab. Here, we see an analysis of the top 1,000 results returned, and showing which websites showed up and how many times.

I also want to look at the links tab, which analyzes all of the outbound links across the entire result set:

Here, ontolo again divides up the analysis by where on the page various links, etc, are displayed. Additionally, for outbound links, ontolo will look at if the link is followed or not, then also segment it by hostname, URL, and TLD/Suffix (which will be explained Friday).

The initial view here is of the most important links: followed links that appear in the main article, sorted by how many times the hostname/site is linked to.

You’ll often see here sites like WordPress, Wikipedia, Slideshare, Blogger, etc. so the top 100 for each section are displayed.

And as we scroll down a bit, we begin to see a bit more of the sites we would expect: Neil Patel’s QuickSprout, Joost de Valk’s Yoast.com, SearchEngineJournal, SEOMoz (remember, these are URLs that are linked-to, so the old URL showing up makes sense), and even John’s site. Turns out, John knows a thing or two about SEO.

The reason these two reports are where I’d start any sort of new project is because these two Overview tabs will show you who are the influencers, who are the publishers, and who are the names in the space.

Remember, the ontolo crawler and indexer are industry-agnostic. They know nothing about the seo industry or marketing industry. But, when given a few keywords, it’s able to help you understand an industry that you may be completely unfamiliar with, in only a matter of seconds.

With these URLs, we could pick out QuickSprout, SearchEngineJournal, moz, CopyBlogger, et al, and begin building a list of great blogs to monitor, read up on an industry, target for promotions/ad buys, target for guest posts, etc. ontolo will help you do this for any industry.


Finding Lots More Prospects

Now that we have a list of top SEO blogs, I’m going to go ahead and add some more Prospecting Terms.

John has proven to be not only a great writer on the topic of SEO and marketing, but also on entrepreneurship on his blog at GetCredo.com/blog. He also possesses one of the top qualities needed for any sort of writing campaign: consistency.

So I’m going to go ahead and research to find Guest Posts. I’m also going to expand this into many more Prospecting Terms. Remember that ontolo can only do what you tell it to, and can only research topics you tell it to research. ontolo gives you a very high daily keyword limit for this reason: use them up. Follow your whims and tangents, putting whatever keywords you can think of into ontolo, and let ontolo take care of the rest.

To get an idea of the scale of ontolo, the Guest Post tool helps here.

Behind the scenes of the prospecting process, ontolo takes your keywords, then creates various search combinations from them. The Guest Post tool takes each keyword you input, and creates more than 20 search permutations. So, while we just added 76 keywords, over 1,500 different queries were created. Given that we have over 100 prospecting sources, and how some of them overlap results, a conservative estimate is that over 100,000 prospecting queries are being run. And it all happens in mere minutes.

For the sake of this guide, I’m going to give this some time to run, and come back to it 15 minutes later. Here are some stats for the first couple minutes it ran, and how many prospects were then in the index:

  • 30s: 10,194
  • 60s: 11,717
  • 2m: 14,467
  • 3m: 17,264
  • 15m: 31,215

…and it’s still finding more prospects. Continuing on.


Narrowing in on Guest Posts

Now, instead of just analyzing the overview, we’re actually going to perform a couple of searches in order to find specific guest posts. By default, ontolo will display in the overview the most relevant guest posts it’s found. Note that, because there are no keywords yet added to the search box, these are purely relevant to being guest posts:

Now, we’re going to *just* search “seo.” We don’t need to add things like “seo guest post” or “seo guest author” because ontolo handles all of that behind the scenes in the queries as well.

Also, here’s a bit of a tip, and a deeper understanding of ontolo: you’ll see that, with the default “guest posts” search, the overview is still relevant. And you might be able to make the assumption that, with HubSpot in the third position with 5 twitter links made in the articles, that HubSpot has – maybe – written five of the guest posts that appear below.

Obviously, you can’t be certain without going through each one, but there’s a lot of information within ontolo that isn’t always obvious. This is the power and insight you receive when you’re able to analyze so much data, so quickly, and with much more precision like this.

You now see a number of guest posts listed below. And it makes sense that you might have some lower-quality sites near the top of *some* searches. Especially in the SEO industry.

But we don’t have to scroll very far to find links from HongKiat, CopyBlogger, and the wonderful Heather Lloyd-Martin’s seocopywriting.com, which specializes in copywriting for SEO.

And hey, look at that. There’s a guest post on John’s site at #47.

So as you can see, ontolo helps you filter prospects far faster than any other tool on the market, for a wider variety of marketing purposes than any other tool on the market.

And that’s the entire point of ontolo: to significantly reduce the time you spend on research by getting you better, more in-depth research, faster.

No more aimless trolling through exports of backlinks to find specific kinds of opportunities. ontolo gets your right there, quickly.

And speaking of…


Breathing New Life into Backlink Exports

Let’s say John wanted to use some of his moz, AHrefs, and MajesticSEO exports.

I actually recommend all of those services if you’re looking for backlinks, something that ontolo does not do. And ontolo gives you a way to make them even more useful: Uploads.

Using the upload tool, just drag and drop your CSVs or plain text files. ontolo will find all of the URLs in there, then add them to your prospecting queue. And because there’s no delay in prospecting, this all happens very fast.

But how do you know which sites to get backlinks? Well, we check those overviews we ran before.

I’m going to run Open Site Explorer backlink reports for 10 sites that showed up in the Broad Search overview list as links to websites that were followed and in the main article content:

  • moz.com
  • copyblogger.com
  • seocopywriting.com
  • quicksprout.com
  • searchenginejournal.com
  • marketingland.com
  • searchengineland.com
  • hubspot.com
  • kissmetrics.com
  • seroundtable.com

I also then ran a bunch of reports from Fresh Web Explorer. I uploaded all of them and in just a minute or so almost doubled the size of the index.

And the index size is now over 60,000 URLs.

If you’ve got a bunch of reports to upload, go for it. Right now, there’s not a restriction on how many you can upload, but when a limit is implemented in the next couple of weeks, I expect it might be similar to the 100/500 split for account levels as they stand. And, really, if you’ve got 100 reports, go ahead and drag ‘n drop them right on over. ontolo will cruise through them.

One note on that is that you might want to upload them 2-3 times due to the queuing system that’s currently set up for rate limiting. This is one of the first improvements that’s going to be made soon, maybe even this week, but in the meantime, go ahead and upload the files a few times with a few seconds in-between to get more prospects from the same site.

As a final step, let’s check out our “seo guest posts” query.

Over 5,000 web pages…that’s a pretty good, highly-targeted prospect list. And if you wanted to narrow it even further, you could do just that:

So if I’ve just written – or plan to write – a great piece on keyword research, and I’d like to publish it as a guest post, that’s 784 prospects I’d want to check out. And because they’re so highly targeted, getting that piece placed would take far less time, with far less outreach, than with any other tool.


Sign Up Now; 10% Off, This Week Only

Remember that this week is 10% off all plans, that’s for Pro both pro and Agency, including an *additional* 10% off yearly plans, which are already discounted by 10%.

Click Here to Sign Up Now

Enjoy.

– Ben


p.s. If you’re looking for help from a consultant or an agency, be sure to check out John Doherty’s new service, Credo.

ontolo v7 is Out of Beta. +25% Off, Today Only

The New ontolo is Here.

New Features Include:

  • Full Exports with Contact Emails and Contact URLs.
  • Much Bigger Prospecting that now includes more keywords behind the scenes, as well as now over 100 different prospecting sources.
  • Additional Search Analysis to identify authorities, experts, affiliates, advertising, and more.
  • And lots, lots more

Let’s walk through it.


The Dashboard

Includes statistics on approximately how many of each prospecting type is included in your current database.


The Home Screen

The home login screen.

Prospecting Categories

The left navigation includes categories not only for prospecting, but also for the various searches you can perform on your database.

The categories shown here are the high-level categories. There are nearly 100 sub-categories so you can precisely find the marketing opportunities you’re looking for.


Prospecting: Prospecting Terms

ontolo takes your prospecting terms, then creates lots of different permutations of each. Each of those permutations is then searched across over 100 different prospecting sources.

Once those prospects are found, each one is then crawled and added to your database. Because ontolo is so fast, these prospects are often found and added to your database in a matter of seconds.

Prospecting: Uploading CSV/TXT Files

Uploading files is the fastest way to prospect and will push ontolo’s crawler as fast as it can go: over 4,000 web pages per second, from the time the crawler receives it, until it’s in your database.


Searching Your New Prospects

Once you have prospects in your database, the best way to access them is to perform various searches.

Here, we’re saying “show me pages that mention seo, marketing, the phrase ‘link building’, or the phrase ‘search enigne optimization’.”

Additionally, since we’re on the “Tools” page, this will search the entire database a number of different ways in order to find prospects that are about seo tools, etc.

With searches broken out into these pages, you don’t have to think of all the ways you’d look for things like “seo tools,” or “seo guest posts.” ontolo takes care of all of that for you.


Search Results: Overview: Social Media Accounts

Each time you search, the top 1,000 results are analyzed as their own segment. The idea here is that a search for “seo tools” is going to have a different slice of relevance from your entire index. This also lets you use a single ontolo index for many clients and many different industries, and the searches take care of filtering out irrelevant prospects.

The default tab for the results overview is for social media accounts.

You may also notice that accounts are split up into three different columns. This is one of several things that only ontolo can do… You’ll also see this further on down. Here’s what these columns mean…

The three columns are: “Articles,” “Comments,” and “Navigation.”

One of the ways that ontolo is able to so precisely find link prospects among millions, is that it looks at a web page unlike any other tools, but just like a human does. So it’s able to extract information just from the main article, from the comments, or from the navigation.

In truth, ontolo splits web pages into more than 200 different sections, and uses this ability to more precisely find prospects.

How this is useful for you is that ontolo is able to separate out social media accounts (and links, etc) and show you who’s *actually* being referred to with authority. Links or social accounts in the navigation are likely to promote the website. But links (which we’ll get to in a second) and social accounts, etc, in the main article indicate that this is a highly-valued account. And you can see here in this list, these is a nice segment of the Who’s Who of SEO.

Keep in mind this can be applied to any industry, and any market. This unique ability of ontolo’s makes ontolo a useful research tool, not just for online marketing, but also sales, lead generation, content research, and more.

Search Results: Overview: Contact Information

Ontolo also extracts email addresses, and makes pretty good guesses about contact pages as well.

Search Results: Overview: Outbound Links

Here, you see links split up in a similar was as the social media accounts, showing which sites and pages are linked to from the article, comments, and navigation.

Additionally, links are segmented into followed and nofollowed links.

Again, this simple analysis can show you the authorities and experts in a space, with you knowing absolutely nothing about it. And it all happens in mere seconds.

Search Results: Overview: Advertising Opportunities

Another unique feature of ontolo is the ability to identify advertising opportunities. It’s safe to call this a beta feature, but you can see how easily opportunities are identified.

One of the best use cases for this kind of report is to idenfiy sites that display adsense ads. Then, if you were to spend a budget buy ads through the Content Network, you could much more easily target a huge range of relevant websites and, ideally, even drive down your spend while your clickthroughs increase.

In addition to the sections you see above for “display/banner ads,” and “adsense,” ontolo also identifies web pages and links to sponsorship opportunities (e.g. conferences), donations and non-profit-related pages, and more.

Search Results: Overview: Website Features

A number of website features are also identified. This could be used for competitive analysis, or simply as another lense into the market you’re analyzing.

These features include:

  • Web Analytics (and not just Google Analytics)
  • User Login Forms
  • Search Forms
  • Language Translation Tools
  • User Registration Forms
  • Video Analytics
  • Geo-IP Tracking
  • And Live Chat

Search Results: Overview: Community/Commenting

If you’re looking to build your community beyond social, this section of the overview will help you find blog posts that allow commenting, as well as sites with forums and messageboards.

Additionally, comment forms are separated into two different types: 1, where the form is embedded into the web page, and 2, where the form is handled by a third-party like Disqus.

Search Results: Overview: Affiliate Publishers and Programs

The affiliate overview helps you to either find new affiliates to pitch on your products or affiliate programs for signup.

Consider the identifying of affiliate programs to be in beta.

Search Results: Overview: Linked Files

If you have new PDFs, reports, spreadsheets, etc, you’re looking to promote, this can be a great way to do some research to round out your resource.

Additionally, you may find opportunities for who to pitch your content to if they’ve already linked to another similar resource.

Search Results: Overview: Result URLs

The final section of the overview includes an analysis of the search results, which we’ll go over next.


Search Results: Data: Exports

First off, the new ontolo has a major up grade in terms of exports. Everything you seen in the following tables is include; contact information, social media accounts, etc. It’s all there.

(exports for the overview, with even more data, are coming in the near future.)

Here are a couple of the report features you’ll be interested in, and which will help you understand everything else that’s included with each result.

Search Results: Data: Emails and Contact Forms

You’ll notice that, in addition to result titles and links, you also have two more columns of data on the right.

These icons are greyed out when there are none of these bits of data on the page, then light up to their respective color if they are. You can see that each of the results here has an RSS feed.

Likewise, for all ofthe icons in the right-hand side, when you hover over it, you’ll see the link to the social account, contact form, file download, email address, etc. You can also click those links, which will open the link in a new browser tab.

The single exception here is for affiliate links. (a green percentage symbol) When you hover the affiliate icon, you’ll simply see a count of how many affiliate links appear on the page.

Search Results: Data: Social Accounts

The same applies here to social media accounts as well.

Search Results: Data: Outbound Links

Finally, in the second column, is the link data. I saved this for last since it requires a little more explanation.

First, you’ll notice that data is segmented into three columns, and two rows, with rows and columns of icons surrounding it. It might look overwheling, but it’s pretty straightforward.

Remember the “article,” “navigation,”, and “comments” sections we mentioned above? That’s each column here, respectively.

Likewise, the bottom two rows are for followed and nofollowed links.

The numbers are greyed out until there’s at least one link, at which point, it becomes a light blue color. It become a darker blue only when links in that particular section reaches 10 or more.

Then, the icon in the top corner, the standard “outbound link” icon, will light up if there are at least 10 *followed* links in one of the sections on the page.

It should be noted that none of these particular links are clickable. That may be added in the future. But it comes down to the fact that it’s such a huge amount of data, that web browsers slow down and download speeds become quite sluggish when they’re downloading and analyzing 20k+ links. It’s simply not practical to include them. But remember, the overview includes the most-frequently linked sites and pages.


Final Notes

The last thing I want to add here, that I think is important and is essentially impossible to understand from screenshots is the speed of ontolo.

I often use other marketing research tools to see how they compare, etc. It seems to be incredibly common to make you, and other users, wait several minutes – sometimes up to 20 minutes or more – to download reports. This always frustrated me tremendously.

For that reason, I’ve been completely unreasonable and unwavering when it comes to the speed and performance of ontolo. I actually wanted to release this four weeks ago, but one critical piece of the report creation was taking 3-4 seconds. Most folks would say “it’s fine” and move on. But I didn’t want to do that, and I didn’t want to do that to you.

Three weeks later of coding for 14+ hours a day and building an entirely new layer to the application, and I got the analysis down to less than a quarter of a second, then spent a few days making sure that everything loads quickly in the browser…because it gets very sluggish when this amount of data is processed (usually about 2MB of html).

I share that with you but I want you to think of ontolo as a tool that is hefty and can churn through tons of data, yet remains incredibly lightweight, quick, and easy to use. If there’s any part of it that you feel doesn’t fit that ethic, please do let me know.

So that’s it. That’s the new ontolo.


Since it’s Friday, and since this wraps up 19 months of 80-100 hour weeks (and I’ve already begun the next version), I’m going to give a discount for today only. It’s 25% off, both for monthly and yearly plans.

And the same deal applies; if you’re not happy, let me know within 30 days and I’ll get you a full refund, no questions asked.

Click Here Now to Sign Up

I hope you enjoy it.

Ben

Major Update Coming: The Prospector and More: Feb 2, 2016

After a few weeks of delays, this will be released tomorrow, March 1, 2016.

I had anticipated releasing this a week ago. Then I got pretty sick before it could be rolled out. I’m still fairly under the weather, but able to start working on wrapping up this release. Assuming I don’t get worse, this will be out next Tuesday, February 2nd.

As a number of you might know (and have contacted me about it), getting this part of the tool out is about a month late. This was for a few reasons, but the largest reason was the desire to get this piece of the new ontolo toolset done right.

Here’s what I mean…

When developing software, it’s a common choice to release something quickly, or take some extra time and do it “right.” Of course, it’s not always tell what’s the best design decision to make, but you can usually feel if the path you’re on isn’t quite right. And not being “quite right” means that you’re going down a path that might be easier now, but will cause problems later.

And I made the decision to take some extra time to get it done right. I thought it would be an extra couple of weeks, but getting sick (for the first time in about 18 months) definitely affected that.

What this means for you, as an ontolo user, is that future updates and additions (of which the list is growing…I’ll get to that in a second) are able to:

  • Be implemented very, very quickly. Far faster than if I had gone down the path I was originally planning.
  • Scale much, much more easily. Ontolo is already fast and, in its current iteration, scales very, very well. But as the additions of prospecting sources grows, it was clear there would be a bottleneck somewhere not far down the road. Rethinking this piece of the design now allows ontolo to use a (virtually) unlimited number of prospecting sources and it won’t break a sweat, no matter how many other customers also sign up. This solution wasn’t obvious, and a lot of other decisions were weighed for quite some time. But the end decision – to value scalability over ease of development – I think that’s going to be a great decision for everyone and well worth the wait.

This brings us to what’s coming up and a bit about why these decisions (and extra development time) were so important.

The Three Updates

There are three major updates coming next week. The CSV exports, additional search tools, and The Prospector.

CSV Exports

CSV exports are pretty self-explanatory.

Additional Search Tools

Additional search tools… Right now, there are a couple-dozen search tools you can use to find things like affiliate opportunities, link opportunities, content marketing opportunities, etc. That number will jump to around 100 next wee and they all sort really well into a few, clear categories.

The Prospector

The biggest update, though, is The Prospector. This is the tool I get the most requests about, and the most customers following up on it. (Really, thank you all for being so patient.) In the previous version of ontolo, The Prospector essentially just pulled search results from, let’s say, one of the major search engines into a database for you.

What’s coming next week is a prospecting tool that aggregates data from multiple different sources. While I don’t expect to ever reveal those sources, you can expect about a half-dozen to be included in next week’s release. It’s a pretty great set of sources and, on top of that, the queries used on those sources is being expanded to accommodate the scale of prospects that ontolo can now handle in your account.

But here’s where it gets interesting, and where the extra development time went…

Scaling The Prospector

With the way the new Prospector is designed, adding new sources will take less than thirty minutes after a new prospecting source has been tested and identified. Obviously we would run a number of tests against the data and that would take longer than a half hour. But to actually implement the new set of code across the several dozen machines in the network, would be very fast and very easy.

This is important, because in the process of seeking out new sources, I have a list of over 200 more to run through that have passed my initial qualification.

On top of that, we would need, quite literally, tens of thousands of prospecting sources and a huge number of customers for the prospecting network to become bogged down due to resource usage. If I had continued down the path I was on, ontolo would surely be buckling and require a complete rewrite of the Prospector very, very soon. And until that rewrite happened, it could be a pretty bad experience for you. So I decided to avoid all of that and get that work done now, and push the release back a bit.

The One Caveat

There’s one caveat here. And that’s that, as fast as ontolo is, we’re still at the mercy of politeness. Some of our sources in the future are not as robust as the huge sites out there that you know the names of. So there will be sources where prospecting will simply take longer. Along the same lines, there is a limit to our ability to prospect from sources given the size of our network. Depending on how quickly folks use it, I’ll start scaling up the network as needed.

This ability to scale the network with minimal effort was another piece that took a fair amount of time. In essence, the question becomes one of “how do you introduce an untrusted computer, with no code, into an existing network, and have it all just work and work with a minimal amount of time and effort in getting it set up?” Or, in other words “How do you add 100 computers to a network, just as easily as adding one computer to a network?” So that’s taken care of, too. And I’ll begin scaling out the network as I see more how folks are using the Prospector.

Even with that caveat, it will still be very fast. The previous version of the prospecting tool took about 5-10 minutes to run a report that might have up to 10,000 prospects in it, exported to CSV. That new set of prospecting queries could now run in as little as 20 or 30 seconds. And, instead of just spitting the data into a CSV, it’ll go into your search index, allowing you quickly find those prospects that matter most to you, without having to spend so much time sifting through prospects.

What to Expect Next Week

This will all be rolling out next week. I expect it to be out Tuesday morning, though it could be as early as Sunday night.

On top of that, some of you have noticed that documentation has not been finished. This is due to all of these changes and it becoming clear that it would be much faster to roll everything out at once, rather than incrementally. That’s a limitation of the current design, which is also resolved in next week’s update. Updates will be much easier to incrementally implement.

Thank You

I really want to say thank you for being so patient. I know this has taken longer than some folks were willing to wait, and I really, really understand that. I hate that it’s taken this long, but I feel much better that it’s being done right for you.

One thing I could have done better was to properly set expectations with this. I went dark on communication because it wasn’t clear what the updated information would look like or when it would be out. I’ve also brought someone on board who will be helping with support emails, which is an area that’s personally challenging for me. It might be a more personal post I write about later, as I’d imagine some other folks might struggle with similar things. But for now, I think that’s about as much as I’d prefer to say about it. And the person who’s starting, she’s fantastic, super kind, caring, and helpful, and will be starting next week when the new update is pushed live. If you end up needing any help with your account, etc, I think you’ll really enjoy working with her.

Additionally, future announcements will now be released /after/ updates have been made, implemented, and available to you. Exceptions to this will be major releases, which I expect to occur about once a year. For those, there will be pre-announcements to help people understand what to expect.

That said, be on the lookout next week for the announcement on this. And stay tuned as new updates are made. There are a number of features and updates I’ve never mentioned publicly. And you’ll begin seeing those showing up on a regular basis. I likely won’t announce each new, individual source added to The Prospector, but I’ll find some way to let you know when they’ve been added via a status page, etc.

Thank you all again. I’m looking forward to getting these major features out to you soon.