We test top search engine optimization (SEO) tools for keyword research, position monitoring, web crawling, and more to help businesses gather SEO data, rank higher, and execute effective digital marketing strategies.
(click image to enlarge)
What Is Search Engine Optimization?
Search engine optimization (SEO) is really a never-ending turf war. Every business, no matter the size or industry, has a website that usually includes webpages listing the key products and services it’s trying to sell. Most also have other webpages devoted to targeted content the organization needs surfacing more prominently in Google’s search results than its competitors. That’s what the “optimization” part of SEO is looking to accomplish. It’s a nitty-gritty game of keyword- and research-driven leapfrog made significantly more complicated by Google’s penchant for changing all of the rules every few months. That’s where SEO tools come in.
The emphasis on tools, meaning plural, is important. There’s no one magical way to plop your website atop every single search results page, at least not organically. If you want to buy a paid search ad spot, then Google AdWords will happily take your money. This will certainly put your website at the top of Google’s search results but always with an indicator that yours is a paid position. To win the more valuable and customer-trusted organic search spots (meaning those spots that start below all of those marked with an “Ad” icon), a balanced and comprehensive SEO strategy is a must-have. This can be broken down into three primary categories: ad hoc keyword research, ongoing search position monitoring, and crawling. In this roundup, we’ll explain what each of those categories means for your business, the types of platforms and tools you can use to cover all of your SEO bases, and what to look for when investing in those tools.
Breaking Down the SEO Tool Landscape
This roundup covers 10 SEO tools: Ahrefs, AWR Cloud, DeepCrawl, KWFinder.com, LinkResearchTools, Majestic, Moz Pro, Searchmetrics Essentials, SEMrush, and SpyFu. The primary function of KWFinder.com, Moz Pro, SEMrush, and SpyFu falls under keyword-focused SEO. When deciding what search topics to target and how best to focus your SEO efforts, treating keyword querying like an investigative tool is where you’ll likely get the best results.
These cloud-based, self-service tools have plenty of other unique optimization features, too. Some, such as AWR Cloud and Searchmetrics, also do search position monitoring—which means tracking how your page is doing against popular search queries. Others, such as SpyFu and LinkResearchTools, have more interactive data visualizations, granular and customizable reports, and return on investment (ROI) metrics geared toward online marketing and sales goals. The more powerful platforms can sport deeper analytics on paid advertising and pay-per-click (PPC) SEO as well. Though, at their core, the tools are all rooted in their ability to perform on-demand keyword queries.
We concentrated on the keyword-based aspect of all the SEO tools that included the capabilities because that’s where most business users will primarily focus. Monitoring particular keywords and your existing URL positions in search rankings is important but, once you’ve set that up, it’s largely an automated process. Automated position-monitoring features are a given in most SEO platforms and most will alert you to issues, but they don’t actively improve your search position. Though, in the best position monitoring and rank tracking tools such as AWR Cloud, Moz Pro, and Searchmetrics, position monitoring can become a proactive process that feeds back into your SEO strategy. It can spur further keyword research and targeted site and competitor domain crawling.
Click image to enlarge
When your business has an idea about a new search topic for which you think your content has the potential to rank highly, the ability to spin up a query and investigate it right away is key. Even more importantly, the tool should give you enough data points, guidance, and recommendations to confirm whether or not that particular keyword, or a related keyword or search phrase, is an SEO battle worth fighting (and, if so, how to win). We’ll get into the factors and metrics to help you make those decisions a little later.
Crawlers are largely a separate product category. There is some overlap with the self-service keyword tools (Ahrefs, for instance, does both), but crawling is another important piece of the puzzle. We tested several tools with these capabilities either as their express purpose or as features within a larger platform. Ahrefs, DeepCrawl, Majestic, and LinkResearchTools are all primarily focused on crawling and backlink tracking. Moz Pro, SpyFu, SEMrush, and AWR Cloud all include domain crawling or backlink tracking features as part of their SEO arsenals.
Website Crawling 101
There are three types of crawling, all of which provide useful data. Internet-wide crawlers are for large-scale link indexing. It’s a complicated and often expensive process but, as with social listening, the goal is for SEO experts, business analysts, and entrepreneurs to be able to map how websites link to one another and extrapolate larger SEO trends and growth opportunities. Crawling tools generally do this with automated bots continuously scanning the web. As is the case with most of these SEO tools, many businesses use internal reporting features in tandem with integrated business intelligence (BI) tools to identify even deeper data insights. Ahrefs and Majestic are the two clear leaders in this type of crawling. They have invested more than a decade’s worth of time and resources, compiling and indexing millions and billions, respectively, of crawled domains and pages.
Website-specific crawlers, or software that crawls one particular website at a time, are great for analyzing your own website’s SEO strengths and weaknesses; they’re arguably even more useful for scoping out the competition’s. Website crawlers analyze a website’s URL, link structure, images, CSS scripting, associated apps, and third-party services to evaluate SEO. Not unlike how a website monitoring tool scans for a webpage’s overall “health,” website crawlers can identify factors such as broken links and errors, website lag, and content or metadata with low keyword density and SEO value, while mapping a website’s architecture. Website crawlers can help your business improve website user experience (UX) while identifying key areas of improvement to help pages rank better. DeepCrawl is, by far, the most granular and detailed website crawler in this roundup, although Ahrefs and Majestic also provide comprehensive domain crawling and website optimization recommendations. Another major crawler we didn’t test is Screaming Frog, which we’ll soon discuss in the “The Enterprise Tier” section.
What Are Backlinks?
The third type of crawling tool that we touched upon during testing is backlink tracking. Backlinks are one of the building blocks of good SEO. Analyzing the quality of your website’s inbound backlinks and how they’re feeding into your domain architecture can give your SEO team insight into everything from your website’s strongest and weakest pages to search visibility on particular keywords against competing brands.
LinkResearchTools makes backlink tracking its core mission and provides a wide swath of backlink analysis tools. LinkResearchTools and Majestic provide the best backlink crawling of this bunch. Aside from these two backlink powerhouses, many of the other tools we tested, such as Ahrefs, Moz Pro, Searchmetrics, SEMrush, and SpyFu, also include solid backlink tracking capabilities.
The Enterprise Tier
The last piece of the complicated SEO tool ecosystem is the enterprise tier. This roundup is geared toward SEO for small to midsize businesses (SMBs), for which these platforms are likely priced out of reach. But there’s a handful of enterprise SEO software providers out there that essentially roll all of the self-service tools into one comprehensive platform. These platforms combine ongoing position monitoring, deep keyword research, and crawling with customizable reports and analytics.
The enterprise platforms can also tie in larger content marketing campaigns and ROI metrics for integrated digital marketing teams. Large enterprise organizations with flush SEO and digital marketing budgets may want to consider enterprise SEO platforms such as Brightedge, Conductor, and Linkdex.
In the enterprise space, one major trend we’re seeing lately is data import across the big players. Much of SEO involves working with the data Google gives you and then filling in all of the gaps. Google Search Console (formerly, Webmaster Tools) only gives you a 90-day window of data, so enterprise vendors, such as Conductor and Screaming Frog, are continually adding and importing data sources from other crawling databases (like DeepCrawl’s). They’re combining that with Google Search Console data for more accurate, ongoing Search Engine Results Page (SERP) monitoring and position tracking on specific keywords. SEMrush and Searchmetrics (in its enterprise Suite packages) offer this level of enterprise SERP monitoring as well, which can give your business a higher-level view of how you’re doing against competitors.
Platforms such as AWR Cloud still give you great ongoing position monitoring in an affordable tool, but that aspect of the market is becoming a more high-end affair. For SMBs, a smart combination of self-service tools is the much better bang for your SEO buck.
Running the Numbers
Evaluating which self-service SEO tools are best suited to your business incorporates a number of factors, features, and SEO metrics. Ultimately, though, when we talk about “optimizing,” it all comes down to how easy the tool makes it to get, understand, and take action on the SEO data you need. Particularly when it comes to ad hoc keyword investigation, it’s about the ease with which you can zero in on the ground where you can make the most progress. In business terms, that means making sure you’re targeting the most opportune and effective keywords available in your industry or space—the words for which your customers are searching.
The terms SEO experts often start with are page authority (PA) and domain authority (DA). DA, a concept in fact coined by Moz, is a 100-point scale that predicts how well a website will rank on search engines. PA is the modern umbrella term for what started as Google’s original PageRank algorithm, developed by co-founders Larry Page and Sergey Brin. Google still uses PageRank internally but has gradually stopped supporting the increasingly irrelevant metric, which it now rarely updates. PA is the custom metric each SEO vendor now calculates independently to gauge and rate (again, on a scale of 100) the link structure and authoritative strength of an individual page on any given domain. There is SeO industry debate as to the validity of PA and DA, and how influence the PageRank algorithm still holds in Google results (more on that in a bit), but outside of Google’s own analytics, they’re the most widely accepted metrics out there.
That’s why PA and DA metrics often vary from tool to tool. Each ad hoc keyword tool we tested came up with slightly different numbers based on what they’re pulling from Google and other sources, and how they’re doing the calculating. The shortcoming of PA and DA is that, even though they give you a sense of how authoritative a page might be in the eyes of Google, they don’t tell you how easy or difficult it will be to position it for a particular keyword. This difficulty is why a third, newer metric is beginning to emerge among the self-service SEO players: difficulty scores.
Difficulty scores are the SEO market’s answer to the patchwork state of all the data out there. All five tools we tested stood out because they do offer some version of a difficulty metric,or one holistic 1-100 score of how difficult it would be for your page to rank organically (without paying Google) on a particular keyword. Difficulty scores are inherently subjective and each tool calculates it uniquely. In general, it incorporates PA, DA, and other factors, including search volume on the keyword, how heavily paid search ads are influencing the results, and how the strong the competition is in each spot on the current search results page.
It’s exceedingly easy as an SEO team to spin your wheels trying to optimize a term into which your business will never be able to break. Difficulty scores quantify that potential opportunity versus risk proposition into one number, which is why we spent a lot of time in this roundup testing how effectively each tool represented keyword difficulty.
For example, let’s say the keyword difficulty of a particular term is in the 80s and 90s in the top five spots on a particular search results page. Then, in positions 6-9, the difficulty scores drop down into the 50s and 60s. Using that difficulty score, a business can begin targeting that range of spots and running competitive analysis on the pages to see who your website could knock out of their spot.
It All Comes Back to Google
The caveat in all of this is that, in one way or another, most of the data and the rules governing what ranks and what doesn’t (often on a week-to-week basis) comes from Google. If you know where to find and how to use the free and freemium tools Google provides under the surface—AdWords, Google Analytics, and Google Search Console being the big three—you can do all of this manually. Much of the data the ongoing position monitoring, keyword research, and crawler tools provide is extracted in one form or another from Google itself. Doing it yourself is a disjointed, meticulous process, but you can piece together all the SEO data you need to come up with an optimization strategy should you be so inclined.
As Google tweaks its algorithms, tools, and search engine policies, so goes the SEO industry. The AdWords API is still the foundation of much of the data with which SEO tools work. Google used to make much of its ad hoc keyword search functionality available as well, but now the Keyword Planner is behind a pay-wall in AdWords as a premium feature. Difficulty scores are inspired by the way Google calculates its Competition Score metric in AdWords, though most vendors calculate difficulty using PA and DA numbers correlated with search engine positions, without AdWords data blended in at all. Search Volume is a different matter, and is almost always directly lifted from AdWords. Not to mention keyword suggestions and related keywords data, which in many tools come from Google’s Suggest and Autocomplete APIs.
An SEO expert could probably use a combination of AdWords for the initial data, Google Search Console for website monitoring, and Google Analytics for internal website data. Then transform and analyze it using all a BI tool. The problem for most business users is that’s simply not an effective use of time and resources. These tools exist to take the manual data gathering and granular, piecemeal detective work out of SEO. It’s about making a process that’s core to modern business success more easily accessible to someone who isn’t an SEO consultant or expert.
Where the free Google tools can provide complementary value is in fact-checking. If you’re checking out more than one of these SEO tools, you’ll quickly realize this isn’t an exact science. If you were to look at the PA, DA, and keyword difficulty scores across KWFinder.com, Moz, SpyFu, SEMrush, Ahrefs, AWR Cloud, and Searchmetrics for the same set of keywords, you might get different numbers across each metric separated by anywhere from a few points to dozens. If your business is unsure about an optimization campaign on a particular keyword, you can cross-check with data straight from a free AdWords account and Search Console. Another trick: enable Incognito mode in your browser along with an extension like the free Moz Toolbar and you can run case-by-case searches on specific keywords to get an organic look at your target search results page.
The SEO tools in this roundup provide tremendous digital marketing value for businesses, but it’s important not to forget that we’re living in Google’s world under Google’s constantly evolving rules. Oh and don’t forget to check the tracking data on Bing now and again, either. Google’s the king but the latest ComScore numbers have Bing market share sitting at 23 percent.
New Vectors: Mobile, Voice Search, and AI
Over the past year or two, we’ve also seen Google begin to fundamentally alter how its search algorithm works. Google, as with many of the tech giants, has begun to bill itself as an artificial intelligence (AI) and machine learning (ML) company rather than as a search company. In essence, Google is changing what it considers its crown jewels. As the company builds ML into its entire product stack, its core search product has begun to behave a lot differently. This is heating up the cat-and-mouse game of SEO and sending the industry chasing after Google once again.
The introduction of new search vectors such as Featured Snippets, Quick Answers, and mobile-optimized formats such as Accelerated Mobile Pages(AMP) are indicative of the way search is evolving around three key factors: mobile search, predictive recommendations, and voice-activated queries from smart home devices and virtual assistants.
For traditional SEO, this has meant some loss of key real estate. For SERP results pages that once had 10 positions, it’s not uncommon now to see seven organic search results below a Featured Snippet or Quick Answer box. Rather than relying on PageRank algorithm for a specific keyword, Google search queries rely increasingly on ML algorithms and the Google Knowledge Graph to trigger a Quick Answer or pull a description into a snippet atop the SERP.
SEO platforms are leaning into this shift by emphasizing mobile-specific analytics. What desktop and mobile show you for the same search results is now different. Mobile results will often pull key information into mobile-optimized “rich cards,” while on desktop you’ll see snippets. SEMrush splits its desktop and mobile indexes, actually providing thumbnails of each page of search results depending on the device, and other vendors including Moz are beginning to do the same.
On the voice and natural language side, it’s all about FAQs (frequently asked questions). Virtual assistants and smart home devices have made voice recognition and natural language processing (NLP) not only desirable but an expected search vector. To predict how to surface a business’ results in a voice search, SEO professionals now need to concentrate on ranking for the common NL queries around target keywords. Google’s Quick Answers exist to give its traditional text-based search results an easy NL component to pull from when Google Assistant is answering questions.
This is also where you can see Google’s ML algorithms at work. Powered by Google Cloud Platform Learn More at Google, the way Quick Answers and Featured Snippets are extracted gets increasingly smarter as Google introduces new innovations in deep learning and neural networks. These constantly evolving algorithms are baked into how the search engine surfaces information.
All of this plays into a new way businesses and SEO professionals need to think when approaching what keywords to target and what SERP positions to chase. The enterprise SEO platforms are beginning to do this but the next step in SEO is full-blown content recommendation engines and predictive analytics. By using all of the data you pull from your various SEO tools, Google Search Console, and keyword and trend data from social listening platforms, you can optimize for a given keyword or query before Google does it first. If your keyword research uncovers a high-value keyword or SERP for which Google has not yet monetized the page with a Quick Answer or a Featured Snippet, then pounce on that opportunity.
How to Choose Your SEO Tool Suite
When it comes to finally choosing the SEO tools that suit your organization’s needs, the decision comes back to that concept of gaining tangible ground. It’s about discerning which tools provide the most effective combination of keyword-driven SEO investigation capabilities, and then on top of that, the added keyword organization, analysis, recommendations, and other useful functionality to take action on the SEO insights you uncover. If a product is telling you what optimizations need to be made to your website, does it then provide technology to help you make those improvements?
The self-service keyword research tools we tested all handle pricing relatively similarly, pricing by month with discounts for annual billing with most SMB-focused plans ranging in the $50-$200 per month range. Depending on how your business plans to use the tools, the way particular products delineate pricing might make more sense. KWFinder.com is the cheapest of the bunch but it’s focused squarely on ad hoc keyword and Google SERP queries, which is why the product sets quotas for keyword lookups per 24 hours at different tiers. Moz and Ahrefs price by campaigns or projects, meaning the number of websites you’re tracking in the dashboard. Most of the tools also cap the number of keyword reports you can run per day. SpyFu prices a bit differently, providing unlimited data access and results but capping the number of sales leads and domain contacts.
For the purposes of our testing, we standardized keyword queries across the five tools. To test the primary ad hoc keyword search capability with each tool, we ran queries on an identical set of keywords. From there we tested not only the kinds of data and metrics the tool gave, but how it handled keyword management and organization, and what kind of optimization recommendations and suggestions the tool provided.
Enterprise SEO platforms put all of this together—high-volume keyword monitoring with premium features like landing page alignments and optimization recommendations, plus on-demand crawling and ongoing position monitoring—but they’re priced by custom quote. While the top-tier platforms give you features like in-depth keyword expansion and list management, and bells and whistles like SEO recommendations in the form of automated to-do lists, SMBs can’t afford to drop thousands of dollars per month.
Ultimately, we awarded Editors’ Choices to three tools: Moz Pro, SpyFu, and AWR Cloud. Moz Pro is the best overall SEO platform of the bunch, with comprehensive tooling across keyword research, position monitoring, and crawling on top of industry-leading metrics incorporated by many of the other tools in this roundup. SpyFu is the tool with the best user experience (UX) for non-SEO experts and the deepest array of ROI metrics as well as SEO lead management for an integrated digital sales and marketing team.
AWR Cloud, our third Editors’ Choice, is rated slightly lower than Moz Pro and SpyFu as an all-in-one SEO platform. Yet breaking down our SEO tool categories further, it leads the pack in ongoing position monitoring and proactive search rank tracking on top of solid overall functionality. On the ad hoc keyword research front, KWFinder.com tool. DeepCrawl’s laser focus on comprehensive domain scanning is unmatched for site crawling, while Ahrefs and Majetic can duke it out for the best internet-wide crawling index. When it comes to backlinks tracking, LinkResearchTools and Majestic are the top choices. SEMrush and Searchmetrics do a bit of everything.
When it comes down to it, you want to choose a platform or invest in complementary tools that provide a single unified SEO workflow. It begins with keyword research to target optimal keywords and SERP positions for your business, along with SEO recommendations to help your rank. Those recommendations feed naturally into crawing tools, which should give you insight into your website and competitors’ sites to then optimize for those targeted opportunities. Once you’re ranking on those keywords, vigilant monitoring and rank tracking should help maintain your positions and grow your lead on competitors when it comes to the search positions that matter to your organziation’s bottom line. Finally, the best tools also tie those key search positions directly to ROI with easy-to-understand metrics, and feed your SEO deliverables and goals right back into your digital marketing strategy.
The tools we tested in this round of reviews were judged on which do the best job of giving you the research-driven investigation tools to identify SEO opportunities ripe for growth, along with offering enterprise-grade functionality at a reasonable price. Whether one of these optimization tools is an ideal fit for your business, or you end up combining more than one for a potent SEO tool suite, this roundup will help you decide what makes the most sense for you. There’s a wealth of data out there to give your business an edge and boost pages higher and higher in key search results. Make sure you’ve got the right SEO tools in place to seize the opportunities.
Source: PCMag Australia