As you might have noticed, I haven’t blogged here since August. Well, that was for a very good reason. This year I founded a new search marketing agency called Screaming Frog and have been working on a new onsite SEO tool, which has prompted this post.

It has been a little while in development, but I am pleased to announce our first SEO tool for release, the Screaming Frog SEO Spider. I am a big fan of the Xenu Link Sleuth tool for finding broken links, redirects and other site issues, but wanted to build a site spider with more emphasis on SEO. I also wanted something that was usable for all SEOs that you can point and click at anytime and didn’t have to log into an interface. The ability to export everything to Excel was a must as I often like to map out SEO recommendations in this format.

What Is The Screaming Frog SEO Spider?

The Screaming Frog SEO Spider is Java based program which spiders websites links, images, CSS, script and apps. It also fetches key onsite page elements for SEO, presents them in tabs by type and allows you to filter for common SEO issues, or slice and dice the data how you see fit by exporting and importing into Excel. You can view, analyse and filter the information as it’s gathered and updated continuously in the programs user interface.

The Screaming Frog SEO Spider allows you to quickly analyse or review a site from an onsite SEO perspective. It’s particulary good for analysing large sites where manually checking every page would be extremely labour intensive (or impossible!) and where you can easily miss a redirect, meta refresh or duplicate page issue.

The spider allows you to export key onsite SEO elements (url, page title, meta descriptions, headings etc) to Excel so it can easily be used as a base to make SEO recommendations from. The following video should give you a better idea -

What Information Does The Screaming Frog SEO Spider Report?

A quick summary of some of the data collected -

  • Errors – Client & server errors (4XX, 5XX)
  • Redirects – (3XX, permanent or temporary)
  • External Links – All followed links and their subsequent status codes
  • URI Issues – Non ASCII characters, underscores, uppercase characters, dynamic uris, over 115 characters
  • Duplicate Pages – Hash value / MD5checksums lookup for duplicate pages
  • Page Title – Missing, duplicate, over 70 characters, same as h1, multiple
  • Meta Description – Missing, duplicate, over 156 characters, multiple
  • Meta Keyword – Mainly for reference as it’s only (barely) used by Yahoo the last time I checked. Missing, duplicate, multiple
  • H1 – Missing, duplicate, over 70 characters, multiple
  • H2 – Missing, duplicate, over 70 characters, multiple
  • Meta Robots – Index, noindex, follow, nofollow, noarchive, nosnippet, noodp, noydir etc
  • Meta Refresh – Including target page and time delay
  • Canonical link element
  • File Size
  • Page depth level
  • Inlinks – All pages linking to a URI
  • Outlinks – All pages a URI links out to
  • Anchor Text – All link text. Alt text from images with links
  • Follow & Nofollow – At link level (true/false)
  • Images – All URIs with the image link & all images from a given page. Images over 100kb, missing alt text, alt text over 100 characters

Enough Reading? Download Now

The lite version of the tool is completely free and allows you to crawl up to 500 URIs. The configuration options for crawling and excluding of file paths are not included.

A license for £99 per annum will remove the 500 URI crawl limit and allow access to the spider options menu.

What Is Next For The Screaming Frog SEO Tool?

The tool is a very early version and we have some big plans over the coming months with many new features in the pipeline. If there is anything specific you think we should add, please do leave the feedback below, send an e-mail or tweet it to me @screamingfrog. Although, looking at the list of features to add, it might well already be on there!

Keep Updated

Keep updated with future release of the tool by checking for updates in the interface, subscribing to the PPC Blog RSS, signing up to our e-mail above or following me on Twitter @screamingfrog.