Although the basic layout is similar, the new design has a lot more space between each result, which will (in my opinion at least) mean an increased CTR on the very top positions.

You can see a full size screenshoot here.

I personally think this makes the organic results look pretty cluttered and I found them harder to read.

I captured a screenshot of a query for ‘flowers’ here in Google.co.uk. Notice how the words ‘Flower’ and ‘Flowers’ are capitalised in display URLs. Like serenataFlowers.com, Flowersdirect.co.uk, asda-Flowers.co.uk etc.

Almost immediately after I ran this query, Google stopped the test and I couldn’t replicate.

It has been a little while in development, but I am pleased to announce our first SEO tool for release, the Screaming Frog SEO Spider. I am a big fan of the Xenu Link Sleuth tool for finding broken links, redirects and other site issues, but wanted to build a site spider with more emphasis on SEO. I also wanted something that was usable for all SEOs that you can point and click at anytime and didn’t have to log into an interface. The ability to export everything to Excel was a must as I often like to map out SEO recommendations in this format.

What Is The Screaming Frog SEO Spider?

The Screaming Frog SEO Spider is Java based program which spiders websites links, images, CSS, script and apps. It also fetches key onsite page elements for SEO, presents them in tabs by type and allows you to filter for common SEO issues, or slice and dice the data how you see fit by exporting and importing into Excel. You can view, analyse and filter the information as it’s gathered and updated continuously in the programs user interface.

The Screaming Frog SEO Spider allows you to quickly analyse or review a site from an onsite SEO perspective. It’s particulary good for analysing large sites where manually checking every page would be extremely labour intensive (or impossible!) and where you can easily miss a redirect, meta refresh or duplicate page issue.

The spider allows you to export key onsite SEO elements (url, page title, meta descriptions, headings etc) to Excel so it can easily be used as a base to make SEO recommendations from. The following video should give you a better idea -

What Information Does The Screaming Frog SEO Spider Report?

A quick summary of some of the data collected -

• Errors – Client & server errors (4XX, 5XX)
• Redirects – (3XX, permanent or temporary)
• External Links – All followed links and their subsequent status codes
• URI Issues – Non ASCII characters, underscores, uppercase characters, dynamic uris, over 115 characters
• Duplicate Pages – Hash value / MD5checksums lookup for duplicate pages
• Page Title – Missing, duplicate, over 70 characters, same as h1, multiple
• Meta Description – Missing, duplicate, over 156 characters, multiple
• Meta Keyword – Mainly for reference as it’s only (barely) used by Yahoo the last time I checked. Missing, duplicate, multiple
• H1 – Missing, duplicate, over 70 characters, multiple
• H2 – Missing, duplicate, over 70 characters, multiple
• Meta Robots – Index, noindex, follow, nofollow, noarchive, nosnippet, noodp, noydir etc
• Meta Refresh – Including target page and time delay
• Canonical link element
• File Size
• Page depth level
• Inlinks – All pages linking to a URI
• Outlinks – All pages a URI links out to
• Anchor Text – All link text. Alt text from images with links
• Follow & Nofollow – At link level (true/false)
• Images – All URIs with the image link & all images from a given page. Images over 100kb, missing alt text, alt text over 100 characters

Enough Reading? Download Now

The lite version of the tool is completely free and allows you to crawl up to 500 URIs. The configuration options for crawling and excluding of file paths are not included.

A license for £99 per annum will remove the 500 URI crawl limit and allow access to the spider options menu.

What Is Next For The Screaming Frog SEO Tool?

The tool is a very early version and we have some big plans over the coming months with many new features in the pipeline. If there is anything specific you think we should add, please do leave the feedback below, send an e-mail or tweet it to me @screamingfrog. Although, looking at the list of features to add, it might well already be on there!

Keep Updated

Keep updated with future release of the tool by checking for updates in the interface, subscribing to the PPC Blog RSS, signing up to our e-mail above or following me on Twitter @screamingfrog.

However, there is still a geo problem in the Google UK SERPs which is more noticable on the longer tail queries (as you might expect). Higher authority domains that don’t target the UK, still manage to overcome the geo filtering and outrank UK targeted websites with more relevant content which from a user experience perspective is simply put, poor.

As SEOs we find ourselves running search queries a lot for clients, but my experience today was as a normal user wanting to buy a product. I am after a Diesel man bag to replace the one I managed to break recently. Hence, I searched for ‘Diesel messenger bags’ -

So lets look at the results that Google.co.uk gave me (without personalisation) -

1. Ebay.co.uk – Ok, cool. But I want to buy brand new and don’t fancy browsing Ebay to look through reputable dealers.
2. About.com – Not really after a content farm.
3. Zappos.com – Doesn’t deliver to the UK!
4. Ebay.com – US Ebay site. I just decided not to buy from the UK site so…
5. Bizrate.com – Don’t really want to buy from a shopping comparison site but I’ll take a look. Oh wait there, it’s the US version anyway.
6. Nextag.com – Same as above.
7. Shopstyle.com – It has handbags in the url and is the US shop…
8. Askmen.com – A blog post, no help as I want to buy, buy, buy!
9. Shopping.Yahoo.com – Diesel handbags for the US market. Great.
10. Rushfaster.com.au – An Aussie site. Got to have an .com.au ranking in the top 10 of Google UK SERPS these days.
11. Ebay.co.uk – Oh FFS, Ebay and their bloody subdomains.
12. About.com – The content farm AGAIN….!!
13. Purseblog.com – AHHH another blog.
14. Thebaglady.tv – At least this snippet is telling me I can buy them at ASOS but its just a product review with aff link!
15. Nextag.com – Seriously, again?
16. Amazon.co.uk – WA-HEY! Finally a UK site that sells Diesel messenger bags! Oh, it’s out of stock though.

Now at this point you feel like shooting yourself in the face right?

Try and refrain from doing that and run the search again. This time, click on the PPC results. I found what I was looking for with the 1st click. The Adwords team must be loving the revenue right now.

There has been a fair amount of movement for the search query ‘search engine optimisation‘ in the Google UK SERPs recently. I’ll let you perform your own analysis and work out what’s going on if you haven’t already.

But what can we learn from the state of the results?

1) The power of exact match domains – searchengineoptimisation.org has been ranking consistently in 3rd. searchengineoptimization.co.uk the US spelling is 2nd page. The only real question here, is why Click Consult are not utilising searchengineoptimisation.co.uk which is parked.

2) Link networks, link buying, sponsoring website themes (for wordpress etc) and utilising your own clients to rank for SEO all still work.

3) Hiding links (from users) using CSS is difficult for the search engines to discover.

4) There is still far to much weight placed on anchor text and it makes the SERPS easy to manipulate.

5) SEO hasn’t moved on as far as many believe.

Anyway, I don’t normally call out sites for link buying, but I thought this might be a fun one as it’s a search engine. Yahoo seem to be text link advertising on http://www.iagora.com/. From the sites advertising page -

Advertisers on iAgora range from travel sites (i.e. Yahoo Travel) to recruiters who are looking for candidates with international profiles (ie Cisco, P&G), international universities (ie EBS, RBS, ESC Reims) or telecommunications companies (ie Phonevalley).

Now obviously we all know, that when advertising, particularly when you use a keyword rich anchor text like ‘flights’ a link condom should be used.

Surely Yahoo are not paying for a toolbar PR7 link for SEO purposes? :-)  (Yes these links will pass PR and anchor text even with the onClick event).

I particularly like the bingo advert above it which makes it look even more spammy.

Simply checking the UK agencies ranking for ‘SEO’ (as an example) it’s very easy to spot that some of them are using their own clients as a link network to rank for that very term. I am not going to call anyone out or make examples of anyone, they know who they are.

Presumably by doing it they think its fine, so perhaps I can give examples.

Historically web design and development agencies have always left a signature in the form of a hyperlink on client’s sites often in the footer to show who produced the site. This for me is not a problem at all, but is it right for search marketing agencies often acting merely as consultants to do the same?

Do any other types of consultant do this? No. What benefit is this to the client? None. Does it do them any harm? Mmm.

Is It Ethical?

While I am sure most clients are aware of the ‘SEO’ hyperlinks, I know for a fact that some clients are either not aware, or do not fully understand the implications of having the link.

I have also seen cases where the links are hidden extremely well, or hidden completely in a few cases which obviously makes this a little darker as it goes against the search engines guidelines.

Perhaps these clients have been given code to insert on their site as part of their SEO service (a scrolling text box…) with the keyword rich anchor & hyperlink included, perhaps it’s a requirement of the contract or even incentivised as an option for reduced fees, or they have been told that linking out to reputable sources will ‘help them rank’.

I also know some of these agencies hard sell their SEO services off the back of their ranking for the said term to justify to potential clients why they should work with them over the competition. Obviously ranking for ‘SEO’ by using their clients as a link network does not say anything about their actual skills as an SEO agency, but they often pass it off that it does.

There are definitely varying levels of ethicality in this, I believe some agencies are very open about it while some are less so and I am not pointing fingers at anyone in particular. But it does make you think, if some are willing to use their own clients as a link network for no benefit to the client, what does it say about them?

In an industry that is so often criticised for lacking credibility & transparency, does this not simply support these assumptions?

Perhaps some view this as simply a form of advertising similar to web design agency signatures, a clever technique to outrank the competition that is no more than any other type of ‘partner’ (if reciprocal) or supplier link or the incentivised links at least could be seen as no more than a form of paid links you see everyday.

Perhaps some would argue that despite all the criticisms above and questionable ethical nature, if it helps them rank for their desired term it’s clearly worth it in the long-term regardless.

My personal opinion is that everyone involved owe it to their own industry to keep standards as high as possible and in particular for their own clients.

Using your own clients as a link network for your own benefit no matter how you pass it off, to me, is taking advantage of your position. It is of no benefit to the client and it sends out the wrong message about our industry. I certainly wouldn’t recommend anyone work with an agency that uses this technique regardless of how good they might actually be in reality.

I noticed this quite sometime back when Google started crawling and indexing new blog pages within minutes and ranking this content very quickly in their search results.

Crawling and indexing content so quickly was developed to support their query deserves freshness (QDF) algorithm which enables Google to show hot new content almost immediately for the freshest and most up to date search results.

However what I noticed is that while blog pages can be indexed and ranking within minutes, a couple of days later the blog post is nowhere to be seen anymore. It disappears. The page is still in Googles cache, but when you either search for the url or the exact page title the post will no longer come back under search.

So I performed a little test with my last post to highlight this phenomenon.

• 17th March – The ‘Linkfromdomain Command – Much underused‘ post was written and published. It was subsequently crawled and indexed that day and showing in the SERPs for url and blog post title searches.
• 18th March – Page still showing in SERPs when searching for url and blog post title.
• 19th March – Page still showing in SERPs when searching for url and blog post title.

• 20th March – Page cannot be found in SERPs when searching for url or blog post title. The page is still in Google cache though.
• 21st March – Page cannot be found in SERPs when searching for url or blog post title.
• 22nd March – Page cannot be found in SERPs when searching for url or blog post title.

• 23rd March – Page can now be found again in the SERPs when searching for url and blog post title.

The results above show the page was showing in the SERPs for around 3 days before disappearing for around 3 days. Looking at the crawling behaviour of Googlebot it visited the page on the 17th 4 times, before visiting again on the 18th and has since not returned.

So Why Is This Happening?

From my experience this is certainly not this blog alone where this happens although it’s hard to say for certain whether this happens to every blog. Is it something to do with domain authority or links? I have seen content both trusted and new sites with both few and lots of links fall inline with this behaviour. I have seen this on at least half a dozen other sites for example. Does this happen with news articles which are crawled and indexed quickly in the same way? Possibly.

My theory is that Google either takes time to migrate the minty fresh index content into their usual index or after the initial bump in the QDF algo Google takes time to re-evaulate & intergrate the page into their usual algorithm before showing it in the SERPS again. I would be interested to hear Matts comments.

Is This Important?

It’s frustrating to have content disappear from the results obviously as you can lose traffic. But especially if you want to take advantage of Googles QDF algo or have content in Googles index at a certain date, you need to strategically time your publishing of posts with this in mind. Or you could find that the planned content you wanted to rank will not show just when you need it!

1) Check Your Outbound Links – You can’t always control who links to you, but you can control who you link out to which means it’s a fantastic indicator of trust to the search engines. We have seen time and time again when sites start linking out to poor neighbourhoods or spam (hacked sites or the DP network as extreme examples) your rankings die. So use the linkfromdomain command to see if you are linking out to anything you shouldn’t be.

You can use the excellent free Xenu link sleuth for this but with the linkfromdomain query you can add the usual spam queries to the end to narrow your search down.

For example for this blog with viagra. They all seem fine, although Yahoo still has some of it’s old hacked pages in the cache.

If you want to find out exactly what page you are linking to the site from use this command on Yahoo.

2) Identify Authority Hubs – Ok, so we believe that Googles trust algo was built around an initial hand picking of ‘tier 1′ authoritative trusted sites initially. The more links you receive from sites like these and the ‘closer’ you are to these sites, the more authority and trust you will have in the search engines eyes. It can be extremely difficult to get links from these sites, so identify who they link out to and get links there.

For example, the BBC. Then you can narrow your query down, for example if you wanted to find out who the BBC links out to with content around ‘flowers‘.

3) A little Naughty – How about as a way to find old domains with established pagerank / links? Using the BBC as an example again, how about expired domains or domains for sale which you can look to pick up. But obviously you can identify the authority sites in your niche or sector and analyse those sites.

So are you still using linkfromdomain?