This is something I have been meaning to discuss for a longtime and is not something that I have really seen anyone talk about much.

I noticed this quite sometime back when Google started crawling and indexing new blog pages within minutes and ranking this content very quickly in their search results.

Crawling and indexing content so quickly was developed to support their query deserves freshness (QDF) algorithm which enables Google to show hot new content almost immediately for the freshest and most up to date search results.

However what I noticed is that while blog pages can be indexed and ranking within minutes, a couple of days later the blog post is nowhere to be seen anymore. It disappears. The page is still in Googles cache, but when you either search for the url or the exact page title the post will no longer come back under search.

So I performed a little test with my last post to highlight this phenomenon.

  • 17th March – The ‘Linkfromdomain Command – Much underused‘ post was written and published. It was subsequently crawled and indexed that day and showing in the SERPs for url and blog post title searches.
  • 18th March – Page still showing in SERPs when searching for url and blog post title.
  • 19th March – Page still showing in SERPs when searching for url and blog post title.
  • 20th March – Page cannot be found in SERPs when searching for url or blog post title. The page is still in Google cache though.
  • 21st March – Page cannot be found in SERPs when searching for url or blog post title.
  • 22nd March – Page cannot be found in SERPs when searching for url or blog post title.
  • 23rd March – Page can now be found again in the SERPs when searching for url and blog post title.

The results above show the page was showing in the SERPs for around 3 days before disappearing for around 3 days. Looking at the crawling behaviour of Googlebot it visited the page on the 17th 4 times, before visiting again on the 18th and has since not returned.

So Why Is This Happening?

From my experience this is certainly not this blog alone where this happens although it’s hard to say for certain whether this happens to every blog. Is it something to do with domain authority or links? I have seen content both trusted and new sites with both few and lots of links fall inline with this behaviour. I have seen this on at least half a dozen other sites for example. Does this happen with news articles which are crawled and indexed quickly in the same way? Possibly.

My theory is that Google either takes time to migrate the minty fresh index content into their usual index or after the initial bump in the QDF algo Google takes time to re-evaulate & intergrate the page into their usual algorithm before showing it in the SERPS again. I would be interested to hear Matts comments.

Is This Important?

It’s frustrating to have content disappear from the results obviously as you can lose traffic. But especially if you want to take advantage of Googles QDF algo or have content in Googles index at a certain date, you need to strategically time your publishing of posts with this in mind. Or you could find that the planned content you wanted to rank will not show just when you need it!