Monday, December 28, 2009

How to get yourself delisted from search engines

With all this talk about NewsCorp delisting from Google early in 2010, I thought I'd get my thoughts straight on how it is possible to do this:

There are two main ways to ensure that your site doesn't appear in search engines:
  1. Use a lot of underhand Search Engine Opimisation techniques
    This 'blackhat' activity will get your website delisted quickly and with a chance your URL may never reappear in the listings!

  2. Tell the search engine spiders not to visit
    What? Yes, you can tell all the major search engines that you don't want all or part of your site spidered & indexed. Its called a robots.txt file and it sits in the root directory of your website.
    Note: This isn't an exactly new thing done by Google, Robots.txt was devised in 1994... two years before Google was even incorporated.

So why, if Mr Murdoch has been threatening to delist his properties from every search engine (apart from those he wants to do a deal with) why hasn't he just got his Tech people to make this simple change and instantly get removed?
Note: This is even something Google took some time earlier in the year to point out how to do.

No comments: