More News International sites block aggregator

Following on from my post last Friday explaining how News International started blocking the news aggregation site www.newsnow.com, word is now out that other Murdoch-owned sites are following suit.

As revealed yesterday by PaidContent, The Sun and News Of The World were planning to block the search spider of NewsNow. However this has already started and The Sun's site has already implemented this change (Hat tip to Malcolm Coles for checking the robots.txt file on thesun.co.uk and getting proof)

There is still no reason given by NewsCorp on why NewsNow has been singled out for this treatment. Perhaps this is just the proof-of-concept before more search engines are excluded from their sites. NewsCorp may be looking at the effects on their organic search results and traffic as a consequence of excluding just one player in the news aggregation space......

News International starts blocking search engine spiders....

.... but only the ones it can afford to set its lawyers on.

Today we have had a further development in the ongoing battle between news aggregators and Rupert Murdoch's News International company. In fact its one that aggregator NewsNow has felt strongly enough to issue a Press Release about today.

For those who aren't familiar with NewsNow's service, it spiders news sites in a similar way to other search sites such a Yahoo and Google News. It then builds links to these articles from its own site, thereby sending traffic off to the content it finds (helping them build visitor traffic, which sells more adverts, which makes the sites more money).

News International has previously stated that it will delist itself from search engines in 2010. We're all watching for them to make themselves invisible to the search spiders of Google and others (but not Bing.com, assuming News International and Microsoft get into bed together as rumoured).

So as I've stated before, the easy way to do this is to make changes to a simple file called a robots.txt which sits on the server displaying the content. If you use this file to tell the spider not to index the specific site it won't, its as simple as that. You can even go so far as to tell the spider (e.g. Google's Googlebot) that you specifically don't want it to look through all or part of your site.
I have done this in the past when I've had a particularly fragile site that could not take all its pages being indexed at once and some site owners do this when they don't want their content read & stolen... which is fair enough.

So what have The Time Online done? Well, if you look at their robots.txt file you will see that they have entered the line:

#Agent Specific Disallowed Sections
User-agent: NewsNow
Disallow: /

Now for those who don't speak fluent search engine spider, I'll translate...

"Dear NewsNow spider, go away"

I'll leave you to decide if The Times are playing fair in this matter. However NewsNow's boss Struan Bartlett is fairly clear on the matter:

“The question remains whether News International, in arbitrarily blocking individual search engines, is trying to use its muscle to gain unreasonable control over the public’s freedom to choose the way they access information and news online."

I personally want to know why The Times has not done the same to block the Googlebot. Could it be it brings in far too much reveue for them right now?

The battle continues!

NewsNow gives in to NLA demands

Information is just surfacing that NewsNow (as mentioned in a previous posting of mine) has bent to the demands of the Newspaper Licensing Authority and is to pull all links to a load of national newspaper sites from its paid for/ B2B area on its site..

Struan Bartlett from NewsNow.co.uk stated:

“We have worked extremely hard to seek clarification from the NLA and its solicitors on the legal basis for either NewsNow or our customers requiring a licence. I am sorry to say that the NLA has not substantiated the legal basis for its licence. Indeed in our view its arguments do not hold water. We believe that other organisations who privately agree with our position have reluctantly signed the NLA agreement under pressure. However, we are not in a position on our own to fund an extremely costly legal case on behalf of an entire industry.

So... rather than fight a costly legal battle against the NLA, they will imminently remove the links from their paid-for part of their site.

The full news release is here.

£10,000 a year plus £155 per B2B user

That's the fee that the Newspaper Licensing Agency want to start charging NewsNow... (for now).
"I don't think it is being unduly greedy to suggest that some of this comes back to us on behalf of the organisations creating the content"

states Andrew Hughes NLA's Commercial Director (note the use of "unduly" there)

But what reciprocal value does the NLA place on the links to its member sites? If only this value could be quantified...

in reference to:

"NLA's commercial director"
- http://econsultancy.com/blog/4902-the-nla-explains-why-it-is-going-after-the-news-aggregators (view on Google Sidewiki)

NewsNow provides more detail about its threats

Struan Bartlett has given us further insight into the threats made by the major (National and Regional) UK newspapers.

In his new open Q&A session (http://www.newsnow.co.uk/press/openletter-qa.html) he states that the newspaper groups are demanding control & money on the grounds of Copyright Law. As Struan is clear to point out, there is no specific UK law that allows news aggregators and intermediaries to do what they do, just as there a no laws to allow you to do a lot of things in business an (most laws say what you CANNOT do).

Perhaps it may be the right time to go to court to resolve this... I just wonder if Google will get dragged into the fight, as there is every indication that the newspaper groups don't want them involved (maybe because they will have a bigger and better legal team?)

in reference to:

"They claim copyright law as a legal justification."
- NewsNow: NewsNow.co.uk Free Linking Q&A (view on Google Sidewiki)