Screen scraping: not dead, just renamed

A few years back I blogged about the act of screen scraping and why it was wrong to do this.

Since that time I've seen clients (primarily eCommerce ones) develop feeds for:

• Affiliates



• Google product search



• 3rd party applications (e.g. Mobile apps)



• and other such things

This however means that a decent website may quickly end up developing a number of different feeds that all do different things. It can therefore create a mesh of XML files & API's that can become quite complex to maintain and manage.

One approach is to create a single feed that is then used for everything, perhaps going via a marketing agency, who can then reformat it for different purposes. However this can turn out to be a pretty bulky file (e.g. if you have a large catalogue, this can quickly become several megabytes in size) or can contain details that you might not wish all parties to have (e.g. links to your hi-res images from your Content Delivery Network that you may be paying by the megabyte for).

So I was reasonably interested in this article from eConsultancy that seemed to address this very issue. Had they really found a decent solution to this problem? One that I think will only get worse over time as the needs of eCommerce sites grow.....

Well the answer lies in this part of the posting:

Next-generation data feed solutions allow feeds to be generated and deployed
quickly and at low cost by extracting the ‘front end’ product-related HTML code
from the website, with no requirement for any ‘back end’ data – or expertise on
the part of the merchant. By harvesting elements such as pricing, availability
and product attributes directly from the merchant’s website, it is possible to
ensure that the extracted data feed is comprehensive and accurate

So let me get that straight. This 'next generation' method doesn't use an actual data feed from the site owner. It works by 'harvesting elements' from the HTML of the merchant's site without their actual involvement.

And that's not screen-scraping how exactly?

Why newspapers block Google

Newspapers are stopping their sites being indexed by Google in an effort to preserve their 'unique content'.

Yes, that's correct. They believe they have the monopoly on the facts they find & report and don't want the 'Big G' to find it, display it and link to them.

So why are they doing this? Well, they believe that a hyperlink from Google's news site isn't the same value as a user finding the content themselves..... http://thenoisychannel.com/2009/04/18/is-differentiated-content-enough-to-save-newspapers/

Note:
The arguement about what constitutes 'screen-scraping' and what is proper 'spidering & indexing' gets a little blurred here. But its not as if Google passes this information off as its own, instead it links to the newspaper site (albeit it does put up relevant paid adverts and make money from this)

Preventing screen scraping

Following my recent posts on screen scraping, I've had various email discussions on the prevention of screen scraping. In other words how can sites prevent users or automated scripts taking their data, without punishing real users at the same time?

CAPTCHA's (Completely Automated Public Turing Test to Tell Computers and Humans Apart) are used to stop automated scripts pretending to be real people. Before anyone comments.. yes I am aware that CAPTCHAS can be broken/beaten, even ones by Google & Yahoo!

Rendering is a newer approach that is being used to defeat scripts/bots and is being used online e.g. by financial institutions, coupon and ticketing industries. By creating an image of the data a site can display the information which is easily readable by a user, but much harder by a machine....

Why screen scraping is wrong

I've recieved a few emails following my recent post about screen scraping. Its my belief that nearly all websites that publish large amounts of data/information suffer from this at some point, so its not suprising its turned out to be a fairly popular subject.

Here's my thought about why sites need to stop this activity:

• Any user that steals your content, your hard work, your thoughts or the information you own without your permission is commiting theft. If you are charging access (subscription model) or have revenue based on views (advertising model) then this is affecting your ability to make revenue. Some markets (e.g. travel) are affected more. Blockscraping.com say:

  ... it's not uncommon that over seven percent of the ticket sales of a low fare
  air line comes from scraper sites.

• Once this content is saved somewhere else beyond your control, it can be out of date. Anyone accessing the 'scraped' information' and then coming to your site may get different information and therefore be dissappointed.


• The scraping of content (particularly automated scraping by scripts & robots) adds additional load to your website. This may affect other users (e.g. the site may be slower for them) or cost you more (bandwidth, hosting, etc.).

Screen Scraping

Following an evening in the company of an old colleague and friend John G, we discussed the pros & cons of screen scraping. "What's that then Hayden?" I hear various people asking.

Screen scraping is the process of electronically grabbing content from an interface designed for human viewing. In the pre-web days it was used as a way of getting displayed system information from terminals. Now it generally refers to the technique of grabbing the HTML on a web page and inserting that content into a file or database for subsequent use.

So, what are the pros?
Well, by running a screen scraping routine, you can obtain data from a website that you would either have to manually copy & paste to another source(e.g. a spreadsheet). This routine could be automated to run at a particular time (e.g. just after it was updated at midnight) and may save you having to integrate with the site directly or paying the site owner for an export of the content you need.

But what are the cons?
Well, firstly its rather under-hand. Yes, it is just automating a manual process you may-well be doing anyway, but the question should be raised as to why you need to obtain lots of information from the original source in this way (and presumably without their permission)? The terms & conditions of many sites will prohibit you from doing this, especially if you have to register / pay for browsing premium information that you then want to scrape. It should also be noted that you are obtaining information from a website in a known layout/code structure... any change to that code will mean your routine will not work (and some websites deliberately do this for that very reason).
In addition, some sites will be very quick to notice screen scraping, especially if it is likely to affect their revenue or purpose. Using network techniques they could then block your access and counter your efforts.


To quote (without his permission) Eric Raymond of the Jargon File:
http://catb.org/~esr/jargon/html/S/screen-scraping.html

screen-scraping is an ugly, ad-hoc, last-resort technique that is very likely to break on even minor changes to the format of the data being snooped.