Altova Mailing List Archives

Re: [xml-dev] HGRAB. Syndication. Google. Grey area.

From: Paul T <pault12@-------.--->
To: "K. Ari Krupnikov" <ari@------.--.--.-->
Date: 1/8/2002 11:40:00 PM
> > So, what HGRAB actually does? It polls
> > the HTML pages ( once in a while,
> > no harm done to the load of original
> > website ). Then it places some part of the
> > content into HGRAB database (for future
> > searching). Then it provides the end-user
> > with some 'part' of the original news item
> > and with the URL to the original news source.
> > 
> > Google does *exactly* this (and also Google
> > provides a cached copy of the original content)
> > 
> > That means:
> > 
> > Either both HGRAB and Google should be sued,
> > because they both sell the content
> > *which does not belong to them*, or both
> > HGRAB and Google should be considered
> > 'just a service'.
> Have a look at

I don't understand your point. Could you pelase 

So, some websites may have robots.txt, 
saying that some pages should not be 
indexed by robots.

Because HGRAB, for example, is 
usually polling only home page of the website, 
they are all allowed for polling.

Also, I'm not sure if search engines do 
really care about the robot.txt, but that's another 

Also, the interesting twist is that when the 
robot encounters the website with *no*
robots.txt ( most of the sites have no robots.txt ) 
the robot assumes that it is *safe* for him to 
'steal' the content.

Shouldn't it be 'if there is no robots.txt - 
get out of here'?

I think this is really gray area and 
robots.txt is not a solution. 
At the moment, at least.

What I'm missing?



These Archives are provided for informational purposes only and have been generated directly from the Altova mailing list archive system and are comprised of the lists set forth on Therefore, Altova does not warrant or guarantee the accuracy, reliability, completeness, usefulness, non-infringement of intellectual property rights, or quality of any content on the Altova Mailing List Archive(s), regardless of who originates that content. You expressly understand and agree that you bear all risks associated with using or relying on that content. Altova will not be liable or responsible in any way for any content posted including, but not limited to, any errors or omissions in content, or for any losses or damage of any kind incurred as a result of the use of or reliance on any content. This disclaimer and limitation on liability is in addition to the disclaimers and limitations contained in the Website Terms of Use and elsewhere on the site.