[WikiEN-l] Broken link on your website

Mon Sep 29 21:48:09 UTC 2003

At 09:22 AM 9/30/03 +1200, Richard Grevers wrote:
>On Mon, 29 Sep 2003 11:07:26 -0700,  Brion Vibber <brion at pobox.com> gave 
>utterance to the following:
>
>>If one of the list admins sent this through; please don't. It's *spam*.
>>
>>They crawl the web for e-mail addresses and then send these things out
>>as advertisements for their link-checking / availability service. Which,
>>of course, means they're crawling our web site and loading it down.
>>*Thanks* so much.
>I wonder if their crawler respects robots.txt?

I would assume not--robots.txt is a protocol for honest people, not
spammers and liars.
-- 
Vicki Rosenzweig
vr at redbird.org
http://www.redbird.org

Aug	SEP	Oct
	22
2003	2004	2005