('binary' encoding is not supported, stored as-is) In-Reply-To: <5.1.0.14.0.20020706205941.00ba75c8at_private> >From: Octavio / Super <alvarezpat_private> > >Google (or any other robot/search engine) might as well ignore robots.txt. I think not to trust these. > Keep an eye on your logs: I've always noticed Google droping the hunt after the first hit it got was the robots.txt file. There are other search robots that are not quite as polite though, they get blocked. Skot.
This archive was generated by hypermail 2b30 : Mon Jul 08 2002 - 10:04:30 PDT