Domainers at Tiki Central? - Beyond Tiki

FG Futura Girl Member Joined: Oct 01, 2002 Posts: 933	FG Futura Girl Posted posted on Thu, Dec 2, 2004 3:03 PM On 2004-12-02 11:50, Tangaroa wrote:You can place a robots.txt file in your code that instructs them not to archive your old pages. See more info here: http://www.archive.org/about/exclude.php the only problem with that is once you tell 'em not to crawl, it removes ALL the pages from your website for now and forever. They will not selectively remove page(s). I found this out the hard way - a couple of years ago they crawled a page on the modcom site that had been erroneously linked that included all sorts of private names and phone numbers . well, i had to get it off there asap - but they had to remove modcom from all future crawls, too... that's no good when you're trying to get as many links and hits as possible to build your webby kingdom.

*On 2004-12-02 11:50, Tangaroa wrote:*You can place a robots.txt file in your code that instructs them not to archive your old pages. See more info here: http://www.archive.org/about/exclude.php

the only problem with that is once you tell 'em not to crawl, it removes ALL the pages from your website for now and forever. They will not selectively remove page(s).

I found this out the hard way - a couple of years ago they crawled a page on the modcom site that had been erroneously linked that included all sorts of private names and phone numbers . well, i had to get it off there asap - but they had to remove modcom from all future crawls, too... that's no good when you're trying to get as many links and hits as possible to build your webby kingdom.

Post #128550 by Futura Girl on Thu, Dec 2, 2004 3:03 PM