# temporarily disallow ALL indexing... Disallow: / # usual robots directives... User-agent: * Disallow: /autofile/ Disallow: /cgibin/ Disallow: /css/ Disallow: /pilot/ Disallow: /traffic/ Disallow: /hterror/ Disallow: /hterrors/ Disallow: /css/ # slow down abusive Yahoo spider... User-agent: Slurp Crawl-delay: 300 # http://info.webcrawler.com/mak/projects/robots/exclusion-admin.html