webmasterworld.com/robots.txt excludes all bots
User-agent: *
Disallow: /
Robots are “…are a major drag on the system”, wow, while most other sites try everything to get indexed WBM can turn away bots and still attract thousands of visitors.
But what is more interesting about their robots.txt is what appears after the disallow, a “Bot Blog” in plaintext complete with an advert and copyright notice.
0 Responses to “Innovative use of robots.txt”