Sometime we need to block all robots from crawling a web site. This can be needed if you have a stage or sandbox website for development purpose.
robots.txt disallow all
Here is the robots.txt you can use to block all robots from crawling a site:
User-agent: * Disallow: /
robots.txt disallow all except Mediapartners-Google
Sometimes we beed to test Google adsense on stage/sandboxbox site. Google crawls a site as Mediapartners-Google to be able to display ads. Here is example robots.txt which disallows all bots except Mediapartners-Google
User-agent: * Disallow: / User-agent: Mediapartners-Google Disallow:
Note that when Disallow: has not value, it is equivalent of allowing everything for that bot.
