I guess most of us maintain a test or sandbox blog to try out new plugins and do experiments which we can’t afford on main production blog. To ensure Google and other crawlers/bots don’t crawl your test/sandbox blog, this robots.txt file can be used:
User-agent: * Disallow: /
This would work fine from web search crawler perspective. But in case you display adsense ads on your blog, those would also be blocked. Google crawls each web page as “Mediapartners-Google” bot to decide what type of ads to serve on the page. If test site is blocked for all crawlers, then it would not serve ads on the test blog site. One option is to use this robots.txt to allow it for just Mediapartners-Google.
User-agent: * Disallow: / User-agent: Mediapartners-Google Disallow:
Now why would someone want to serve Google ads on a test site? It is to see how the whole site looks to user before you propagate a change to your production server.
Also note that the order of entries does not matter in robots.txt. Most specific entry takes precedence over generic one.
