robots.txt disallow all example

Sometime we need to block all robots from crawling a web site. This can be needed if you have a stage or sandbox website for development purpose.

robots.txt disallow all

Here is the robots.txt you can use to block all robots from crawling a site:

User-agent: *
Disallow: /

robots.txt disallow all except Mediapartners-Google

Sometimes we beed to test Google adsense on stage/sandboxbox site. Google crawls a site as Mediapartners-Google to be able to display ads. Here is example robots.txt which disallows all bots except Mediapartners-Google

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Disallow:

Note that when Disallow: has not value, it is equivalent of allowing everything for that bot.

Share this article: share on Google+ share on facebook share on linkedin tweet this submit to reddit

Comments

Click here to write/view comments