InfoHeap
Tech tutorials, tips, tools and more
Navigation
  • Home
  • Tutorials
    • CSS tutorials & examples
    • CSS properties
    • Javascript cookbook
    • Linux/Unix Command Line
    • Mac
    • PHP
      • PHP functions online
      • PHP regex
    • WordPress
  • Online Tools
    • Text utilities
    • Online Lint Tools
search

robots.txt

  • Online robots.txt check
  • wildcard
  • disallow all
  • resubmit robots.txt to Google
  • Googlebot and js, css crawl
  • robot noindex, follow for wordpress
  • Allow Google adsense bot
  • parse robots.txt using python
 
  • Home
  • > Tutorials
  • > Web Development
  • > SEO
  • > robots.txt

robots.txt disallow all example

By admin on Jan 27, 2016

Sometime we need to block all robots from crawling a web site. This can be needed if you have a stage or sandbox website for development purpose.

robots.txt disallow all

Here is the robots.txt you can use to block all robots from crawling a site:

User-agent: *
Disallow: /

robots.txt disallow all except Mediapartners-Google

Sometimes we beed to test Google adsense on stage/sandboxbox site. Google crawls a site as Mediapartners-Google to be able to display ads. Here is example robots.txt which disallows all bots except Mediapartners-Google

User-agent: *
Disallow: /

User-agent: Mediapartners-Google
Disallow:

Note that when Disallow: has not value, it is equivalent of allowing everything for that bot.

Suggested posts:

  1. How to use wildcard in robots.txt
  2. How to prevent your test blog from google crawling but show adsense ads
  3. Should I block Googlebot from crawling javascript and css?
  4. How to resubmit robots.txt to Google
  5. Using robotexclusionrulesparser python module to parse robots.txt and crawl a site
  6. How to undo HTTP 301 site/domain redirect
  7. How to automate wordpress sandbox setup on Linux
  8. Using python to analyze bots from apache logs
Share this article: share on facebook share on linkedin tweet this submit to reddit
Posted in Tutorials | Tagged robots.txt, Tutorials, Web Development, Webmaster

Follow InfoHeap

facebook
twitter
googleplus
  • Browse site
  • Article Topics
  • Article archives
  • Recent Articles
  • Contact Us
  • Omoney
Popular Topics: Android Development | AngularJS | Apache | AWS and EC2 | Bash shell scripting | Chrome developer tools | CSS | CSS cookbook | CSS properties | CSS Pseudo Classes | CSS selectors | CSS3 | CSS3 flexbox | Devops | Git | HTML | HTML5 | Java | Javascript | Javascript cookbook | Javascript DOM | jQuery | Kubernetes | Linux | Linux/Unix Command Line | Mac | Mac Command Line | Mysql | Networking | Node.js | Online Tools | PHP | PHP cookbook | PHP Regex | Python | Python array | Python cookbook | SEO | Site Performance | SSH | Ubuntu Linux | Web Development | Webmaster | Wordpress | Wordpress customization | Wordpress How To | Wordpress Mysql Queries

Copyright © 2023 InfoHeap.

Powered by WordPress