robots.txt Tutorials and Examples

In case you are writing robots.txt and want to test it for a specific crawler/bot/user-agent, you can use this tool to test it.

By default Disallow entry in robots.txt is based on url prefix. To block every url beginning with /foo/ the following robots.txt can be used. User-agent: read more

Sometime we need to block all robots from crawling a web site. This can be needed if you have a stage or sandbox website for read more

An error in robots.txt can cause Google to stop crawling a section of your site. In such case it is good to update and resubmit read more

I noticed that google bot is crawling javascript and css regularly from by wordpress blog site. Here are some entries from my apache log: 66.249.75.66 read more

In case you want search engines to either not index or not follow links on a page, you can use robot meta tag. The default read more

I guess most of us maintain a test or sandbox blog to try out new plugins and do experiments which we can’t afford on main read more

Here is a small code snippet to demonstrate how to parse robots.txt and crawl a web page in python using robotexclusionrulesparser module url = “http://www.foo.com/bar” read more