Free Robots.txt Generator

Search Engine Optimization

Free Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Free Robots.txt Generator

Robots.txt Generator tool!

 

Now let’s see what kind of instructions you can give to robots through your robots.txt file.

If you want all robots to access everything on your website, then your robots.txt file should look like this:


User-agent: *
Disallow:

Basically, the robots.txt file here disallows nothing, or in other words, is allowing everything to be crawled. The asterisk next to “User-agent” means that the instruction below applies to all types of robots.

On the other hand, if you don’t want robots to access anything, simply add the forward slash symbol like this:


User-agent: *
Disallow: /

Note that one extra character can render the instruction ineffective, so be careful when editing your robots.txt file.

In case you want to block access to a specific type of GoogleBots, like those that search for images, you can write this:


User-agent: googlebot-images
Disallow: /

Or if you want to block access to a certain type of files, like PDFs, write this:


User-agent: *
Allow: /

# Disallowed File Types
Disallow: /*.PDF$

If you want to block access to a directory within your website, for example, the admin directory, write this:


User-agent: *
Disallow: /admin

If you want to block a specific page, simply type its URL:


User-agent: *
Disallow: /page-url

And if you don’t want Google to index a page, add this instruction:


User-agent: *
Noindex: /page-url

If you’re not sure what indexing means, it’s simply the process that makes a page part of web searches.

Lastly, for big websites that are frequently updated with new content, it’s possible to set up a delay timer to prevent servers from being overloaded with crawlers coming to check for new content. In a case like this, you could add the following directive:


User-agent: *
Crawl-delay: 120

Thus all robots (except for Googlebots, which ignore this request) will delay their crawling by 120 seconds, preventing many robots from hitting your server too quickly.