Robots.txt Generator tool!
Now let’s see what kind of instructions you can give to robots through your robots.txt file.
If you want all robots to access everything on your website, then your robots.txt file should look like this:
Basically, the robots.txt file here disallows nothing, or in other words, is allowing everything to be crawled. The asterisk next to “User-agent” means that the instruction below applies to all types of robots.
On the other hand, if you don’t want robots to access anything, simply add the forward slash symbol like this:
Note that one extra character can render the instruction ineffective, so be careful when editing your robots.txt file.
In case you want to block access to a specific type of GoogleBots, like those that search for images, you can write this:
Or if you want to block access to a certain type of files, like PDFs, write this:
# Disallowed File Types
If you want to block access to a directory within your website, for example, the admin directory, write this:
If you want to block a specific page, simply type its URL:
And if you don’t want Google to index a page, add this instruction:
If you’re not sure what indexing means, it’s simply the process that makes a page part of web searches.
Lastly, for big websites that are frequently updated with new content, it’s possible to set up a delay timer to prevent servers from being overloaded with crawlers coming to check for new content. In a case like this, you could add the following directive:
Thus all robots (except for Googlebots, which ignore this request) will delay their crawling by 120 seconds, preventing many robots from hitting your server too quickly.