robots.txt

The robots.txt is a text file that is located in the root directory of a website. In it, the behavior of the crawlers / bots is regulated.

Notice: The robots.txt is only a recommendation for the crawler. With the included rules it is not possible to protect directories from unwanted access. Malicious crawlers can access the content without problems despite robots.txt.

A standard robots.txt file in WordPress looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent:

• defines which crawlers are addressed. An * addresses all crawlers. If, for example, you only want to address the Google Bot you can do so with User-Agent: google
  This addresses all bots that start with google*.

Disallow:

• defines which directories should not be accessed or crawled. This automatically includes all subdirectories. If you want to open a subdirectory to the bot, you can do so with Allow as described below.
• In our experience, after a CMS change Google may well try to crawl subdirectories of the old CMS. Since these no longer exist, the bot can simply be restricted from them.

Allow:

• In the example, the admin-ajax.php is released in the “wp-admin” directory, as the entire directory with all files and subdirectories was previously locked.

We target websites specifically to search terms. Learn more at SEO support.

Interested in other topics? Here we have a few more suitable explanations.