robots.txt

The robots.txt is a text file that is located in the root directory of a website. In it, the behavior of the crawlers / bots is regulated.

Notice: The robots.txt is only a recommendation for the crawler. With the included rules it is not possible to protect directories from unwanted access. Malicious crawlers can access the content without problems despite robots.txt.

A standard robots.txt file in WordPress looks like this:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent:

• defines which crawlers are addressed. An * controls all crawlers. For example, if you only want the Google Bot address, is this possible with User-Agent: google
  This addresses all bots that start with google*.

Disallow:

• defines which directories are not to be called or crawled. This automatically includes all subdirectories. If you want to release a subdirectory for the bot, this can be done with Allow be made.
• We have made the experience that it can happen during a CMS change that Google wants to search any subdirectories of the old CMS. Since these no longer exist, the bot can simply be restricted for this.

Allow:

• In the example, the admin-ajax.php is released in the “wp-admin” directory, as the entire directory with all files and subdirectories was previously locked.

We target websites specifically to search terms. Learn more at SEO support.

Interested in other topics? Here we have a few more suitable explanations.