The robots.txt is a text file that is located in the root directory of a website. In it, the behavior of the crawlers / bots is regulated.
[su_note note_color=”#f8f8f8″ radius=”0″]Notice: Die robots.txt ist nur eine Empfehlung für den Crawler. Mit den enthaltenen Regeln ist es nicht möglich, Verzeichnisse vor ungewolltem Zugriff zu schützen. Bösartige Crawler können trotz robots.txt ohne Probleme auf die Inhalte zugreifen.[/su_note]
A standard robots.txt file in WordPress looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
User-agent:
- defines which crawlers are addressed. With a * all crawlers are controlled. For example, if you only want to address the Google Bot, you can do this with User-Agent: google
This addresses all bots that start with google*.
Disallow:
- defines which directories are not to be called or crawled. This automatically includes all subdirectories. If you want to release a subdirectory for the bot, this can be done with Allow be made.
- We have made the experience that it can happen during a CMS change that Google wants to search any subdirectories of the old CMS. Since these no longer exist, the bot can simply be restricted for this.
Allow:
- Im Beispiel wird die admin-ajax.php im “wp-admin” Verzeichnis freigegeben, da zuvor das komplette Verzeichnis mit allen Dateien und Unterverzeichnissen gesperrt wurde.
