Robots Element Editor

In the Robots Element Editor you can put together all the elements you need for your robots.txt file.

As Element Type, you can select

  • Robots Exclusion: Use this element to exclude specific or all robots from certain directories of your website.

  • Sitemap: Use this element to enter the path where your sitemap is located, so that robots can easily find it.

  • Comment: Add your own comments here which will be ignored by the robots.

When you select Robots Exclusion, you can add robots and Disallow paths by clicking the according button on the right.

To add robots, you can select either all robots by selecting the * character or specific ones from the drop down list. For adding disallow paths, simply enter the directory which you want to exclude.

Of course, you can also edit or delete elements from the list.

In the end, a Robots exclusion might for example look like this:

User-agent: *

Disallow: /temp/

This would mean that all robots are not allowed to index the /temp/ directory of your website.

To specify the location of your Sitemap, enter its URL into the according field and optionally a comment as well.

To add any personal comments about the whole robots.txt, select the element type Comments and enter your text.