The Site Scanner in the Robots Generator section helps you to read the structure of your site and to select directories which you want to be locked for the selected agents.
To use it, enter an URL into the field URL to scan. As a next step, decide if you would like to have only HTML files and if you would like to have only internal links. From the list, choose what link HTML tags you want to be used for the scanning. Select All to activate or deactivate all possibilities. Click Scan to start the scanning process.
The Site Content section shows you the results of your scan. By clicking on a folder, you can see its content displayed in the File name area. Double-click on an item or drag and drop it into the Disallow list.
Note: You have to select an agent in the list in order to add items to the Disallow-List.
If you select the option Clear content on new project, the site content is automatically deleted if you start a new project.