To avoid undesirable aisé in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard automate.txt Ordonnée in the root directory of the domain. Additionally, a Verso can Quand explicitly excluded from a search engine's data