robots.txt
Website and technology
Website and technology
robots.txt is a file in your site's root telling crawlers which parts they may and may not fetch.
It's an instruction, not a lock: well-behaved crawlers respect it, malicious bots don't. You usually reference your sitemap in it too.
Blocking a page in robots.txt is not the same as removing it from the index. To keep something out of results, use a noindex instruction and let the crawler in.
In practice
A 'Disallow: /' accidentally carried over from a staging environment blocks your entire site. Check it right after every launch.
Related terms
- CrawlingCrawling is a search engine fetching your pages by following links. No crawl means no index, and no index means no ranking.
- SitemapAn XML sitemap is a list of the URLs you want indexed, meant for search engines.
- llms.txtllms.txt is a proposed text file in your site's root with a concise overview of your key pages, aimed at language models.