robots.txt
Website and technology
Website and technology
robots.txt is a file in your site's root telling crawlers which parts they may and may not fetch.
It's an instruction, not a lock: well-behaved crawlers respect it, malicious bots don't. You usually reference your sitemap in it too.
Blocking a page in robots.txt is not the same as removing it from the index. To keep something out of results, use a noindex instruction and let the crawler in.
Tip
Check your robots.txt right after every launch, so a 'Disallow: /' accidentally carried over from staging gets fixed quickly.
Related terms
- CrawlingCrawling is a search engine fetching your pages by following links. No crawl means no index, and no index means no ranking.
- SitemapAn XML sitemap is a list of the URLs you want indexed, meant for search engines.
- llms.txtllms.txt is a proposed text file in your site's root with a concise overview of your key pages, aimed at language models.