Crawling
SEO and AI visibility
SEO and AI visibility
Crawling is a search engine fetching your pages by following links. No crawl means no index, and no index means no ranking.
A crawler starts from known URLs — your sitemap, links on your own site, links elsewhere — and follows on from there. How much it fetches per visit depends on your site's size, speed and reliability.
Pages nothing links to (orphan pages) are often found late or not at all. Internal links aren't only for visitors.
Related terms
- IndexingIndexing is a crawled page being stored in the search engine's database. Only indexed pages can appear in results.
- robots.txtrobots.txt is a file in your site's root telling crawlers which parts they may and may not fetch.
- SitemapAn XML sitemap is a list of the URLs you want indexed, meant for search engines.
- Technical SEOTechnical SEO is the part of SEO about findability and readability for search engines: crawling, indexing, speed, structure and error handling.