Screaming Frog SEO Spider is a program that crawls your website the way Google does. It visits every page, follows every link and gives you an overview of what's technically wrong: broken links, missing titles, duplicate content, redirects, slow or oversized files. It runs on your own computer (Windows, Mac or Linux) and the basic version is free.
Unlike cloud tools that crawl your site on a schedule, you start a crawl yourself whenever you want. That makes it ideal for targeted checks: right before a launch, after a migration, or when you suspect a theme change broke something.
What Screaming Frog finds
| Problem | Why it matters |
|---|---|
| Broken links (404) | Poor experience for visitors and wasted crawl time |
| Redirects and chains | Slower and loss of link value |
| Missing or duplicate titles and descriptions | Fewer clicks from Google and AI answers |
| Duplicate or thin pages | Pages competing with each other |
| Canonicals and noindex | Pages you accidentally exclude |
| Images without alt text or oversized | Accessibility and speed |
| Structured data | Errors in your schema for rich results |
| Orphan pages | Pages no internal link points to |
| Hreflang errors | Wrong language version shown in other countries |
| XML sitemap vs. crawl | Pages in the sitemap that aren't crawlable, or vice versa |
How a crawl actually works
Screaming Frog starts at the URL you enter and follows every link it finds, just like a search engine. It distinguishes internal links (within your domain) from external links (to other sites) and checks the status code for each. You can also point the spider at a list of URLs instead of a full site, for example to quickly check whether a hundred specific pages still exist after a migration.
By default it respects your robots.txt and won't follow links excluded there, just like Google. You can turn that off if you want to see what's behind that block, for example to check you haven't accidentally hidden important pages.
Free versus paid
| Free | Paid licence | |
|---|---|---|
| Maximum URLs per crawl | 500 | Unlimited (depends on your computer) |
| Save and compare crawls | No | Yes |
| JavaScript rendering | No | Yes |
| GA4, Search Console, PageSpeed integration | No | Yes |
| Custom extraction and source code search | No | Yes |
| Scheduled crawls | No | Yes |
| Spelling and grammar check | No | Yes |
The paid licence is per user per year. For a small site of a few hundred pages, free is often enough. For web shops and larger sites, the licence quickly pays for itself, if only to compare crawls: if you suddenly see 40 new 404s after a release, you'll want to know whether it's related.
When to use it
- Before and after a migration. Crawl the old site, build a redirect list, crawl the new one and check everything lines up. See WordPress migration and Shopify migration.
- Periodic check. Once a quarter, hunt down broken links and missing titles, even if nothing else changed: links to external sites expire on their own.
- After a new release or theme. Check that no noindex tags or odd canonicals suddenly appeared. This goes wrong more often than you'd think, for example when a staging setting accidentally goes live.
- For competitor research. Look at another site's structure and titles (respecting their server: keep crawl speed low).
- For content audits. An export of all titles and word counts quickly shows thin pages you could merge or remove.
- For international sites. Check hreflang tags for errors, for example a Dutch page pointing to itself as the English variant.
How to work with it
- Enter your site's URL and click Start.
- Check the key alerts in the Response Codes, Page Titles, Meta Description, Images and Directives tabs.
- Use the Issues tab for a prioritised list with explanations per problem.
- Export to Excel or Google Sheets and split the work across your team or agency.
- Fix what affects visitors and Google most first: broken links on key pages, wrong noindex, redirect chains.
- Run a new crawl after the fixes to check the problems are actually resolved.
A practical example: a web shop after a platform switch
A web shop moves from one platform to another. Before going live, you fully crawl the old site and save all the URLs. After launch, you crawl the new site and compare the lists: which old URLs now return a 404 instead of a redirect? That prevents you losing traffic and rankings on product pages that have built up link value for years.
A practical example: a B2B site with a new theme
A B2B service provider updates the site's theme. After launch it turns out the new theme automatically adds a noindex tag to pages without a featured image. Without a crawl, that would probably only have been noticed weeks later, once traffic had already dropped.
Common mistakes
- Treating everything as equally urgent. A hundred "title slightly too long" alerts matter less than five broken links on your homepage.
- Forgetting JavaScript content isn't seen without rendering. If your site builds pages with JavaScript, the free or non-rendering crawl will see an empty page.
- Unintentionally overloading a busy server. On weak hosting, a fast crawl can slow the site down for real visitors.
- Skipping a pre-crawl before a migration. Without a crawl of the old site, you won't know which URLs need a redirect.
- Ignoring the Issues tab. It automatically prioritises what you should tackle first.
When not to use it
For a five-page site with no history, a full technical crawl is overkill; a quick manual check is enough. Also, if your site runs entirely on a closed platform where you can't adjust technical settings (some simple website builders, for example), the findings are of limited use: you can't fix the problems anyway.
How it works with other tools
Screaming Frog shows what's on your site; Search Console shows what Google does with it and how your site is indexed. You combine them: Screaming Frog might find a page with a noindex tag, and Search Console confirms that page indeed isn't indexed. For structured data, you check the alert in Screaming Frog and test the actual output on the page, see structured data. For site speed, you supplement the technical crawl with a separate check via Google PageSpeed Insights, which goes deeper into load time per page.
Log file analysis as a supplement
Screaming Frog also has a separate Log File Analyser that reads your server's log files and shows which pages Googlebot actually visits, and how often. That's more valuable than a regular crawl for seeing whether Google regularly visits your key pages, or wastes time on pages that don't matter.
Reporting to your team or client
An export full of technical terms often means little to a client. Translate the findings into impact: not "214 pages are missing a meta description" but "214 pages now show a random snippet in Google instead of a compelling description, which costs you clicks". Group by priority (critical, important, nice-to-have) and add a concrete action to each point: who fixes what, and how do you check it's resolved.
Crawl budget and large sites
On sites with thousands of pages, crawl budget matters: Google doesn't spend unlimited time crawling your site. Screaming Frog helps you see which pages waste crawl time, for example endless filtered URLs on a web shop that all show nearly identical content. By blocking those via robots.txt or setting canonicals to the main version, you keep more crawl budget for the pages that actually matter.
Alternatives
Sitebulb (more visual, also desktop), the site audits in Semrush and Ahrefs (cloud-based, part of a larger suite) and the free Ahrefs Webmaster Tools for your own site. Sitebulb is often easier to read if you don't work with technical SEO daily; Screaming Frog is more flexible with custom extraction if you want to know exactly where something sits on the page. Also read our comparison of SEO tools: Semrush, Ahrefs and Mangools.
Frequently asked questions about Screaming Frog
Is Screaming Frog free?
Yes, up to 500 URLs per crawl with basic features. For larger sites and extra features like saving crawls and JavaScript rendering, you buy an annual licence.
Is Screaming Frog difficult?
The basics are manageable: enter a URL, crawl, check the Issues tab. The advanced features, like custom extraction and combining multiple data sources, do require some SEO knowledge.
Does Screaming Frog work on a Mac?
Yes, there are versions for Windows, macOS and Linux.
Can Screaming Frog damage my site?
No, it only reads and doesn't change anything on your site. On a weak server a fast crawl can temporarily slow the site down; lower the crawl speed in the settings if needed.
What's the difference with Google Search Console?
Screaming Frog crawls your site itself and shows what's technically there, even before Google has looked at it. Search Console shows how Google actually indexes your site and what you're found for.
Can I crawl a competitor with Screaming Frog?
Technically yes, but crawl at a low speed so you don't strain their server, and only use it to look at structure and titles, not to copy content.
Do I need Screaming Frog if I already use Search Console?
Search Console shows what Google already knows, but misses things like a full overview of all internal links or a quick check right after a change. Screaming Frog fills that gap with a direct, on-demand crawl.
Does Screaming Frog work on a site that isn't live yet?
Yes, as long as the staging environment is reachable by the crawler (and not blocked by a password or IP restriction) you can point it there too.
Further reading
Want to know more about being found in Google and AI answers? Read indexing, structured data and Google PageSpeed Insights. Want to look at your site together? Book a free strategy session.
Questions, or just want to spar?
We're happy to think along — call, email or drop by in the heart of Eindhoven.