Skip to content
All terms

What is Crawl Budget?

Crawl budget is the number of URLs a search engine will fetch from a site in a given period, determined by the site's authority, health, and demand for its content.

Deep dive: automated site audits

Small sites rarely exhaust crawl budget, but new and low-authority domains experience its cousin: crawl reluctance. Google discovers URLs (via sitemaps and links) but declines to fetch them — surfacing in Search Console as 'Discovered – currently not indexed'. The cure is demand-side, not technical: fewer, deeper pages; internal links from pages Google already values; and external links that justify the crawl.

Wasting budget still hurts at any size: redirect chains, parameter duplicates, thin near-duplicate pages, and orphaned URLs all spend fetches that deeper pages needed.

Related terms

Primary technical references & standards

See how your site scores on this

The free Website Verdict scan measures your SEO and AI visibility in under a minute.

Run a free scan