Crawl budget

Crawl budget is the number of pages a web crawler will visit on your site within a given time period. Both traditional search engine crawlers and AI crawlers operate under crawl budget constraints, making it essential to ensure your most important content is discovered and indexed.

How crawl budget works

Crawl budget is determined by two factors:

Crawl capacity: How many requests the crawler can make without overloading your server
Crawl demand: How much the search engine or AI system wants to crawl your content based on perceived value, freshness, and popularity

Why crawl budget matters for AI visibility

If AI crawlers exhaust their crawl budget on low-value pages, they may never discover your most important content:

Product pages with key brand information may go uncrawled
Blog posts with original research and statistics may be missed
Landing pages optimized for AI citations may not be indexed

Managing crawl budget for AI search

Prioritize valuable content

Use robots.txt to block crawling of low-value pages (admin panels, duplicate content, paginated archives)
Submit an XML sitemap highlighting your most important content
Use llms.txt to guide AI crawlers to key pages

Improve crawl efficiency

Fix broken links and redirect chains that waste crawl budget
Reduce duplicate content and pagination issues
Ensure fast server response times (slow sites consume more crawl budget)
Use proper canonical tags to consolidate crawl signals

Monitor crawl activity

Check server logs to see which AI crawlers visit and how frequently
Monitor crawl rate for GPTBot, ClaudeBot, PerplexityBot, and others
Identify pages that are crawled frequently but provide little value

Crawl budget and site size

Crawl budget is primarily a concern for larger sites (10,000+ pages). Smaller sites are typically crawled completely. However, even smaller sites should ensure AI crawlers can efficiently access their most important content.