ClaudeBot
Anthropic's web crawler — indexes content for Claude's knowledge base and retrieval systems. Also identified by the user-agent string anthropic-ai.
RE: ClaudeBot // ZYGUR KNOWLEDGE BASE
KEY FACTS
- —Two user-agent strings to allow: ClaudeBot/1.0 and anthropic-ai
- —Crawls sites referenced in Claude conversations for live retrieval
- —Respects robots.txt Disallow rules fully
- —Allowing ClaudeBot is required for your business to appear in Claude recommendations
- —Anthropic publishes IP ranges for ClaudeBot at anthropic.com
- —ClaudeBot activity increased 340% in 2025 alongside Claude's web search rollout
FREQUENTLY ASKED
What is ClaudeBot?
ClaudeBot is Anthropic's web crawler that indexes your site for use in Claude AI recommendations. It uses two user-agent strings: ClaudeBot and anthropic-ai. Both should be allowed in robots.txt.
How do I allow ClaudeBot in robots.txt?
Add: User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: /
Does Claude use real-time web search?
Yes. Claude has web retrieval capability that uses ClaudeBot to fetch current content. Sites that allow ClaudeBot and have strong AI visibility signals are more likely to be cited.
RELATED TERMS
ZYGUR TECHNOLOGIES CORP. · AI SEARCH OPTIMIZATION KNOWLEDGE BASE · ZYGUR.COM/KB
FREE TOOL
Check Your AI Visibility Score — Free in 60 Seconds.
See exactly how you score on claudebot and four other AI visibility factors. Average score is 23/100.
SCORE MY SITE →