AI crawler robots.txt checker
Paste a URL. We fetch its robots.txt and report whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot can crawl it. Free, no signup, no card.
If GPTBot is disallowed, you are invisible in ChatGPT.
AI answer engines respect robots.txt the same way Google does. A copy-pasted privacy-focused robots.txt is the most common reason small business sites never get cited in modern search.
Citation eligibility
ChatGPT, Claude, and Perplexity will not cite a page they cannot fetch. The verdict is binary.
Training-time recognition
GPTBot and CCBot feed future model generations. Blocking them means future AI never learns your brand exists.
Real-time fetches
ChatGPT-User and Claude-Web fetch pages live when a user pastes a URL. Blocking returns 'I cannot access that page' to your prospect.
Google AI surfaces
Google-Extended controls Gemini and AI Overviews. Blocking removes you from Google's most visible AI surface.
Set the policy once
A few lines in robots.txt cover every major AI crawler. The crawlability guide has the exact rules to copy.
Re-check after deploys
The fix is one line. The regression is also one line. Re-run this checker after any infrastructure change.
Want the exact rules to allow every major AI crawler? Read the crawlability guide.
Frequently asked questions
- Why does it matter if AI bots can crawl my site?
- If GPTBot, ClaudeBot, or PerplexityBot is blocked, your site will never be cited in ChatGPT, Claude, or Perplexity answers — even when your content is the right answer to the user's question. Most marketing-oriented sites should allow them by default.
- What if I do not have a robots.txt file?
- No robots.txt is equivalent to allowing all crawlers (per RFC 9309). We will report that as 'allowed — no robots.txt'. If you want explicit control, add a robots.txt that names each AI bot and uses Allow or Disallow rules.
- Which bots do you check?
- GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, Claude-Web (Anthropic); PerplexityBot (Perplexity); Google-Extended (Google's AI surfaces); CCBot (Common Crawl). Each is fetched against the rules in your robots.txt, including matching to wildcard groups.
- Does this run a full audit?
- No. This is a single-purpose check on robots.txt only. For an end-to-end audit covering technical SEO, content clarity, structured data, and the full AI readiness lens, use the main audit on the home page.
robots.txt is one check. Run all 60+.
Free audit covers crawlability, technical SEO, metadata, content clarity, AI readability, entities, schema, and internal linking — all eight lenses, one report.
Run free audit