Robots.txt AI Bot Checker
Check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and 8 more AI crawlers can access any domain. Results in seconds.
Enter a domain
We fetch /robots.txt directly from the site — no login, no rate limits for reasonable use.
Parse the rules
Each of 12 known AI crawlers is checked against specific blocks and wildcard rules.
See the results
Clear table showing which bots are allowed, partially blocked, or fully blocked.
Frequently asked questions
Why would I want to block AI crawlers?▾
Some publishers block AI crawlers to prevent their content from training models they aren't compensated for. Others keep them allowed because being cited in AI answers drives branded visibility and traffic. There's no single right answer — it depends on your content strategy.
What's the difference between GPTBot and Google-Extended?▾
GPTBot is OpenAI's training crawler for ChatGPT. Google-Extended is Google's opt-out flag specifically for Gemini/Vertex AI training — it doesn't affect regular Googlebot indexing. Blocking Google-Extended won't hurt SEO rankings.
If a bot isn't in my robots.txt, is it allowed?▾
Yes. If robots.txt doesn't mention a specific user agent and has no `User-agent: *` block, that bot is implicitly allowed. Only explicit `Disallow` directives block crawling.
Does blocking AI bots hurt my SEO?▾
No — regular search bots (Googlebot, Bingbot) are separate from AI training bots. Blocking GPTBot won't affect your Google search rankings. It only affects whether your content is used to train/serve AI models.
How do I block a specific AI bot?▾
Add this to your robots.txt: User-agent: GPTBot Disallow: / Replace `GPTBot` with any user agent name. Place it at the top level of your domain at `/robots.txt`.
Want to know what AI engines actually say about you?
The bot checker tells you what's allowed. TurboAudit tells you whether AI engines actually mention you — and what you can do about it. 5 free audits, no credit card.