AI Crawlers & robots.txt: Complete Reference Guide

No — blocking GPTBot only affects whether your content is included in future OpenAI model training. ChatGPT's live search answers (via ChatGPT Search) are powered by OAI-SearchBot, which fetches pages through Bing's index in real time. Many publishers block GPTBot for IP reasons while keeping OAI-SearchBot allowed — this is the correct approach if you want AI citations without contributing to model training.

Yes. robots.txt supports path-specific rules: 'User-agent: GPTBot / Disallow: /private/' blocks GPTBot from /private/ while allowing it on all other pages. You can also use a robots meta tag on specific pages: — though this is less standardized across AI crawlers than the robots.txt approach. For fine-grained control, use path-specific Disallow rules in robots.txt.

Check your server access logs for user-agent strings matching: 'GPTBot', 'OAI-SearchBot', 'PerplexityBot', 'ClaudeBot', 'Google-Extended'. Most hosting dashboards (Cloudflare, Vercel Analytics, Nginx access logs) capture these. You can also use a honeypot URL: create a page not linked from your site, add it only to your sitemap, and see which crawlers find it via the sitemap — AI crawlers that respect sitemaps will show up in logs.

Your pages will not appear in AI-generated answers from ChatGPT, Perplexity, Claude, or Google AI Overviews. Some AI systems (like Google AI Overviews) use Googlebot data — blocking Googlebot also blocks Google Search, which is usually not intended. For most sites, blocking all AI crawlers significantly reduces AI visibility without a meaningful benefit. If IP protection is the concern, blocking training crawlers (GPTBot, Google-Extended, CCBot) while allowing citation crawlers is the right balance.

No — Google-Extended controls AI training data only. Google AI Overviews use standard Googlebot crawl data from Google's existing search index. Blocking Google-Extended does not affect your eligibility for AI Overviews citation. Blocking Googlebot does affect AI Overviews (since Overviews draw from the search index). This is a common confusion: Google-Extended and Googlebot have completely separate purposes.

Blocking CCBot is reasonable if you don't want your content in open training datasets (used by many open-source AI models). It does not affect any major commercial AI citation systems — ChatGPT, Perplexity, Google AI Overviews, and Claude all use their own crawlers. Blocking CCBot has no negative effect on AI search visibility. It's an optional choice based on whether you want to contribute to open-source AI training data.

AI crawlers generally cannot authenticate. Login-gated content (behind a sign-in wall) is not accessible to any AI crawler, regardless of robots.txt settings. This means paywalled content, members-only sections, and private dashboards are not crawled and will never appear in AI citations. If you want AI to cite specific content, it must be publicly accessible. This also means your robots.txt rules for authenticated paths are largely irrelevant — the crawler can't get past the login anyway.

Crawl-delay tells crawlers to wait N seconds between requests. Syntax: 'Crawl-delay: 5' (5 seconds between requests). Google ignores Crawl-delay; Bing and some AI crawlers respect it. Use it if AI crawlers are generating measurable server load — check with your hosting provider. For most sites, AI crawler traffic is negligible and Crawl-delay is unnecessary. Setting a very high Crawl-delay (60+) effectively blocks crawlers without a Disallow.