Robots.txt AI Bot Checker

Robots.txt is a plain-text file at the root of your domain (https://yourdomain.com/robots.txt) that tells web crawlers which parts of your site they may access. It uses two directives: User-agent (which crawler) and Disallow / Allow (what they can fetch). Originally defined in 1994 (the Robots Exclusion Protocol), it was formalized as an IETF standard in 2022 (RFC 9309). Robots.txt is advisory — it's a request, not enforcement. Well-behaved crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot) respect it. Malicious crawlers ignore it.

GPTBot is OpenAI's training crawler — it fetches pages to improve future OpenAI models. OAI-SearchBot is OpenAI's live retrieval crawler for ChatGPT Search — it fetches pages in real time to answer user queries. ChatGPT-User is the user-fetched browsing agent — triggered when a ChatGPT user asks ChatGPT to read a specific URL. They're three separate crawlers with different purposes. Block GPTBot to opt out of training; allow OAI-SearchBot and ChatGPT-User to remain visible in ChatGPT answers. Source: platform.openai.com/docs/bots.

Anthropic operates three crawlers, mirroring OpenAI's pattern. ClaudeBot is training; Claude-User is user-fetched browsing inside Claude.ai; Claude-SearchBot is live search retrieval added through 2024-2025 as Claude expanded web search capability. They're listed at support.anthropic.com/en/articles/8896518. To be cited in Claude answers, allow Claude-SearchBot and Claude-User. If you want to opt out of training only, block ClaudeBot but allow the other two.

Three legitimate reasons. (1) Compensation: your content is the product (paywalled news, premium research, syndication), and you don't want AI vendors training on it without paying — the NYT, Reuters, and BBC pattern. (2) Quality control: low-quality crawlers (Bytespider has been flagged for aggressive scraping) can degrade site performance or scrape at undesirable volumes. (3) Brand control: in highly regulated industries (healthcare, finance), you may not want AI-paraphrased versions of your content circulating without your context. For most B2B SaaS and agency sites, blocking AI bots is a net loss because it eliminates AI citation traffic.

No. Googlebot (search indexing) and Google-Extended (Gemini AI training) are separate. Blocking Google-Extended does not affect your position in Google search results — Google has stated this explicitly in their crawler docs. Same logic for GPTBot (separate from Googlebot), ClaudeBot, and PerplexityBot — they have nothing to do with traditional search ranking. The only SEO-relevant crawlers are Googlebot, Bingbot, and Yandex-bot (depending on your market). AI crawlers affect AI citation visibility, not search ranking.

Yes, by default. The Robots Exclusion Protocol treats absence as permission. If robots.txt doesn't mention a User-agent AND has no `User-agent: *` Disallow block matching the requested path, the bot is implicitly allowed to crawl. Only explicit Disallow directives block crawling. This means if you want to block GPTBot, you must add an explicit `User-agent: GPTBot` + `Disallow: /` block — relying on `User-agent: *` works but is overbroad (it blocks everyone).

Add this to your robots.txt: User-agent: GPTBot Disallow: / Replace `GPTBot` with the exact User-agent name (case-sensitive in some implementations — use the exact casing from the vendor's docs). Place the file at the root of your domain at https://yourdomain.com/robots.txt. You can stack multiple blocks: User-agent: GPTBot User-agent: ClaudeBot User-agent: Bytespider Disallow: / Use our tool above to verify the rules took effect.

Use targeted Allow + Disallow combinations: User-agent: GPTBot Allow: /docs/ Allow: /blog/ Disallow: / This allows GPTBot to crawl /docs/ and /blog/ but blocks everything else. Useful for sites that want their public content indexed by AI but want to protect customer dashboards, paywalled archives, or pricing pages.

Robots.txt is access control — tells crawlers what they may fetch. Llms.txt is content curation — tells LLMs which content matters most (emerging spec; partial adoption). Sitemap.xml is URL discovery — lists every page worth crawling. They're complementary. Most well-configured sites have all three. Use our /tools/sitemap-checker for sitemap.xml validation and /tools/llms-txt-generator to create an llms.txt.

Whenever you change site structure, launch a new section, or want to update AI bot policy. AI vendors add new crawlers periodically (Anthropic added Claude-SearchBot through 2024-2025; Apple added Applebot-Extended in 2024). Check vendor docs every 3-6 months. Also re-test with our tool whenever you change robots.txt — typos in User-agent names are the #1 source of accidentally-public content.