Indexability & AI Crawl Access: Complete Guide

The robots.txt file tells crawlers which paths they can and cannot access. When a robots.txt rule disallows an AI crawler from your page’s path, that crawler will not request the page at all — your content is completely inaccessible to it. The major AI crawlers you need to explicitly permit are: - **GPTBot** — OpenAI’s primary training and browsing crawler - **OAI-SearchBot** — OpenAI’s real-time search crawler for ChatGPT search - **Claude-SearchBot** (also ClaudeBot) — Anthropic’s web crawler - **PerplexityBot** — Perplexity AI’s crawler - **Google-Extended** — Google’s crawler for Gemini and AI Overviews training - **Amazonbot** — Amazon’s crawler for Alexa and AI product features **BLOCKER condition:** Any rule matching Disallow: / for these crawlers blocks the entire site. More targeted rules (e.g., Disallow: /admin/) are acceptable but must be audited to confirm they do not cover important pages. **Pass condition:** The robots.txt either omits the crawler’s user-agent (meaning it follows the * default, ideally Allow: /) or explicitly sets Allow: / for each named crawler. **Common trap:** Many SEO plugins auto-add protective robots.txt rules that inadvertently block AI crawlers. Vercel, WordPress with Yoast, and some Shopify themes have added blanket Disallow rules for all non-Google crawlers.

Every page you want AI to index must return HTTP status code 200 OK. AI crawlers interpret the HTTP response code the same way Google does — a non-200 status communicates a problem with the page’s availability. **Status code categories:** - **200 OK** — Pass. Page is available and can be crawled. - **301/302** — Redirect. AI crawlers follow a single redirect but lose some indexability signal. - **404 Not Found** — Fail. AI crawler records the page as unavailable. - **410 Gone** — Fail. Explicit signal that page has been permanently removed. - **403 Forbidden** — BLOCKER. Crawler is denied access, content is invisible. - **429 Too Many Requests** — BLOCKER. Crawler is being rate-limited and cannot access the page. - **500/503 Server Error** — BLOCKER. Server failure prevents crawling. **Soft 404s** are a special case: the server returns 200 OK but the page content is an error message. AI crawlers that parse these pages extract error text instead of useful content. **Pass example:** Server returns HTTP/2 200 with the page’s actual content. **Fail example:** Server returns HTTP/2 200 but the page body contains an error message — a soft 404.

The canonical tag (link rel=canonical) tells crawlers which URL is the authoritative version of a page. For AI indexability, the canonical tag must point back to the page’s own URL — a self-referencing canonical. **Three canonical failure modes:** **1. Canonical pointing to a different page.** If /blog/article-a has a canonical pointing to /blog/article-b, the AI crawler treats /blog/article-a as duplicate content. Your original URL gets zero indexability credit. **2. Missing canonical tag.** Without a canonical, AI crawlers must infer the authoritative URL. If your page is accessible via multiple URLs, crawlers may split indexability between URL variants. **3. Duplicate canonical tags.** Two or more canonical link elements in the same head. Crawlers treat this as contradictory signals and may ignore both. **Pass example:** A self-referencing canonical matching the actual page URL — https://yourdomain.com/blog/article-a. **Fail example:** A canonical pointing to a different URL — https://yourdomain.com/blog/article-b — declaring this page a duplicate.

The robots meta tag lives in your page’s head element and provides page-level crawl and indexing directives. Unlike robots.txt (which controls crawler access), the robots meta tag controls what the crawler can do with the page content after accessing it. Two directives are BLOCKERS for AI indexability: **noindex** — Tells crawlers not to include this page in their index. For AI systems, this means the page will not be cited. Intended for staging pages and admin panels — catastrophic on content pages. Usage: meta name=robots content=noindex **nosnippet** — Tells crawlers not to extract content from the page. This effectively prevents citation even if the page is indexed. Usage: meta name=robots content=nosnippet **max-snippet:0** — Sets the allowed snippet length to zero characters, preventing all content extraction. Usage: meta name=robots content=max-snippet:0 **Pass example:** meta name=robots content=index, follow **Common cause of accidental noindex:** CMS preview modes, staging environments where robots tags are copied to production, or A/B testing tools that inject noindex during experiments.