AI Crawler Access Checker (GPTBot, ClaudeBot & More)

GPTBot, ClaudeBot, PerplexityBot, and Google-Extended each respect robots.txt independently, each looking for its own specific user-agent name, and a site can easily end up blocking one of them without meaning to, especially after a robots.txt edit made for an entirely different reason. This tool checks a robots.txt file against all four crawlers at once for a given path, showing allowed or blocked status side by side, useful for anyone actively deciding whether their content should be accessible to AI systems.

Why one wildcard rule doesn't cover every AI crawler

A common assumption is that a single User-agent: * rule in robots.txt governs every crawler visiting a site, including AI crawlers. In practice, any crawler with its own explicitly named group in the file follows that specific group instead of the wildcard fallback, so a site with User-agent: * Disallow: /private/ and a separate User-agent: GPTBot Disallow: / group has effectively blocked GPTBot from the entire site, not just the /private/ folder the wildcard rule targets. Checking each AI crawler's actual effective rule, not just assuming the wildcard applies uniformly, is exactly the gap this tool closes.

What each of these four crawlers actually is

GPTBot is OpenAI's crawler, primarily associated with gathering content to help improve their models. ClaudeBot is Anthropic's equivalent crawler. PerplexityBot supports Perplexity's AI-powered search and answer engine, which is somewhat distinct from a pure training crawler since it can also fetch content to help answer a live user query. Google-Extended is a specific control introduced by Google, separate from the standard Googlebot used for search indexing, that lets a site opt out of having its content used for Google's AI training and Gemini-related features specifically, without affecting standard Google Search indexing at all.

Blocking an AI crawler is a real, separate decision from blocking search crawlers

Because Google-Extended, GPTBot, and similar crawlers are specifically separated from standard search indexing crawlers, blocking one does not affect a site's visibility in regular search results, these are genuinely independent, deliberate settings. Whether to allow AI crawlers is ultimately a judgment call weighing potential visibility in AI-powered answers and chat interfaces against a site's own content usage preferences, this tool reports the current effective access status accurately, it does not make that policy decision for you.

Checking a specific path, not just the site root

Access can be granted broadly but restricted for specific sections, or vice versa, a site might allow AI crawlers generally but explicitly disallow a paywalled or private content directory. Testing a specific path rather than only the root confirms the actual effective rule for the exact content you care about, which section of a real robots.txt file, applying the same longest-match rule-selection logic real crawlers use, decides the outcome for that specific path and crawler combination.

Runs entirely in your browser

All parsing and rule matching happen entirely client side, nothing you paste is sent to a server. If you want to test a path against a specific crawler not on this fixed list, or explore the full rule set for any single bot in more detail, our Robots.txt Tester supports testing any named user-agent individually, part of the same Technical SEO Tools collection this tool belongs to.