Robots.txt Tester, Check Any URL Against Any Bot

A robots.txt file can quietly block pages nobody meant to hide, or accidentally allow a crawler that was supposed to be restricted, and both mistakes are easy to make since the rules for multiple bots often live in the same file with subtly different, overlapping patterns. This tool tests a specific URL path against robots.txt rules for any named crawler, showing clearly whether that path is allowed or blocked and exactly which rule decided the outcome, catching a misconfiguration before it costs real indexing.

How a crawler actually picks which rules apply to it

A robots.txt file can contain multiple groups, each starting with one or more User-agent lines followed by Allow and Disallow rules. When a crawler reads the file, it looks specifically for a group matching its own exact user-agent name, GPTBot looks for a group naming GPTBot specifically. If no exact match exists, it falls back to the wildcard group, User-agent: *, which applies to any crawler not named individually elsewhere in the file. This tool follows that exact same selection logic, so testing against a specific bot name shows you precisely which group of rules that bot would actually use, not just whatever rules happen to be listed first in the file.

Why the longest matching rule wins, not the first one

A common misunderstanding is assuming robots.txt rules are read top to bottom, with the first matching rule deciding the outcome. That is not how the specification actually works: when multiple rules within a group match a given path, the most specific rule, the one with the longest matching pattern, takes priority, regardless of which order the rules appear in the file. A Disallow: / combined with a more specific Allow: /public/ means /public/ pages are genuinely allowed, even though the broad disallow rule appears first, because the longer, more specific allow rule wins. This tool implements that exact longest-match logic, along with the standard tie-breaking rule that Allow wins over Disallow when two rules of equal length both match.

Wildcards and end-of-string anchors

Beyond simple prefix matching, robots.txt patterns support two special characters widely adopted by major crawlers. An asterisk, *, matches any sequence of characters, useful for a rule like Disallow: /*.pdf to block every PDF regardless of what directory it sits in. A dollar sign, $, anchors the end of the pattern to the end of the URL, so Disallow: /*.pdf$ blocks URLs ending exactly in .pdf, without accidentally also blocking a URL like /report.pdf.html that merely contains that substring partway through. This tool's matching logic correctly interprets both of these special characters exactly as major crawlers do.

Pasting your robots.txt content, live URL fetching not yet supported

This tool works by pasting your robots.txt content directly, rather than fetching it live from a domain. Fetching an arbitrary external URL directly from a browser tab runs into the same cross origin restriction, CORS, that blocks most client side tools from reading another domain's files without that domain explicitly permitting it, robots.txt files are not generally served with permissions allowing this. Live URL fetching is on the roadmap behind a small server side proxy, for now, copying your robots.txt content directly from your site, or viewing it at yourdomain.com/robots.txt and pasting it in, gets the same accurate result.

Runs entirely in your browser

Parsing and rule matching happen entirely client side, nothing you paste is sent to a server. If you specifically want to check access for AI crawlers like GPTBot and ClaudeBot across your whole file at once rather than testing one path and bot at a time, our AI Crawler Access Checker is built specifically for that broader view, part of the same Technical SEO Tools collection this tool belongs to.