HTTP Header & Status Code Checker

Checking a URL's response headers, status code, and redirect behavior usually means opening browser dev tools, and checking a whole batch of URLs that way means repeating that process one page at a time. This tool checks many URLs at once, follows each one's full redirect chain automatically, and flags the specific headers that actually matter for SEO and security, all with the option to check as a specific search engine crawler rather than a regular browser.

Why checking as Googlebot specifically matters

Some sites, intentionally or through a misconfigured caching or bot-detection layer, serve different content or headers to search engine crawlers than to a regular browser, a practice known as cloaking when done deliberately, and a real, damaging bug when done accidentally. Checking a page's response using a spoofed Googlebot or Bingbot User-Agent, exactly what this tool's User-Agent selector does, is standard, legitimate diagnostic practice for confirming your own site serves what you think it serves to crawlers specifically, not something exclusive to malicious use. If a page returns a 200 status to a browser but a 404 or an unexpected redirect to Googlebot's user-agent, that is a real, otherwise invisible indexing problem this comparison catches directly.

The one header most people never check: X-Robots-Tag

A page can be told not to index in two different ways, a meta robots tag inside the HTML itself, or an X-Robots-Tag HTTP header set at the server level, invisible in the page's visible HTML entirely. This second method is especially easy to miss, since it doesn't show up when viewing page source, only in the raw response headers, and it applies with equal force, a page with X-Robots-Tag: noindex is excluded from search results just as effectively as one with a meta noindex tag, even if every visible SEO element on the page looks perfectly optimized. This tool checks for exactly this header on every URL and flags it prominently when found, since it is one of the most consequential, and most overlooked, checks in a technical SEO audit.

Checking canonical and redirect consistency

Enabling the canonical and redirect consistency check compares each checked URL against where it actually ends up after following every redirect, flagging any protocol change, http upgrading to https, a host change, www being added or removed, or a path change along the way. This is particularly useful for auditing a list of URLs from an old sitemap, a backlink profile, or an internal link inventory, confirming they consistently resolve to the exact canonical form of each page rather than an inconsistent mix of www and non-www, or http and https, versions that could otherwise dilute ranking signals across duplicate variations of the same page.

Checking a competitor, or your own site, at scale

Because this tool accepts a full list of URLs at once, one per line, and processes them concurrently, it works just as well for auditing a competitor's key pages, checking their caching strategy, security header posture, and redirect hygiene, as it does for a full internal audit of your own site's top pages. Checking a competitor's public-facing response headers this way is no different from what a browser's own dev tools would show you visiting each page manually, this tool just does it for an entire list at once instead of one tab at a time.

Runs through a small server-side proxy

Since reading full response headers across domains isn't possible directly from a browser tab, checks run through a small server-side proxy that fetches each URL and reports its status, headers, and redirect behavior back to your browser. Nothing about the URLs checked is logged or stored beyond completing the request. If you specifically want to trace a longer redirect chain in more visual detail, our Redirect Chain Checker is built around that single-URL, hop-by-hop view, both part of the same Technical SEO Tools collection this tool belongs to.