CitantScanBot

CitantScanBot

CitantScanBot is the crawler behind the free AI Crawler Access Report. It only ever visits a site because a person asked for a scan of that site. It never crawls on its own schedule and never follows links beyond the scan scope.

What a scan fetches

All requests are read-only GET or HEAD requests, capped at roughly 46 requests per scan with a 10 second timeout each and at most one retry. Diagnostic requests are sent with the user agent strings of the AI bots being tested; that is the test itself: it reveals whether your firewall treats those bots differently from your stated robots.txt policy. Every other request identifies as:

Mozilla/5.0 (compatible; CitantScanBot/1.0; +https://check.citant.ai/scanbot)

What we store

Structured findings (verdicts, status codes, scores) plus the page titles, meta descriptions and headings the checks read, never full page text. A private results page lives for 7 days behind its unguessable link. A result becomes permanent and publicly viewable at its /scan/r/{domain} page only when the person who ran the scan takes an action that says so on its face: pressing Share, unlocking the full report, or opening the print report. Separately, we keep a permanent internal record of every scan so we can measure how AI crawler access changes over time: the address scanned, the time, the scores and findings above, and about the visitor only their country, browser type, the page they came from, and a one-way hash of their IP address (never the address itself). That record is never published per domain; anything we publish from it is aggregated.

Blocking CitantScanBot

Add this to your robots.txt and scans of your site will report it and stand down:

User-agent: CitantScanBot
Disallow: /

Questions: citant.ai