CitantScanBot

CitantScanBot is the crawler behind the free AI Crawler Access Report. It only ever visits a site because a person asked for a scan of that site. It never crawls on its own schedule and never follows links beyond the scan scope.

What a scan fetches

• robots.txt, sitemap.xml, llms.txt and llms-full.txt
• The homepage and up to 5 additional pages from your sitemap (or pages the user specified)
• A small sample of sitemap URLs, status check only

All requests are read-only GET or HEAD requests, capped at roughly 35 requests per scan with a 10 second timeout each and at most one retry. Diagnostic requests are sent with the user agent strings of the AI bots being tested; that is the test itself: it reveals whether your firewall treats those bots differently from your stated robots.txt policy. Every other request identifies as:

Mozilla/5.0 (compatible; CitantScanBot/1.0; +https://citant.ai/scanbot)

What we store

Structured findings (verdicts, status codes, scores), not page content. Scan results live behind private, unguessable links for 7 days and are never published unless the person who ran the scan explicitly shares them. Aggregate statistics are domain-anonymized.

Blocking CitantScanBot

Add this to your robots.txt and scans of your site will report it and stand down:

User-agent: CitantScanBot
Disallow: /

Questions: citant.ai