How this score is measured
The overall score weights Live Answer Visibility at 45%, Search Index Foundation at 30% and Training Data Presence at 25%, rounded to the nearest point. Critical findings can cap the score lower; any applied cap is stated next to it. Live Answer Visibility and Training Data Presence reflect crawler access plus on-page structure, with Training Data Presence also counting Common Crawl. Search Index Foundation reflects Googlebot and bingbot access plus indexability, sitemap and response health.
Our crawler tests send each bot's user agent from our own servers, not from the AI companies' verified IP ranges. Some firewalls, including Cloudflare, verify crawlers by network identity rather than user agent. That means a site can challenge our test request while allowing the real, verified crawler. When that happens this scan over-reports blocking; it never under-reports it. The Googlebot result is the most exposed to this, because fake Googlebots are the most common spoof and are policed hardest.
This scan of chatgpt.com checked 15 AI crawlers across 21 automated checks on 1 page. 11 of 13 scored crawlers are currently blocked. AI Access Score: 49/100 (Partially blocked).
Critical: 6 findings hiding you from AI
Your robots.txt allows OAI-SearchBot, but the server refused its request. Your stated policy says yes; your firewall says no, and you probably did not choose this. The site blocks automated traffic broadly, not only AI crawlers. Our test sends this crawler's name from outside OpenAI's network; a firewall that verifies crawler identity can still admit the real crawler, so your server logs are the final word.
Cloudflare manages AI crawlers per behavior category (Search, Agent, Training) under Security Settings, then Configure AI bot policies. As of September 15, 2026, new Cloudflare domains block Agent and Training crawlers on pages with ads by default, so this block may be a default nobody chose.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" "https://chatgpt.com/" | head -5Your robots.txt allows ChatGPT-User, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" "https://chatgpt.com/" | head -5Your robots.txt allows Claude-SearchBot, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)" "https://chatgpt.com/" | head -5Your robots.txt allows Claude-User, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)" "https://chatgpt.com/" | head -5Your robots.txt allows DuckAssistBot, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 (compatible; DuckAssistBot/1.0; +https://duckduckgo.com/duckassistbot)" "https://chatgpt.com/" | head -5ChatGPT, Claude, Perplexity and DuckAssist all fail to fetch this site; you cannot be cited in live AI answers.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" "https://chatgpt.com/" | head -5The bot matrix
robots.txt policy next to what actually happened when each bot knocked. Rows where your policy says yes but the server says no are highlighted; those are the silent failures.
Live-answer botsFetch content in real time to build the answers and citations users see today.
| Bot | Owner | robots.txt policy | Real fetch | Verify |
|---|---|---|---|---|
| OAI-SearchBot Powers ChatGPT search results | OpenAI | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| ChatGPT-User User-triggered fetches from ChatGPT; Cloudflare classes this as Agent traffic, blocked by default on ad pages for new Cloudflare domains | OpenAI | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| PerplexityBot Index for Perplexity answers | Perplexity | Blocked line 40: Disallow: / | Refused (403) | |
| Perplexity-Userinformational User-triggered; may ignore robots.txt | Perplexity | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| Claude-SearchBot Claude search indexing | Anthropic | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| Claude-User User-triggered fetches from Claude; Cloudflare classes this as Agent traffic, blocked by default on ad pages for new Cloudflare domains | Anthropic | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| DuckAssistBot DuckAssist answers | DuckDuckGo | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. |
Training crawlersCollect content for model training. Blocking them removes you from future models' memory.
| Bot | Owner | robots.txt policy | Real fetch | Verify |
|---|---|---|---|---|
| GPTBot Training corpus | OpenAI | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| ClaudeBot Training corpus | Anthropic | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| CCBot Feeds many labs' training data; blocking it is the quietest way to disappear from future models | Common Crawl | Blocked line 4: Disallow: / | Refused (403) | |
| Meta-ExternalAgent Llama training | Meta | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| Amazonbot Alexa and Amazon AI features | Amazon | Partial line 218: Disallow: /auth/logout | Blocked by firewall (Cloudflare) Policy says yes; firewall says no. | |
| Bytespiderinformational Known to sometimes ignore robots.txt | ByteDance | Blocked line 34: Disallow: / | Refused (403) |
Search-index foundationCopilot rides Bing's index and AI Overviews rides Google's. If these starve, the AI layer above them starves.
| Bot | Owner | robots.txt policy | Real fetch | Verify |
|---|---|---|---|---|
| Googlebot Google Search index; AI Overviews rides this. Cloudflare treats it as mixed Search and Training, so as of September 15, 2026 Cloudflare AI training blocks can hit it | Partial line 218: Disallow: /auth/logout | Reachable (200) | ||
| bingbot Bing index; Copilot and ChatGPT search lean on it. Cloudflare treats it as mixed Search and Training, so as of September 15, 2026 Cloudflare AI training blocks can hit it | Microsoft | Partial line 218: Disallow: /auth/logout | Reachable (200) |
Training opt-out tokensNot crawlers. Directives controlling whether already-crawled content trains AI.
| Token | Controls | Your robots.txt |
|---|---|---|
| Google-Extended | Gemini training use of Google's crawl Blocking Google-Extended does NOT affect your Google Search ranking, but it does remove your content from Gemini training use. | Opted out |
| Applebot-Extended | Apple AI training use Blocking Applebot-Extended does not affect Siri or Spotlight search. | Partial |
Your scanned pages
Real defects live on money pages, not just homepages. We scanned 1 page: your homepage.
| Page | Status | Indexable | Canonical | Title | Words |
|---|---|---|---|---|---|
| / | 200 | Yes | OK | ChatGPT: Chat, Work, Create & Code with AI | 97 |
Warnings (10)
Perplexity cannot fetch your content for live AI answers.
Governing rule (line 40): Disallow: /
curl -s "https://chatgpt.com/robots.txt"Your robots.txt allows GPTBot, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot" "https://chatgpt.com/" | head -5Your robots.txt allows ClaudeBot, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" "https://chatgpt.com/" | head -5Common Crawl's training crawler cannot read you; you are opting out of future models' memory.
Governing rule (line 4): Disallow: /
curl -s "https://chatgpt.com/robots.txt"Your robots.txt allows Meta-ExternalAgent, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 (compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler)" "https://chatgpt.com/" | head -5Your robots.txt allows Amazonbot, but the server refused its request. Same firewall pattern as the OAI-SearchBot finding above; one firewall change covers every crawler it blocks, and the same server-log caveat applies.
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)" "https://chatgpt.com/" | head -5robots.txt points crawlers at https://chatgpt.com/sitemap.xml, and that URL returns HTTP 403. Crawlers that trust the declaration hit a dead end, and the conventional locations serve no sitemap either, so your money pages get crawled last or never.
curl -sI "https://chatgpt.com/sitemap.xml" | head -1AI systems have almost nothing to quote or cite from this page.
LLM retrieval splits pages into chunks along heading structure; broken hierarchy produces orphan chunks with no context.
Without a machine-readable identity, AI systems have to guess who you are instead of knowing.
Good to know (2)
llms.txt is an emerging convention some AI crawlers read for site guidance. Adding one is a five minute quick win.
curl -sI "https://chatgpt.com/llms.txt" | head -1Copilot and ChatGPT search lean on Bing's index; if Bing has not indexed you, two of six AI platforms cannot cite you. Automated Bing verification is unreliable without authenticated APIs, so we will not fake it.
Recommended robots.txt additions (free, no email needed)
# Recommended AI crawler access, generated by the AI Crawler Access Report by Citant.ai # Place these lines ABOVE any general User-agent: * block in robots.txt User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: DuckAssistBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: CCBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Amazonbot Allow: / User-agent: Googlebot Allow: / User-agent: bingbot Allow: / # Optional training opt-outs. These are business decisions, not defects: # blocking Google-Extended removes you from Gemini training but does NOT # affect Google Search ranking. Leave them out to stay fully open. # User-agent: Google-Extended # Disallow: /
Your prioritized fix plan
Criticals first. CMS not detected; generic instructions shown. Open the print-ready report to save it as a PDF.
1. Allow AI crawlers through your firewall or CDN
Fixes: OAI-SearchBot blocked by firewall (Cloudflare)
- Your robots.txt allows these bots but your server refuses their requests, which means a firewall or bot-protection layer is overriding your stated policy. Most site owners do not know this is happening.
- In the Cloudflare dashboard, go to Security Settings, then Configure AI bot policies. Cloudflare manages AI crawlers in three behavior categories: Search (crawlers that index content to answer questions later), Agent (real-time fetches on a person's behalf, such as ChatGPT-User), and Training (crawlers that collect content to train or fine-tune models). Set the category that matches the blocked bot to 'Allow (do not block)'.
- Category map for the bots in this report: Search covers OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, Googlebot and bingbot. Agent covers ChatGPT-User, Claude-User and Perplexity-User. Training covers GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Amazonbot and Bytespider.
- While you are there, confirm the legacy 'Block AI bots' toggle is off. Cloudflare deprecated that toggle on September 15, 2026, and as of that date a Training block also applies to mixed-purpose crawlers such as Googlebot, Applebot, and bingbot. Akamai and Imperva have equivalent verified-bot allowlists.
- Verify with the per-bot command in this report; the response should change from 403/503 to 200.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" "https://chatgpt.com/" | head -52. Unblock AI crawlers in robots.txt
Fixes: PerplexityBot blocked by robots.txt policy
- Add the recommended snippet below ABOVE any general User-agent: * block; specific user-agent groups override the general one.
- Deploy the updated robots.txt and confirm by opening https://yourdomain/robots.txt in your browser.
- Re-run this scan; blocked rows should flip to Allowed.
# Recommended AI crawler access, generated by the AI Crawler Access Report by Citant.ai # Place these lines ABOVE any general User-agent: * block in robots.txt User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: DuckAssistBot Allow: / User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: CCBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: Amazonbot Allow: / User-agent: Googlebot Allow: / User-agent: bingbot Allow: / # Optional training opt-outs. These are business decisions, not defects: # blocking Google-Extended removes you from Gemini training but does NOT # affect Google Search ranking. Leave them out to stay fully open. # User-agent: Google-Extended # Disallow: /
curl -s "https://chatgpt.com/robots.txt"3. Fix your sitemap
Fixes: Sitemap declared in robots.txt returns HTTP 403
- Ensure /sitemap.xml returns 200 and lists your live URLs; remove entries that 404.
- Reference it from robots.txt with a Sitemap: line so crawlers find it without guessing.
Sitemap: https://yourdomain.com/sitemap.xml
curl -sI "https://chatgpt.com/sitemap.xml" | head -14. Add extractable main content
Fixes: Only 97 words of extractable content on the homepage
- The page carries under 150 words of extractable main content; AI systems have almost nothing to quote or cite.
- Write the page copy into the HTML itself rather than images or embedded widgets.
5. Repair the heading hierarchy
Fixes: Heading hierarchy issues: 2 H1s
- Use exactly one H1 per page and do not skip levels; LLM retrieval splits pages into chunks along heading structure, and broken hierarchy produces orphan chunks with no context.
- Restructure headings in the page template rather than styling text to look like headings.
6. Add Organization schema with sameAs links
Fixes: No Organization schema on the homepage
- Add the JSON-LD below to your homepage head, filling in your real details.
- The sameAs links are the entity-corroboration signal that connects your site to your LinkedIn and other profiles.
- Validate at validator.schema.org.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "Organization",
"name": "Chatgpt",
"url": "https://chatgpt.com",
"logo": "https://chatgpt.com/logo.png",
"description": "What Chatgpt does, in one sentence.",
"sameAs": [
"https://www.linkedin.com/company/chatgpt",
"https://x.com/chatgpt"
]
}
</script>7. Add an llms.txt file
Fixes: No llms.txt found
- Save the generated draft below as /llms.txt at your site root.
- It is a plain-markdown guide that tells AI systems what your most important pages are; adoption is emerging, and it costs nothing.
# Chatgpt > One-sentence description of what chatgpt.com does, written for AI systems. Edit this line. ## Key pages - [ChatGPT: Chat, Work, Create & Code with AI](https://chatgpt.com/): Use ChatGPT to answer questions, write, create images, complete work, and code—all in one place. Get started for free ## Contact - [Contact](https://chatgpt.com/contact): How to reach Chatgpt
curl -sI "https://chatgpt.com/llms.txt" | head -1