AI Crawler Access Report

2. Unblock AI crawlers in robots.txt · est. 15 min

Addresses: PerplexityBot blocked by robots.txt policy (WARNING)

1. Add the recommended snippet below ABOVE any general User-agent: * block; specific user-agent groups override the general one.
2. Deploy the updated robots.txt and confirm by opening https://yourdomain/robots.txt in your browser.
3. Re-run this scan; blocked rows should flip to Allowed.

# Recommended AI crawler access, generated by the AI Crawler Access Report by Citant.ai
# Place these lines ABOVE any general User-agent: * block in robots.txt

User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: CCBot
Allow: /
User-agent: Meta-ExternalAgent
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /

# Optional training opt-outs. These are business decisions, not defects:
# blocking Google-Extended removes you from Gemini training but does NOT
# affect Google Search ranking. Leave them out to stay fully open.
# User-agent: Google-Extended
# Disallow: /

3. Fix your sitemap · est. 30 min

Addresses: Sitemap declared in robots.txt returns HTTP 403 (WARNING)

1. Ensure /sitemap.xml returns 200 and lists your live URLs; remove entries that 404.
2. Reference it from robots.txt with a Sitemap: line so crawlers find it without guessing.

Sitemap: https://yourdomain.com/sitemap.xml

4. Add extractable main content · est. half a day

Addresses: Only 97 words of extractable content on the homepage (WARNING)

1. The page carries under 150 words of extractable main content; AI systems have almost nothing to quote or cite.
2. Write the page copy into the HTML itself rather than images or embedded widgets.

5. Repair the heading hierarchy · est. 1 to 2 hours

Addresses: Heading hierarchy issues: 2 H1s (WARNING)

1. Use exactly one H1 per page and do not skip levels; LLM retrieval splits pages into chunks along heading structure, and broken hierarchy produces orphan chunks with no context.
2. Restructure headings in the page template rather than styling text to look like headings.

6. Add Organization schema with sameAs links · est. 30 min

Addresses: No Organization schema on the homepage (WARNING)

1. Add the JSON-LD below to your homepage head, filling in your real details.
2. The sameAs links are the entity-corroboration signal that connects your site to your LinkedIn and other profiles.
3. Validate at validator.schema.org.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Chatgpt",
  "url": "https://chatgpt.com",
  "logo": "https://chatgpt.com/logo.png",
  "description": "What Chatgpt does, in one sentence.",
  "sameAs": [
    "https://www.linkedin.com/company/chatgpt",
    "https://x.com/chatgpt"
  ]
}
</script>