Dev Tools · 8h ago
CDN Configs Silently Block AI Crawlers from Docs, Audit Finds
A developer audit of thousands of sites found 27% unintentionally block AI crawlers like GPTBot at the CDN or WAF layer, not in robots.txt. Common bot-protection rules return 403 errors for these user agents, even when robots.txt is permissive. The article provides curl commands to test and fix the issue, including server-rendering content for crawlers that don't execute JavaScript.
Meridian48 take
The real story is that many companies are paying for AI training data they can't access because their own security stack is too aggressive.
Read the full reporting
Your CDN Might Be Blocking ChatGPT From Your Docs (And robots.txt Won't Show It) →
DEV Community
ai-crawlerscdn-configuration