Security · 2h ago
Cloudflare blocks bots that scrape for both search and AI training
Cloudflare will block crawlers that collect data for both search indexing and AI training, addressing a dilemma for publishers who risk losing search visibility when blocking AI scrapers. The move targets bots that ignore robots.txt or pretend to be benign search crawlers. Publishers can now block such bots without harming their search rankings.
Meridian48 take
This is a practical fix for a growing problem, but it relies on Cloudflare's ability to accurately identify dual-purpose bots, which could be an arms race.
Read the full reporting
Cloudflare to block cynical search-and-scrape bots from ad-supported web pages →
The Register
cloudflarebot-blocking