Dev Tools · 2h ago
Block AI Crawlers: The 15 Bots That Matter and How to Handle Them
A detailed guide identifies 15 AI crawlers across four categories: training, assistant, search, and scrapers. It corrects common misconceptions about Google-Extended and Applebot-Extended tokens. The article helps site owners decide which bots to block based on visibility trade-offs.
Meridian48 take
The piece offers practical nuance beyond the usual copy-paste lists, but its value depends on how quickly the crawler landscape shifts.
ai-crawlersrobots-txt