Cloudflare rolls out fine-grained AI bot controls for website owners
Cloudflare is handing website owners the keys to their own digital front door. Instead of blocking all AI crawlers at once, admins can now pick and choose which bots are allowed on their pages. The shift from a blanket restriction to granular controls aims to balance innovation with protection, giving publishers more say over how their content is used.
A new layer of control for publishers
Under the updated policy, Cloudflare’s customers can set separate rules for Search, Training, and Agent bots. Search crawlers—those indexing pages for search engines—can be permitted while Training bots, often used to feed AI models, remain restricted. Agent bots that interact with websites programmatically can also be managed independently. This flexibility arrives as Cloudflare prepares to flip the switch on default blocks for Training and Agent bots on ad-supported pages starting September 15, 2026.
Why the change matters
The move reflects growing tension between AI developers seeking vast datasets and publishers wary of unchecked scraping. By letting site owners decide which bots get access, Cloudflare hopes to reduce friction for legitimate services while curbing abusive data harvesting. For websites relying on ad revenue, the default block on Training and Agent bots could help preserve user experience and monetization. The policy update underscores how infrastructure providers are becoming central arbiters in the AI data economy.
Source: The Decoder. AI-assisted editorial synthesis — TechnoExpress.