Cloudflare has introduced a way for site owners to stay discoverable in search while disallowing AI training, addressing the growing tension between being indexed by search engines and having content harvested for model training. The company describes the approach as a shared model built with Apple, Google, and Microsoft, rather than a unilateral block that risks removing a site from search results entirely.
The core idea is that not all automated crawlers serve the same purpose. A search indexer that returns traffic to the publisher is fundamentally different from a bot that ingests content purely to train a model. Cloudflare's new controls aim to encode that distinction through policy rather than blunt blocking.
- New site-owner controls to stay discoverable in search while disallowing AI training, rather than choosing one or the other.
- An Accountable designation that establishes a shared model with Apple, Google, and Microsoft, signaling which crawlers operate under agreed norms.
- A policy framework separating mixed-use crawlers by intent, so indexing and AI training can be treated differently at the edge.
For developers, this matters because crawler policy has become an infrastructure concern, not just a robots.txt afterthought. Sites that previously had to choose between visibility and protection can now express intent more precisely, and edge providers can enforce it. That said, enforcement depends on crawlers opting into the shared model, so teams should treat the Accountable designation as a signal to verify in their own logs and access rules. If you maintain a site or an API serving public content, review your current bot rules, confirm how search and AI crawlers are identified, and test that your configuration still returns the indexing behavior you expect before rolling changes out broadly.
Source: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/