Search is how we find nearly everything on the web — creators, merchants, answers. AI is rewriting the rules, leaving creators caught between staying discoverable in an agentic era and getting paid for their work. Today, Cloudflare is launching two initiatives to help.
- AI Audit – A free tool for website owners to see how AI search engines and crawlers interact with their content. It provides visibility into AI agent traffic, including which AI models (e.g.,
GPTBot,Claude-Web) are accessing your site and how often. This helps developers understand their site's AI footprint and identify unauthorized scraping. - AI Earnings for Creators – A new program that enables creators to receive compensation when their content is used in AI training or search results. Participants can set terms via
_ai.txtfiles (similar torobots.txt) to declare usage policies and monetization preferences. Cloudflare will enforce these rules across its network, allowing creators to opt in or out of AI use cases while maintaining discoverability.
These initiatives address a growing tension: as AI search becomes more common, creators risk losing traffic and revenue if their content is freely ingested by AI models. Cloudflare's approach gives developers and content owners granular control over AI access without blocking legitimate human visitors. The _ai.txt file format provides a standardized way to communicate AI usage rights, and Cloudflare's enforcement at the edge reduces the need for server-side changes. For developers building on Cloudflare’s platform, these tools integrate seamlessly with existing Workers and Pages deployments. The AI Audit dashboard is available now in the Cloudflare dashboard, and AI Earnings is rolling out in beta. This is a practical step toward a fairer web where AI search can coexist creator compensation.
Source: https://blog.cloudflare.com/making-ai-search-smarter/