Cloudflare launches tool to help website owners monetize AI bot crawler access

Key takeaways:

  • Cloudflare has launched a new tool that allows website owners to block AI crawlers or charge them using a “pay per crawl” model.
  • This move aims to help content creators and publishers monetize access to their work as AI bots increasingly collect data without driving traffic back to original sources.
  • The initiative is backed by major publishers and platforms, including Condé Nast, Associated Press, Reddit, and Pinterest.
  • Cloudflare’s data shows Google’s crawl-to-referral ratio has worsened (18:1), while OpenAI’s is drastically worse (1,500:1), indicating AI bots are taking content without returning user traffic.
  • The tool is part of a broader effort to protect online content creators as traditional ad revenue from search traffic declines due to AI-generated summaries and chatbot responses.
Cloudflare launches tool to help website owners monetize AI bot crawler access
Site Logo
Reuters
Go to source




Be Part of Something Big

Shifters, a developer-first community platform, is launching soon with all the features. Don't miss out on day one access. Join the waitlist: