Cloudflare Unveils New Tool to Shield Websites from AI Scrapers
Cloudflare, the cloud service provider, has taken a significant step to protect its clients' websites from AI bots that scrape content for training large language models. Understanding the need for content security, Cloudflare has introduced a new free tool that can be utilized by their entire customer base, including those using free plans. The tool is designed to automatically update and identify any bots that are scraping web content for model training.
AI Bots in the Limelight
The rise of bots scraping content has led many website owners to take action. According to Cloudflare's internal data, an overwhelming majority, 85.2 percent, have taken a stance against AI bots, even those that legitimize their presence, by blocking them from their sites. The most active bots over the past year, as identified by Cloudflare, include Bytedance-owned Bytespider and OpenAI's GPTBot which attempted to access 40 and 35 percent, respectively, of websites within Cloudflare's network.
The Battle Against AI Scrapers
The struggle to consistently block AI bots poses challenges as companies strive to build models expeditiously, occasionally bypassing established rules. Instances of unauthorized scraping, such as the recent accusation against Perplexity AI, highlight the urgency of the situation. Cloudflare's commitment to address this issue represents hope for content creators who aim to maintain control over their content's usage.
Cloudflare's Ongoing Efforts
Cloudflare acknowledges the tenacity of some AI companies in trying to evade bot detection and has promised to be vigilant. They plan to advance their machine learning models and implement more bot blocks to their AI Scrapers and Crawlers rule, aiming to preserve the Internet as a domain where content creators can flourish without fear of unauthorized use of their work.
This new development by Cloudflare signifies a step forward in safeguarding original content online, offering website owners some relief in a landscape where AI bots are increasingly common.
