Publishers Raise Shields Against AI Scraping
Digital publishers are increasingly moving to block Google’s web crawlers to prevent their content from being used to train generative AI models without compensation. Cloudflare, a major content delivery network, has introduced a feature allowing website owners to block these “multi-purpose” crawlers, a shift that forces a confrontation between search visibility and content ownership. Major media players, including USA Today’s parent company, have indicated they are prepared to delist from Google Search if licensing agreements for their data are not reached.
The Cloudflare Technical Barrier
Cloudflare has introduced a technical barrier aimed at Google’s dual-purpose crawler, which simultaneously indexes websites for search results and scrapes data to train AI models. Starting September 15, Cloudflare’s default settings for new websites and free-tier customers will block crawlers that serve both search and AI training functions on pages containing advertisements.
According to Cloudflare Chief Strategy Officer Stephanie Cohen, the move provides a technical solution for publishers to remain discoverable while retaining control over their content. While the policy affects various bots, it specifically targets Google, which uses one crawler to both index sites and train its AI models.
USA Today and the Licensing Ultimatum
The decision to block Google reflects a shift in how publishers value search traffic versus the potential revenue from AI licensing. USA Today Inc. CEO Mike Reed confirmed that the company is weighing the removal of its sites from Google Search within the next six to twelve months. Reed stated that the company expects compensation for its data and has already secured licensing agreements with firms such as Meta, Microsoft, and Amazon.
This stance is supported by the creator network Beehiiv, which has partnered with Cloudflare to provide its creators with the ability to block Google’s crawler. For many publishers, the calculation is simple: if search traffic declines, the value of appearing in Google results diminishes, making the withholding of content a more effective bargaining chip for future licensing deals.
Google Defends Its Search Ecosystem
Google maintains that it provides granular controls for publishers to manage how their content is used. A spokesperson for the company stated that Google offers tools such as “Google-Extended” for AI training and a new Search Console control for generative AI features, asserting that neither impacts traditional search visibility. The company emphasized its commitment to designing experiences that drive traffic to publishers.
However, industry analysts note the tension between these tools and the current business reality. SEO consultant Lily Ray observed that while publishers have previously blocked other AI firms, Google presents a unique challenge due to its massive user base. Industry executives suggest that if major publishers continue to block Google’s crawler, the quality of Google’s generative search results could decline, as the “answer engines” rely on premium, authoritative news content to function accurately.
Market Volatility and Alternative Revenue
The pushback against Google occurs as media companies navigate broader declines in search-driven traffic. Industry data from Similarweb indicates significant volatility, with Newsweek experiencing a notable reduction in readership from 100 million in May 2025 to 23 million last month. This contraction has led to layoffs across various departments at the publication, including product and video teams.
Concurrently, platforms like Netflix are expanding their reach into digital publishing through licensing deals with companies such as Condé Nast, Hearst Magazines, and People Inc. By bringing premium video content into its ecosystem, Netflix is positioning itself as a subscription-supported alternative to platforms that rely on user-generated content or open-web scraping, further diversifying where publishers seek to monetize their intellectual property.
Worth a look