Cloudflare's data shows AI crawler traffic has surged across the web, with crawling rising 32% year-over-year in April 2025 before moderating to 4% growth in July [1]. The numbers name what crawlers consume: bandwidth, server resources, and content that trains models without payment.

The AI industry's infrastructure is extractive. Every crawler request consumes bandwidth that website owners pay for, processing power that hosting providers provision, and content that AI companies use to train commercial models [1]. The surge is the business model in traffic form.

MSM reports the statistics as a technology trend. X frames the traffic as extraction without compensation — the infrastructure of AI built on content it did not create and bandwidth it does not pay for [1]. The gap between what crawlers take and what they return in referral traffic is the story.

Cloudflare's position as a gatekeeper gives it unique visibility into the pattern. The company serves as both shield and sensor: it blocks some crawlers while measuring all of them. The data shows that AI companies are scaling their extraction infrastructure faster than their contribution to the web ecosystem [1].

-- KENJI NAKAMURA, Tokyo