Cloudflare's latest data confirms that automated bots now account for 57.5% of all HTTP requests to HTML pages globally, while human-generated traffic has fallen to 42.5% [1]. In the United States, the picture is starker: bots command 71.5% of domestic web requests.

Cloudflare CEO Matthew Prince projected the crossover would arrive by late 2027 when he spoke at SXSW earlier this year. Agentic AI traffic accelerated the timeline by more than a year. A human shopping for a product might visit five websites. An AI agent performing the same task could query 5,000 sites — a thousandfold multiplier that reshapes the economics of the open web [2].

The company's answer is a dormant HTTP status code revived for the AI age: 402, "Payment Required." Cloudflare's AI Crawl Control and Pay Per Crawl systems let site owners set a price per request. When a crawler hits a protected page, the server returns a 402 carrying a price header. If the bot agrees, it resends the request with a payment header and receives the content [3].

The 2025 Imperva Bad Bot Report independently confirmed automated traffic breached 50% for the first time in a decade, reaching 51% of global web traffic in 2024 [1]. For publishers, the implications are material: AI crawlers extract content to train models that compete with the original sources, while generating no ad revenue, no subscription revenue, and no direct traffic.

The question is no longer whether the web will become pay-to-crawl. It is how fast the payment infrastructure can be built before the extraction becomes irreversible.

-- CHARLES ASHFORD, London