Cloudflare CEO Matthew Prince says bots now account for more than half of all internet traffic, a shift driven by AI companies scraping the web and agents acting on behalf of users, and his company is building tools that let website owners charge those AI firms for access. Prince made the comments on The Verge’s Decoder podcast, laying out a vision where content sites can monetize machine visitors the way they once monetized human ones through advertising.
Cloudflare sits between websites and their visitors, which puts it in a position to see the traffic mix across a large slice of the web. In Prince’s account, the bot share has climbed steadily as AI crawlers pull training data and as agents browse, compare, and transact for their users. The share keeps rising, and the company’s own numbers put automated requests above the halfway mark across its network.
The business logic is straightforward. A human visitor might click an ad or subscribe. A bot reads the page and leaves, and the publisher gets nothing. If bots dominate traffic, the advertising model that funds much of the open web loses its base. Prince’s answer is a pay-per-crawl model: site owners can block AI tools entirely, allow them for free, or permit only those willing to pay.
The gatekeeper position
Cloudflare’s proxy position makes it one of the few companies that can enforce such a system at scale. It already offers one-click blocking of AI crawlers and a marketplace concept where crawlers pay for content. If that becomes the default, Cloudflare ends up setting the price of access to much of the web, a role that draws both revenue and antitrust scrutiny. Prince argued the alternative is worse: content gets taken without payment, and the economics of publishing collapse quietly rather than dramatically.
The enforcement problem is real, though. Well-funded crawlers can rotate IPs, spoof user agents, and route through residential proxies to look human. Cloudflare’s detection has improved, but blocking at the network edge is an arms race, not a switch. What Cloudflare can do better than most is make compliance cheaper than evasion: a signed crawler identity, a published rate card, and clean billing are easier for an AI lab’s engineers than maintaining a scraping fleet that gets flagged weekly.
Some publishers have already signed deals with AI companies, licensing archives for training data. Others have blocked crawlers outright. The middle path, metered paid access, has few live examples so far, and the willingness of AI labs to pay per crawl remains untested at scale. Training runs are one-time costs, but agents querying the live web are recurring ones, which is where Cloudflare sees the durable market.
Why agents change the math
The distinction between training crawls and agent traffic matters for pricing. Training data has a substitute problem: a model trained on 2024’s web does not strictly need 2026’s pages, and licensing deals have covered much of the high-value archive. Agents are different. When a user asks an assistant to compare prices, book a flight, or summarize today’s news, the agent must fetch current pages, and it does so repeatedly across millions of sessions. That traffic cannot be replaced with last year’s snapshot, which gives publishers leverage they never had in the training-data fight.
Prince described a future where website owners can monetize access to their content by AI companies, with Cloudflare handling the metering and payment rails. The conversation also touched on Cloudflare’s own use of AI. Earlier this year the company laid off more than a thousand employees, roughly 20 percent of its workforce, and Prince wrote a Wall Street Journal op-ed titled how he chooses which Cloudflare employees to replace with AI. The company is simultaneously selling the infrastructure that meters AI access and using AI to shrink its own payroll, a combination that captures the moment’s contradiction better than most.
What happens to the open web
Critics of pay-per-crawl worry about a two-tier web where well-funded AI companies buy access and smaller researchers, startups, and archiving projects get priced out. The robots.txt convention was always voluntary, and its weakness is what led to the current standoff. Replacing it with paid access formalizes the wall. Supporters answer that a wall with a price is better than a wall without one, since the current alternative is mass scraping with no compensation at all.
Search engines sit awkwardly in this picture. Google’s crawler both indexes the web and feeds AI answers that reduce click-throughs to publishers. If publishers start charging for crawl access, search companies face a choice between paying for the content that trains their answer engines and losing freshness in those answers. That negotiation has not started in earnest, and it may be the real test of whether the model works.
There is also the question of what agents owe the sites they transact on. An agent that completes a purchase or a booking delivers a paying customer, and some publishers argue that deserves a cut of the transaction, not just a crawl fee. Others note that agents can be steered by the platforms that host them, meaning the agent decides which merchant gets the sale. Prince did not claim Cloudflare would solve that problem, but his framing assumes machine traffic becomes a billable line item, and transaction-based pricing is the logical extension of that assumption.
Prince’s broader claim is that the web’s content economy needs a new settlement layer before the current one fails. Whether that layer is built by Cloudflare, by AI companies paying voluntarily, or by regulation is unresolved. What is clear from the traffic numbers is that the old assumption, that most requests to a website come from a person, no longer holds.