A blocklist is three decisions wearing one hat
Somewhere in the last year, "should we block AI crawlers" became a question people answer by pasting a list from a blog post. The list usually contains GPTBot, ClaudeBot, CCBot, Bytespider and Google-Extended under one comment that says AI bots. That grouping hides the decision that matters, because those five entries do four different jobs.
Training crawlers collect pages for model datasets. GPTBot, ClaudeBot, CCBot and meta-externalagent belong here. Blocking them costs you nothing in search visibility today and gives up whatever future models might have learned from your pages.
Search crawlers fetch pages so an assistant can answer a question with a citation. OAI-SearchBot, Claude-SearchBot and PerplexityBot are the main three. These are the ones that put your URL inside a ChatGPT or Perplexity answer. Block them and you leave that surface without an error appearing anywhere.
Google-Extended is neither. It is a token in robots.txt that controls whether Gemini training and grounding can use your content. There is no separate crawler to rate limit, no user agent to match in a log line, and disallowing it does not touch Googlebot or your ranking in Search.
Then there are the fetchers that run when a person asks an assistant to open a URL: ChatGPT-User, Claude-User, Perplexity-User. OpenAI, Anthropic and Perplexity have each said some version of the same thing, that a fetch a user requested is not a crawl and does not have to obey robots.txt. Perplexity has argued this most bluntly, which is why publisher disputes keep landing on Cloudflare's desk rather than in a standards body.
Four groups, four answers. A single blocklist collapses them into one, and the collapsed version almost always blocks the search crawler by accident.
The numbers stopped being small
Cloudflare published crawler data covering May 2024 to May 2025 from its own network. AI crawlers excluding Googlebot averaged 4.2% of all HTML requests, peaking at 6.4% in late June. GPTBot's raw requests grew 305% and its share of crawler traffic went from 2.2% to 7.7%. ChatGPT-User grew 2,825%. ClaudeBot went the other way, down 46%, its share falling from 11.7% to 5.4%. PerplexityBot started from almost nothing and recorded the largest relative jump Cloudflare measured, which is what a small number doubling repeatedly looks like.
Network-wide percentages understate the effect on a site that is interesting to a model. Vercel's crawler research found ChatGPT-User spending 34.82% of its requests on 404 pages. Independent log audits point the same way. JetOctopus, which runs these audits for enterprise clients, reports GPTBot 404 rates near two thirds of its traffic, and the same crawler touching 2.2 times as many unique URLs as desktop Googlebot.
That last figure is the one to sit with. A bot asking for more distinct URLs than Googlebot, and wasting a third to two thirds of them on pages that do not exist, is not crawling your site. It is enumerating it and finding out.
Read your own logs before copying someone else's list
Every number above is an average from somebody else's traffic. Your site has its own shape, and ten minutes of log reading beats all of it.
First make sure the user agent is in the log line. Default nginx combined format includes it. If your format is custom, add $http_user_agent before doing anything else, then give it a day of data.
Count requests per bot over a fixed window. A rough total is grep -ic gptbot access.log, and awk '{print $1}' access.log | sort | uniq -c | sort -rn | head shows which addresses are doing the asking. Compare the same window for Googlebot so you have a scale instead of an absolute number.
Break the counts down by status code, because that is where the useful finding lives. A bot with 40% 404s is spending your CPU on requests that cannot help it or you. A bot with a clean 200 rate and a low unique-URL count is reading the same twenty pages and ignoring the rest, which tells you something about internal linking you probably did not want to hear.
Then verify the user agent. Any script can send GPTBot and real traffic does. OpenAI, Anthropic, Google and Perplexity publish IP ranges, and Cloudflare publishes a verified bot list you can match against. A log line with the right user agent and an address outside the published range is an SEO tool, a scraper, or somebody testing your rate limits.
The 404 tax, and why it is your problem
AI crawlers find dead URLs the way everything else does: old links, sitemaps that were never pruned, faceted navigation generating a URL for every filter combination, pagination parameters that multiply. Googlebot learned to slow down and stop asking. Several AI crawlers have not.
The fix is not a block. It is telling the truth about which URLs exist.
Pages that are gone permanently should return a 410, not a 301 to the homepage and not a 200 with a soft not-found template. A 410 is a clear answer and it ends the request cycle. The redirect-to-homepage pattern is worse than a 404 because it burns a hop and returns real content, which teaches the crawler that the URL is valid.
Faceted URLs nobody needs should be canonicalized or disallowed, and parameters that only reorder a listing should be handled in code rather than left to accumulate. A sitemap of 30,000 URLs where 9,000 are 404 is a machine for generating repeat 404 traffic, and AI crawlers treat it as a work queue.
None of this is exotic. It is the same hygiene that kept Googlebot out of your filters in 2019. The difference is that the crawlers arriving now have no shared budget and no shared patience.
Where robots.txt stops
robots.txt is a request defined in RFC 9309. Nothing enforces it. Googlebot and the documented training crawlers mostly comply because compliance is cheaper than the reputational damage, and because they can be identified and negotiated with. Undocumented fetchers and anything spoofing a user agent do not.
So the block you write in robots.txt and the block that actually stops traffic are two different systems. The file is a policy statement you can point at later. The enforcement is a server or edge rule.
In nginx that means a map on $http_user_agent feeding a limit_req zone, or a return 403 for specific strings. On a CDN it means a WAF rule or bot management keyed on verified identity rather than the header. Rate limit rather than hard block where the crawler is one you want citations from, because a 429 with a sensible retry-after is a correction and a 403 is a wall.
One warning on that: rate limiting by user agent string is trivial for a scraper to evade by lying. It works as a speed bump against honest bulk crawlers and is useless against a motivated one. If the traffic actually matters, match on IP range and verified bot status.
Paying for the crawl
Cloudflare's pay per crawl, in private beta since July 2025, answers a priced bot with HTTP 402 Payment Required instead of the page. By April 2026 the public numbers were more than one million enabled customers and more than one billion 402 responses a day across the network.
Those two figures describe a lot of refusals and a modest amount of payment. Most AI crawlers still walk away from a 402 rather than pay, which is why publishers reporting revenue from the programme tend to report small numbers. The mechanism is real and the direction is probably right. The economics are not yet a replacement for ad or subscription revenue.
In September 2026 Cloudflare also changed defaults for free plans: ad-supported pages would allow search crawlers, block training, and block agent use, with mixed crawlers that refuse to separate those purposes blocked outright. If the site sits behind Cloudflare on a free plan, part of your AI crawler policy may already have been decided for you. Check it rather than assume it.
llms.txt is not the lever it was sold as
The file is a Markdown index of your best pages. It is cheap to write and it is not a Google ranking signal. Google said so in its AI optimisation guidance in May 2026 and clarified it again in June: Google Search does not use llms.txt for Search, AI Overviews, or AI Mode.
Nor is there a public commitment from OpenAI, Anthropic, Meta or Mistral that their production systems read the file on arbitrary domains. Limy, which monitors bot traffic for brands, analysed more than 500 million bot events and found AI search crawlers almost never fetching /llms.txt, reading HTML directly instead.
That does not make the file worthless. It makes it a developer and agent convenience, which is a narrower and more honest claim. Coding assistants and agent frameworks that need your documentation can follow an index instead of crawling blind. Ship one if the product has docs. Do not sell it to a client as an AI visibility lever, and keep it away from proposals that also contain real technical work, where it will look like filler.
Adoption is chasing the hype regardless. Google indexed somewhere between 30,000 and 60,000 llms.txt files in October 2025. By May 2026 the count was around 120,000 with a peak near 200,000 in April. A good number of those were written to satisfy a checklist rather than a crawler.
The answer depends on what the site sells
For a publisher or content site, the tension is direct: training crawlers take the value, search crawlers deliver the referrals. Blocking training while allowing search is coherent and costs little today. Blocking both is a decision to exit AI answers, and it should be made by whoever owns the revenue number, not by whoever edits robots.txt last.
For an e-commerce store, product pages are the asset and the waste sits in the long tail: filter combinations, sorted listing pages, out-of-stock variants, session IDs. Allow the search crawlers, because a product that cannot be cited in an AI answer is a product missing from a growing slice of shopping research. Treat the training decision as a negotiating position rather than a technical one. Fix the URL space first, since 404s cost more than either choice.
For a SaaS company, documentation and pricing pages are what get quoted, and the docs are what agents want to fetch mid-task. Allow search and user fetchers on docs, rate limit anything that reads the whole tree in an hour, and keep admin and API endpoints out of every crawler's reach with authentication rather than robots.txt.
What to check every month
Four numbers from the same log window, read on the same day each month. Requests per named bot, so a step change shows up rather than a trend. The 404 rate per bot, which is wasted capacity and a broken URL space in one figure. Unique URLs per bot against the sitemap count, which is your coverage gap. And the share of AI traffic arriving from addresses outside the published IP ranges, which is your spoofing and scraping signal.
An hour of that replaces a year of arguing about whether to block. The crawlers will keep moving. GPTBot grew fivefold in a year while ClaudeBot halved in the same window, which is enough evidence that any fixed list is a snapshot rather than a policy.
Short version: measure first, allow the search crawlers, fix the 404s, and treat robots.txt as the memo rather than the lock.
