Every website owner now has a second audience: the crawlers that feed ChatGPT, Claude, Perplexity, Gemini and the rest. Most advice about them is written by people who have not looked at a log file. We run a proxy business, our server logs are our day job, so we pulled 15 days of ours, 21 August to 4 September 2026, and counted. This is what AI crawlers actually did on a mid-sized site with about 400 pages.
The three kinds of AI bot, and why the distinction matters
The vendors have converged on the same design: three separate user agents with three separate jobs, so that a site can allow one and block another.
| Job | OpenAI | Anthropic | Perplexity | What it does |
|---|---|---|---|---|
| Training | GPTBot | ClaudeBot | (uses index) | Collects pages for model training. Blocking it has no effect on citations. |
| Search index | OAI-SearchBot | Claude-SearchBot | PerplexityBot | Builds the index the assistant searches when a user asks something. |
| Answer-time fetch | ChatGPT-User | Claude-User | Perplexity-User | Fetches a specific page because a user's question needs it right now. This is the request that becomes a citation. |
Google is the exception: Googlebot serves both search and AI Overviews, and Google-Extended is a robots.txt token that opts you out of Gemini training without touching search. Meta crawls for training as meta-externalagent, Apple as Applebot, Amazon as Amazonbot, ByteDance as Bytespider, and Common Crawl, whose archive many models train on, as CCBot.
What 15 days of logs showed
Requests by user agent across all retained logs, one site, 15 days:
| User agent | Requests | Kind |
|---|---|---|
| bingbot | 4,701 | Search (also feeds Copilot) |
| meta-externalagent | 4,266 | Training |
| ChatGPT-User | 3,308 | Answer-time fetch |
| Bytespider | 2,640 | Training (ByteDance) |
| Amazonbot | 2,493 | Training / Alexa |
| Googlebot | 1,469 | Search + AI Overviews |
| OAI-SearchBot | 1,416 | Search index |
| ClaudeBot | 1,172 | Training |
| Claude-User | 999 | Answer-time fetch |
| Applebot | 841 | Siri / Spotlight |
| GPTBot | 759 | Training |
| PerplexityBot | 561 | Search index |
| Perplexity-User | 355 | Answer-time fetch |
| Claude-SearchBot | 163 | Search index |
| CCBot | 14 | Common Crawl |
Three things stand out. First, the answer-time fetchers together made about 4,600 requests, roughly 300 a day, across 862 different pages. Every one of those is a moment when an assistant decided a page of ours was worth reading to answer a real person. Second, the training crawlers from Meta and ByteDance out-requested OpenAI's training bot several times over, and Bytespider is the one that most often ignores robots.txt in other people's reports; ours allows it, so we cannot say from our data. Third, Bing sent more requests than Google, which surprised us until we remembered that Bing's index also powers ChatGPT search and Copilot.
Which pages the assistants pull
The homepage came first, which is what you would expect: assistants fetch it to answer "what is this company". After that the list was not our most-visited pages. It was our most quotable ones: a pricing breakdown of static residential proxies, a free tool that returns a fact, a captcha-tools comparison with prices, a guide to reading robots.txt, a country roundup with a price table. Pages with numbers and direct answers got fetched. Pages with opinions and no data did not.
That is the practical lesson for anyone who wants to be cited. Assistants are looking for a sentence they can quote and a number they can attribute. A table of prices beats three paragraphs about value. A direct answer under a question-shaped heading beats an essay. This is also good writing for humans, which is not a coincidence.
The fakes: 42 percent of "AI bot" requests were not AI bots
Here is the number that most surprised us. Of all requests carrying an AI user-agent string, 4,400 got a 200 and 3,169 got a 404. Real crawlers rarely 404 at that rate; they read your sitemap. The 404s were requests for /.aws/credentials, /@fs/etc/passwd, /actuator, /application_default_credentials.json, /admin and a few hundred similar paths. Vulnerability scanners have learned that operators now whitelist AI user agents, so they borrow the strings. If you are counting AI traffic from user agents alone, you are counting scanners too.
Verification, in order of reliability:
- Behaviour. Real crawlers request URLs that exist, fetch robots.txt first, and follow the sitemap. Anything probing for secrets is not a crawler, whatever it calls itself.
- Published IP ranges. OpenAI publishes the ranges for GPTBot, OAI-SearchBot and ChatGPT-User; Perplexity publishes ranges for its agents. A request claiming those names from outside those ranges is spoofed.
- Reverse DNS. Googlebot and bingbot resolve to their operators' domains. Most others do not offer this.
- Nothing. Bytespider and several smaller agents publish no verification, which is also why they are the most impersonated.
What to put in robots.txt
Decide the two questions separately, because the user agents are separate. Do you want to be cited in answers? Then allow the search and answer-time agents. Do you object to training? Then disallow the training agents. Ours allows everything, because being quoted is worth more to us than the theoretical loss from training, and the crawl load is a rounding error. A site that wants citations without training would write:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /Two cautions. Robots.txt is a request, not a wall; well-run crawlers honour it and scanners never will, so it is not a security control. And Google-Extended is a token, not a crawler: listing it stops Gemini training on your pages without affecting Googlebot, but you will never see it fetch anything, even though a few fetchers in our logs oddly announced it as a user agent.
What we changed after reading our own logs
We stopped counting user agents and started counting behaviour, which cut our "AI traffic" figure nearly in half. We corrected our llms.txt, which had drifted out of date on two prices, because assistants were reading a file that disagreed with our checkout. And we rewrote several pages so that the answer to the question in the heading sits in the first sentence under it, since that sentence is the one that gets quoted. None of this required new technology. It required reading the log. Our guide to reading a robots.txt file covers the syntax if you are starting from scratch, and if you run scrapers yourself, the same logs are what the other side sees when you visit them.
Frequently asked questions
Should I block AI crawlers in robots.txt?
If you want your pages cited in ChatGPT, Claude or Perplexity answers, no: the answer-time fetchers respect robots.txt, and blocking them removes you from the answer. If you object to your content being used for model training, block the training crawlers (GPTBot, ClaudeBot, meta-externalagent, CCBot, Bytespider) and leave the search and user-triggered agents alone. They are separate user agents for exactly this reason.
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects data for training. OAI-SearchBot builds the index behind ChatGPT search. ChatGPT-User fetches a page at the moment a user's question needs it, and that fetch is what turns into a citation. Anthropic and Perplexity use the same three-way split: ClaudeBot, Claude-SearchBot and Claude-User; PerplexityBot and Perplexity-User.
How can I tell a real AI crawler from a fake one?
Behaviour first: real ones request pages that exist, honour robots.txt and read your sitemap; fakes probe for credentials files and admin paths. OpenAI and Perplexity publish the IP ranges their agents use, so a request claiming to be GPTBot from an address outside those ranges is not GPTBot. Googlebot can be verified by reverse DNS. Bytespider publishes nothing and is the most spoofed string in the logs.
How much traffic do AI crawlers generate?
On our site, roughly 300 answer-time fetches a day from ChatGPT, Claude and Perplexity combined, plus index crawls, over a 15-day sample. That is a few percent of total requests. The bandwidth is trivial; the interesting number is which pages they pull, because those are the pages being quoted to real people.
Does an llms.txt file matter?
Nobody has shown that it changes rankings or citations, and no major assistant has committed to reading it. It costs nothing and is a tidy place to state your products and prices accurately, so we keep one. Treat it as documentation, not as a ranking lever.