Googlebot and GPTBot both show up in your server logs wearing the same costume: a user agent string and a stream of GET requests. But they are not the same customer, and they do not pay in the same currency. A search crawler pays in future clicks. Most LLM crawlers pay in, at best, a citation inside an AI answer, and often in nothing at all. Once I started treating them as two audiences with two separate policies, a bunch of confusing decisions became simple.
I did not reach that conclusion from reading think pieces. I run mejba.me, a blog with roughly 580 posts published in six languages, and over the past year I have watched both kinds of bots through server logs and Google Search Console during the worst SEO period of my life: an indexing crash that took the site from about 7,000 indexed pages down to about 1,700. What each type of bot did, and did not do, during that period taught me more about the difference than any documentation.
Two deals, two currencies
The traditional search bot deal is old and well understood. Googlebot crawls your pages, Google indexes some of them, and when a human searches, your page can appear and earn a click. Crawl cost now, traffic later. Every classic SEO practice, sitemaps, canonical tags, crawl budget management, exists to service that deal.
LLM bots run several different deals, and lumping them together is the first mistake:
- Training crawlers (GPTBot from OpenAI, ClaudeBot from Anthropic, CCBot from Common Crawl) collect content that may inform future model training. You get no traffic. Your ideas may surface, unattributed, in a model's answers a year from now.
- AI search indexers (OAI-SearchBot for ChatGPT search, PerplexityBot) build retrieval indexes for AI answer engines. These can send you cited, linked traffic, a small but real currency.
- User-triggered fetchers (ChatGPT-User and similar agents) hit your page live because a human asked an assistant to read it. That is closer to a browser visit than a crawl.
The reason this taxonomy matters: a blanket "block AI bots" decision treats all three deals as the same deal. They are not. Blocking a training crawler costs you nothing measurable today. Blocking an AI search indexer removes you from an answer surface that a growing slice of developers use instead of Google.
What a year of Search Console taught me about the difference
Three first-party numbers shaped how I think about this.
The crash. Over one brutal two-month stretch, my indexed page count fell from around 7,000 to around 1,700. The cause was on my side: structured data I should never have shipped (review-style markup that Google rightly treats as spam) plus site quality issues I have since spent months fixing. Here is the part relevant to this post: Google's reaction was fast, mechanical, and brutal, because Google continuously re-evaluates its index. The LLM crawlers in my logs did not care at all. ClaudeBot and GPTBot kept fetching at the same lazy rhythm before, during, and after. Search bots judge you continuously; training crawlers just collect. If your mental model is "bots evaluate my site," that is only true of one kind.
The multiplication problem. My site serves six locales, so every post exists at six URLs. At the low point, Search Console showed roughly 14,500 pages sitting in "Crawled, currently not indexed." Googlebot happily spent crawl budget fetching every locale variant of every page, then declined to index most of them. That is the search-bot deal misfiring: I paid server time and crawl budget for pages Google never intended to rank. LLM bots exhibit the opposite pattern. They mostly follow prominent links and popular paths, and in my logs they overwhelmingly fetched the English originals and ignored the locale variants. Search bots exhaustively map your site; LLM bots sample it.
The impressions illusion. Over one twelve-month window the site logged about 1.15 million Google impressions at roughly 0.7% click-through. That gap between being seen and being clicked already looked like the future arriving: appearing in a results page is worth less every year, because more questions get answered without a click. AI answer engines are the extreme end of the same curve. Optimizing purely for impressions is optimizing for a currency that is quietly devaluing.
Technical differences that actually change what you do
Setting aside strategy, four mechanical differences matter day to day:
JavaScript rendering. Googlebot renders JavaScript in a second wave. Most AI crawlers fetch raw HTML and do not execute your scripts. If your content only exists after client-side rendering, you are invisible to most LLM bots regardless of your robots.txt hospitality. Server-rendered HTML (my site is Laravel Blade, rendered on the server) gets you both audiences for free.
Sitemaps and crawl budget. Googlebot reads sitemaps religiously and allocates crawl budget by perceived value; my site regenerates its sitemap files every five minutes precisely because that contract is worth servicing. I have never observed an LLM training crawler care about any of it. Sitemap discipline is a search-bot investment.
Extraction, not indexing. Retrieval systems chunk your page into passages. A section that only makes sense after reading three earlier sections quotes badly. Since realizing this, I write H2 sections that stand alone, which conveniently is also better for humans arriving mid-page from a search snippet.
Load behavior. Search crawlers are polite and budget-conscious. AI fetchers can be bursty, especially user-triggered ones. Every guest page on my site is served with public cache headers precisely so bot bursts hit cached HTML instead of PHP. One hard-won detail from that middleware: vary your cache on Accept-Encoding only, never on Accept-Language. Googlebot rotates its language preference headers between requests, and varying on that header shatters your CDN cache into useless fragments. I learned that one from my own cache-hit graphs.
My robots.txt policy, and why I allow AI crawlers
Here is the AI section of my actual robots.txt, trimmed:
# AI/LLM Crawlers - Welcome
User-agent: GPTBot
Allow: /
Disallow: /admin
Disallow: /api/
User-agent: ClaudeBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Allow: /
The same file also welcomes Applebot-Extended, Meta-ExternalAgent, Bytespider, and cohere-ai, and separately spends thirty lines disallowing parameterized duplicate URLs (?page=, ?tag=, ?query=) for the search bots' benefit. Two policies, one file.
Why allow the training crawlers when they pay nothing? Because of what my site sells. I am a consultant. My content exists to demonstrate that I do this work for real, and an AI assistant recommending my writing, or my name appearing in an answer about Laravel deployment, is distribution. If my content were the product itself, a paywalled course, licensed data, original reporting, I would block training bots without hesitation and keep only the AI search indexers. The right answer is a function of your business model, not of principle.
I will be honest about the counterweight: Cloudflare offers a one-toggle AI bot block, and it has been sitting in my "pending decisions" list for months. The fact that I keep not flipping it is the decision. But I re-evaluate it whenever an AI product quotes my work without attribution, and you should expect this policy to be something you revisit yearly, not settle once.
One more concrete step: I serve llms.txt and llms-full.txt from the site root, generated by a controller so they never go stale, following the llmstxt.org spec. It is a low-cost courtesy signal; adoption by AI systems is still uneven, so treat it as cheap insurance rather than a ranking lever.
The playbook I would give a friend
- Grep your access logs for
GPTBot,ClaudeBot,OAI-SearchBot,PerplexityBot,CCBot. You cannot set policy for traffic you have not measured. - Decide per category, not per vibe. Training crawlers: allow if content markets you, block if content is the product. AI search indexers: allow unless you have a specific reason not to. User-triggered fetchers: treat like browsers.
- Serve real HTML. If your content needs JavaScript to exist, fix that before touching robots.txt.
- Write sections that stand alone and keep facts accurate; retrieval systems cross-check sources, and errors get you silently dropped from answers.
- Keep the classic search hygiene anyway. Sitemaps, canonicals, and crawl-budget discipline still govern the channel that pays your bills today. My own recovery depended on it. I keep automated daily SEO checks running through a Claude Code routine so drift gets caught in days, not months, and I've written up the broader Claude-based SEO toolkit I use for audits.
- Watch both scoreboards. Search Console for the old deal; periodically ask ChatGPT, Claude, and Perplexity questions you have answered on your site, and see whether you are cited. If you want to go further, programmatic SEO with Claude covers scaling the content side.
The publishers who do fine through this shift will be the ones who noticed early that "bot traffic" stopped being one thing. Set two policies on purpose, and re-read your logs every quarter, because this roster of user agents changes faster than any other part of SEO.
Where this goes wrong in production is a hurried robots.txt edit that also shuts out the crawler paying your bills — I have watched an indexing graph fall off a cliff for exactly that reason and spent months earning it back. Crawler policy, indexing health, and Search Console recovery audits are part of what I do for clients.
