AI crawlers may already be reading your website, but ordinary analytics often shows nothing. A visit from GPTBot, ClaudeBot, PerplexityBot, or another crawler is not the same thing as a customer visit, a search referral, or a recommendation.
The uncomfortable part of AI discovery is that many crawlers take far more pages than they send back as traffic. SEOmator's 2026 analysis of Cloudflare Radar data describes this as a crawl-to-refer ratio: pages crawled divided by referrals returned to the website.
The exact ratios differ by data source and time window, but the business lesson is stable: a crawler reading your site is not proof that buyers are arriving. You need to separate training crawls, search indexing, and user-triggered fetches.
GPTBot
OpenAI crawler used for content that may help improve foundation models. It is not the main signal for ChatGPT Search traffic.
OAI-SearchBot
OpenAI search crawler used to surface sites in ChatGPT search features. This is the bot to watch for ChatGPT Search discoverability.
ChatGPT-User
User-triggered access. It may appear when a real ChatGPT user asks a question that requires a live page.
ClaudeBot
Anthropic crawler for public web content that may contribute to model development.
Claude-SearchBot
Anthropic crawler for search-related indexing inside Claude. Blocking it can reduce visibility in Claude search scenarios.
PerplexityBot
Perplexity's crawler for search-style indexing. Its traffic should be measured separately from referral visits.
Most AI crawlers do not behave like human visitors. They usually do not run your analytics JavaScript, click buttons, open forms, or behave like a normal browser session. If you only look at client-side analytics, you may conclude that nothing happened while your server logs show thousands of bot reads.
That is why Agntbase treats AI traffic as a server-side event stream. A useful dashboard should show the bot family, requested path, status code, response class, and whether the request hit an agent-ready entrypoint.
Track status classes: 2xx, 3xx, 4xx, 5xx, and edge cases like 499 or reset connections.
Separate training bots from search bots and user-triggered fetchers.
Watch reads of /llms.txt, /agenthub.json, /profile.json, and canonical profile JSON.
Track referrals and UTM data separately. Bot reads are infrastructure signals, not revenue by themselves.
llms.txt, manifest, and JSON-LD.The goal is not to trick AI systems. The goal is to make the business readable, measurable, and diagnosable.
This article is an English adaptation and product-positioning version of a Russian vc.ru note, rewritten for Agntbase.com and expanded with direct crawler documentation.