2026-08-03 · Jack Stovell

AI crawler user agents: the full list (GPTBot, ClaudeBot, PerplexityBot and more)

The AI crawler user agents to know are: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Google-Extended (Google); Applebot and Applebot-Extended (Apple); meta-externalagent and meta-externalfetcher (Meta); Amazonbot (Amazon); CCBot (Common Crawl); and Bytespider (ByteDance). Each is a distinct robots.txt token, and they split into three roles: training crawlers, search-index crawlers and user-triggered fetchers. Every entry below was verified against the operator's own vendor docs on 2026-08-03.

OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User

OpenAI documents three main agents, one per role, plus an ads checker.

GPTBot — OpenAI — crawls for training generative AI foundation models; disallowing it signals your content should not be used in training — respects robots.txt per OpenAI's docs.
OAI-SearchBot — OpenAI — builds the index behind ChatGPT search; sites that opt out are not shown in ChatGPT search answers — respects robots.txt.
ChatGPT-User — OpenAI — fetches a page when a ChatGPT user asks about it (also used by Custom GPTs) — OpenAI says robots.txt rules may not apply because the fetch is user-initiated.
OAI-AdsBot — OpenAI — visits only pages submitted as ads, to validate landing pages; not used to train models.

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

Anthropic mirrors the same three roles and says all its bots honour robots.txt, including the non-standard Crawl-delay extension.

ClaudeBot — Anthropic — collects web content that may contribute to training Claude models — respects robots.txt.
Claude-SearchBot — Anthropic — indexes the web to improve the relevance of Claude's search results — respects robots.txt.
Claude-User — Anthropic — fetches a page when a Claude user asks about it — respects robots.txt, which is unusual: Anthropic applies it even to user-triggered fetches.

Perplexity: PerplexityBot and Perplexity-User

Perplexity runs no training crawler at all, per its own docs.

PerplexityBot — Perplexity — surfaces and links websites in Perplexity search results; explicitly not used to crawl content for AI foundation models — respects robots.txt.
Perplexity-User — Perplexity — visits a page when a user asks a question about it — Perplexity states this fetcher generally ignores robots.txt because a user requested the access.

Google: Google-Extended is a token, not a crawler

Google-Extended has no user-agent string of its own; crawling happens with Google's existing agents, and the token exists purely as a robots.txt control.

Google-Extended — Google — controls whether your content is used to train future Gemini models and for grounding in Gemini apps and Vertex AI — honoured in robots.txt as a control token.
Googlebot — Google — the Search crawler, and one of the few that renders JavaScript. Blocking Google-Extended does not affect inclusion or ranking in Google Search, per Google's docs.

Apple: Applebot and Applebot-Extended

Apple uses the same crawler-plus-control-token pattern.

Applebot — Apple — crawls for Spotlight, Siri and Safari search, and may render pages in a browser — respects standard robots.txt directives; it follows Googlebot rules if Applebot isn't named, and ignores Crawl-delay.
Applebot-Extended — Apple — a control token, not a separate crawler: disallowing it stops Applebot-crawled data being used to train Apple's foundation models, while your content stays discoverable in Spotlight, Siri and Safari.

Meta and Amazon crawlers

meta-externalagent — Meta — crawls for training foundation AI models and improving products — controllable via robots.txt per Meta's docs.
meta-externalfetcher — Meta — fetches individual links a user requested — Meta states it may bypass robots.txt.
facebookexternalhit — Meta — generates link previews when something is shared; not an AI crawler — may bypass robots.txt for security and integrity checks.
Amazonbot — Amazon — improves Amazon products and services and may be used to train Amazon AI models — respects the Robots Exclusion Protocol.
Amzn-SearchBot — Amazon — improves search experiences in Amazon products; does not crawl for generative AI training — follows robots.txt.
Amzn-User — Amazon — supports user-initiated actions such as live Alexa answers; no generative AI training — may not follow all robots.txt directives because a user initiated the request.

Common Crawl and ByteDance

CCBot — Common Crawl, a non-profit — builds the open web archive that many AI labs have used as training data — respects robots.txt; Common Crawl itself warns that other crawlers impersonate CCBot.
Bytespider — ByteDance — no official documentation exists, so its purpose and robots.txt compliance are unverified; independent reports describe an AI training crawler that ignores robots.txt. Block it at the edge if needed — see how Cloudflare blocks AI crawlers.

What is the difference between a training crawler and a search crawler?

A training crawler collects content that may end up baked into model weights months later; a search crawler builds the live index an assistant queries when it answers. GPTBot, ClaudeBot and meta-externalagent are training crawlers; Google-Extended and Applebot-Extended are training controls; OAI-SearchBot, Claude-SearchBot and PerplexityBot are search crawlers; and ChatGPT-User, Claude-User and Perplexity-User fetch a single page on a user's request. Retrieval is the path you can influence this week — blocking search and user-fetch agents removes you from AI answers immediately, while blocking training crawlers only shapes future models. The directive syntax itself is covered in robots.txt rules for AI crawlers.

Which AI crawlers should you allow?

Allow the search and user-fetch agents unless you have a specific reason not to; whether to allow training crawlers is a policy call with no visibility cost in AI search. This robots.txt does exactly that, with the training section commented so you can flip it either way:

# robots.txt — AI crawler policy (tokens verified against vendor docs, 2026-08-03)

# Search-index and user-fetch agents: blocking these removes you from AI answers.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /

# Training crawlers and training-control tokens: currently allowed.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: CCBot
Allow: /

# To opt out of AI training instead, delete "Allow: /" in the group above
# and use "Disallow: /", or block agents individually like this:
# User-agent: GPTBot
# Disallow: /

# Bytespider (ByteDance) has no published compliance docs; robots.txt
# alone may not stop it — use edge/server blocking if it must go.
# User-agent: Bytespider
# Disallow: /

User-agent: *
Allow: /

How do you verify a crawler is really who it claims to be?

Check the requesting IP against the operator's published ranges or reverse DNS — anything can put GPTBot in a user-agent string. OpenAI publishes per-bot IP lists (gptbot.json, searchbot.json and chatgpt-user.json on openai.com); Google documents reverse-DNS checks against googlebot.com, google.com and googleusercontent.com, plus IP ranges; Apple resolves real Applebot traffic to applebot.apple.com and publishes a CIDR list; Amazon publishes IP lists for all three agents; Common Crawl recommends verifying because CCBot is impersonated. A crawler that fails these checks is a scraper wearing a name, and robots.txt will not restrain it.

A correct robots.txt only governs compliant crawlers, and being crawlable does not guarantee being cited — it removes one reason you can't be. To see what AI agents actually get from your site, run the checker or start with check if AI can read your website.

Check your AI visibility — free