Crawlers and AI signals

Every search and AI crawler OctoPing checks against robots.txt, the page signals it reads, what each one controls and which may ignore robots.txt.

Last reviewed Nov 9, 2026

On this page

When the robots check is on, OctoPing reads a site's robots.txt the way search engines and AI companies say their crawlers do, and shows which of 27 search crawlers, AI crawlers and control tokens it lets through for the address you monitor. This page lists every crawler the check covers, how OctoPing picks the rules for each one and what a robots.txt check can't see.

How the check decides

OctoPing fetches /robots.txt from the site's host and asks, for each crawler on this page, whether that crawler may fetch the monitored address. The path and the query string count, so a rule for one folder blocks a monitored page inside it and leaves the home page open. The rules are read the way RFC 9309 and Google's documentation describe them:

A crawler follows the group named for it. With none, it follows its company's documented fallback and then the * group. Applebot follows the Googlebot group when robots.txt has no group named for it. YandexBot follows the Yandex group when robots.txt has no group named for it. ClaudeBot follows the shared AI-training group when robots.txt has no group named for it.
YandexAdditional and YandexAdditionalBot ignore the * group, so with no group that names them they have no rule.
Inside a group, the longest matching rule wins and Allow wins a tie. The * and $ wildcards work, and a group with no rules allows everything.
Field names are matched by their start, as Google's parser does, so Disallowed: / counts as Disallow. A line that reads User-agent: * Googlebot is the * group.

When robots.txt can't be read normally

OctoPing sorts the answer to its robots.txt request into five statuses.

robots.txt answers	Status	What OctoPing does
A 2xx answer, or a redirect that ends in one (up to five hops)	Found	Reads the rules. An empty file reads like a missing one: nobody is restricted, and going from no file to an empty one, or back, isn't a change.
404 and 410	No robots.txt	Treats the file as missing, which means no restrictions. The banner reads No robots.txt (HTTP 404): no restrictions, with the code that came back.
401, 403, 429 and every other 4xx	Refused our checker	Reads the file once more. If it is still refused, the banner says robots.txt refused our checker, with the HTTP code. OctoPing keeps the verdicts from the last reading, sends no robots.txt alert and starts no new baseline. A refusal to OctoPing is often a firewall treating the checker differently from Googlebot, and reading it as a removed file would alert on a change nobody made.
Any 5xx	Server error	Reads the file again at once. If it is still an error, OctoPing sends the robots.txt alert, headed Crawling blocked: robots.txt server error, and sends another headed robots.txt is answering again when the file recovers. A one-off error stays quiet. Companies treat an unreadable robots.txt in different ways. Google's documentation says it stops crawling the site for 12 hours and then uses its last good copy for up to 30 days. Amazon says Amazonbot, Amzn-SearchBot and Amzn-User use their cached copy from the last 30 days and then treat the file as absent. The card counts every other crawler as blocked while the error lasts, which is the default in RFC 9309.
A 5xx, a missing file or a page with no robots.txt rules, while the site is down, failing its checks or in a maintenance window, or its page can't be reached in the same check	Not judged	The answer then says more about the outage than about the file, so OctoPing doesn't judge it. It behaves like a refusal: the last verdicts stay, and no robots.txt alert goes out when it starts or when it ends. A robots.txt that names at least one crawler is still judged, so a real change still alerts during an outage. A page read that is only slow doesn't count as an outage.
A network error or a timeout, or an address OctoPing won't check	No new status	The check fails and stores no new reading.

OctoPing reads and stores the first 500 KiB of the file.

Search engines

These 5 decide the search flag. The site reads Blocked for search when robots.txt stops any of them from fetching the monitored address: Googlebot, bingbot, Applebot, DuckDuckBot and YandexBot. A deliberate block of one search engine reads as Blocked for search too, and the card names which.

Token	Company	What it controls	Follows robots.txt	Source
`Googlebot`	Google	Crawls for Google Search, including Discover, Images, News and Search features. AI Overviews and AI Mode are Search features: the page's nosnippet keeps it out of them.	Yes	Google page
`bingbot`	Microsoft	Bing's standard crawler. Bing's index also grounds Copilot, Microsoft 365 and other Microsoft AI answers.	Yes	Microsoft page
`Applebot`	Apple	Crawls for Spotlight, Siri and Safari search. Apple says its data may also feed Apple's foundation models and AI answers (the page's nosnippet opts out of answers).	Yes	Apple page
`DuckDuckBot`	DuckDuckGo	DuckDuckGo's crawler for its search results.	Yes	DuckDuckGo page
`YandexBot`	Yandex	Yandex's main indexing robot.	Yes	Yandex page

AI training

Crawlers and control tokens for collecting content that AI companies may use to train models. Each row says what the company states its token controls.

Token	Company	What it controls	Follows robots.txt	Source
`GPTBot`	OpenAI	Collects content to train OpenAI's generative AI models. Disallowing it says the site's content shouldn't be used for training.	Yes	OpenAI page
`ClaudeBot`	Anthropic	Collects content to train Anthropic's generative AI models.	Yes	Anthropic page
`Google-Extended`	Google	A control token, not a crawler: it decides whether Google may use the content to train Gemini models and to ground Gemini Apps and Vertex AI. It doesn't affect Google Search, AI Overviews or AI Mode.	Control token	Google page
`Applebot-Extended`	Apple	A control token, not a crawler: it decides whether Applebot's data trains Apple's foundation models. It doesn't affect Apple search.	Control token	Apple page
`Meta-ExternalAgent`	Meta	Crawls to train Meta's AI models or to index content for Meta's products.	Yes	Meta page
`Amazonbot`	Amazon	Improves Amazon's products and services and may be used to train Amazon AI models. Amazon also treats the page's noarchive as a training opt-out.	Yes	Amazon page
`CCBot`	Common Crawl	Builds Common Crawl's open repository of web crawl data.	Yes	Common Crawl page

AI search and answers

Crawlers and control tokens that decide whether a page can show up in an AI product's search results and answers.

Token	Company	What it controls	Follows robots.txt	Source
`OAI-SearchBot`	OpenAI	Surfaces sites in ChatGPT's search features. Not used for training.	Yes	OpenAI page
`Claude-SearchBot`	Anthropic	Crawls to improve the search results Claude gives its users.	Yes	Anthropic page
`PerplexityBot`	Perplexity	Surfaces and links sites in Perplexity's search results. Perplexity says it isn't used to train AI models.	Yes	Perplexity page
`Meta-WebIndexer`	Meta	Crawls to improve Meta AI's search results and its citations of pages.	Yes	Meta page
`Amzn-SearchBot`	Amazon	Improves Amazon's search experiences, such as Alexa answers. Amazon says it isn't used for AI training.	Yes	Amazon page
`YandexAdditional`	Yandex	Disallowing it keeps pages out of Yandex Search's AI answers. It doesn't crawl, and it ignores User-agent: * rules, so only a group that names it counts.	Control token	Yandex page
`YandexAdditionalBot`	Yandex	The second Yandex token for AI answers. Like YandexAdditional, it ignores User-agent: * rules, so only a group that names it counts.	Control token	Yandex page

Fetched when a user asks

Fetchers that visit a page when a person asks an AI product about it. Several of their companies say robots.txt rules may not apply to them (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, Google-Agent and Google-GeminiNotebook), so a block here is a request and not protection.

Token	Company	What it controls	Follows robots.txt	Source
`ChatGPT-User`	OpenAI	Visits a page when a ChatGPT user or a custom GPT asks. OpenAI says robots.txt rules may not apply to these requests.	May ignore	OpenAI page
`Claude-User`	Anthropic	Fetches a page when a Claude user asks. Anthropic says its bots honour robots.txt.	Yes	Anthropic page
`Perplexity-User`	Perplexity	Fetches a page when a Perplexity user asks. Perplexity says it generally ignores robots.txt.	May ignore	Perplexity page
`Meta-ExternalFetcher`	Meta	Fetches a single page when a user of Meta's AI products asks. Meta says it may bypass robots.txt.	May ignore	Meta page
`Amzn-User`	Amazon	Fetches live pages for user actions, such as an Alexa question. Amazon says it may not follow all robots.txt rules.	May ignore	Amazon page
`Google-Agent`	Google	Agents running on Google's infrastructure, acting on a user's request. Google says its user-triggered fetchers generally ignore robots.txt.	May ignore	Google page
`Google-GeminiNotebook`	Google	Fetches pages a user adds as Gemini Notebook sources. Google says its user-triggered fetchers generally ignore robots.txt.	May ignore	Google page
`DuckAssistBot`	DuckDuckGo	Fetches pages in real time for DuckDuckGo's AI-assisted answers, which cite their sources. DuckDuckGo says it isn't used to train AI and that disallowing it stops it within 72 hours.	Yes	DuckDuckGo page

The shared AI-training group

Some companies read one shared group, named __usage_ai-training, as a preference about AI training. A * group doesn't cover it, so it counts only when robots.txt names it. OctoPing shows its state as disallowed, allowed or not set, beside the crawler lists, and judges it for the monitored address like any other group. It comes from an IETF draft, so companies adopt it one at a time.

The card words the three states as AI training is disallowed for every company that reads the shared group, AI training is allowed in the shared group, and No shared AI-training preference is set. A change in the state sends the robots.txt alert, and so does a crawler that follows the group, such as ClaudeBot, going between allowed and blocked.

Company	Reads it	What it does	Source
Anthropic	Now	Anthropic follows it for ClaudeBot when robots.txt doesn't name ClaudeBot.	Anthropic page
Microsoft	February 1, 2027	Microsoft reads it for AI training from February 1, 2027. A page's own ai-training tag overrides it for that page.	Microsoft page
Google	Early 2027	Google plans to treat it as disallowing Google-Extended, when Google-Extended isn't named.	Google page

Page signals

On the same request that reads noindex, OctoPing also reads the directives below from the X-Robots-Tag header and from meta tags in the head of the monitored page, when they are addressed to the agents listed for each. The agent robots stands for a robots meta tag, or for an X-Robots-Tag header that names no crawler or names *. The card and the alerts label some of these signals No snippets, No archive and No cache, and the headings below use the directives for those. Each company's effect is the one that company documents, with the dates it gives. A signal that changes sends the robots.txt alert once a second read agrees, the same rule as noindex. The checks reference describes how noindex itself is read, and the first entry adds what it means for AI. A noindex counts for search when it is addressed to everyone or to Googlebot. Its AI effects apply only when it is addressed to everyone or to bingbot, so a googlebot noindex has none.

noindex (AI effects)

Directives: noindex, none. Read when addressed to: robots and bingbot.

Microsoft (until March 1, 2027): Bing also leaves noindex pages out of Microsoft AI training.
Amazon: Amazon's crawlers don't index the page.

nosnippet and max-snippet:0

Directives: nosnippet, max-snippet:0. Read when addressed to: robots, googlebot, applebot and bingbot.

Google: Kept out of AI Overviews and AI Mode, and no text snippet in Search.
Apple: Kept out of Apple's AI answers.
Microsoft: No description or preview in Bing.

noarchive

Directives: noarchive. Read when addressed to: robots and bingbot.

Microsoft (until March 1, 2027): Not linked in Copilot answers and not used for Microsoft AI training. Bing's ai-training and ai-grounding tags replace this on March 1, 2027.
Amazon: Not used for Amazon model training (Amazon reads the robots meta tag).
Google: Google doesn't use it.

nocache

Directives: nocache. Read when addressed to: robots and bingbot.

Microsoft (until March 1, 2027): Copilot shows only the URL, title and snippet, and Microsoft AI training uses only those. Bing's new tags replace this on March 1, 2027.

Bing AI training

Directives: ai-training:disallow. Read when addressed to: bingbot.

Microsoft (from February 1, 2027): Not used for Microsoft AI training. Overrides the robots.txt AI-training preference for this page.

Bing AI grounding

Directives: ai-grounding:disallow, ai-grounding:disallowextended. Read when addressed to: bingbot.

Microsoft (from February 1, 2027): Not used to inform AI answers in Bing, Copilot or Microsoft 365.
Microsoft (from February 1, 2027): Used in Microsoft's own AI answers, but not in the grounding Microsoft sells to other companies.

noai / noimageai

Directives: noai, noimageai. Read when addressed to: robots. Shown as intent only.

States an intent only. No crawler documents honouring it.

What the check can't see

A robots.txt check shows what the file asks for. It doesn't show what happens at the server.

robots.txt is a request. A CDN or a firewall can refuse a crawler that robots.txt allows, and OctoPing can't see that. Seeing it would take the crawler's real IP addresses, so the check doesn't send requests that pretend to be a crawler.
Several fetchers that act when a person asks say robots.txt may not apply to them (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, Google-Agent and Google-GeminiNotebook). The card shows them with a may ignore robots.txt cue, and the count for each group leaves them out, so a block there is never counted as protection.
Google-Extended, Applebot-Extended, YandexAdditional and YandexAdditionalBot are control tokens. They never fetch anything, so the robots.txt rule is the only signal, and OctoPing reads it like any other group.
robots.txt stops crawling, not indexing. A blocked URL can still appear in search results without its content. The noindex check covers the page side.

How the list is kept

A crawler is on this page only when its company documents the name on its own site. Each row links to that page, and OctoPing re-reads every source at least every 90 days. The oldest check behind this page is from November 9, 2026. The check judges 5 search crawlers and 22 AI crawlers and control tokens, so a crawler that isn't listed isn't judged.

Crawlers and AI signals

Related