See which AI crawlers your robots.txt allows

You want to know which AI crawlers a client's robots.txt lets in, and to hear when that changes. OctoPing reads the file the way each crawler's company says its crawler does, and shows 22 AI crawlers and control tokens as allowed or blocked for the address you monitor. This guide turns the check on, reads the card, tests it on a staging site and sets the alert. It also says what the card can't see, so you know how far to trust it.

What this check reads

OctoPing fetches the site's /robots.txt and judges each crawler against the address you monitor, path and query string included. The AI crawlers and control tokens fall into three groups:

• AI training. GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot and CCBot.
• AI search and answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot, YandexAdditional and YandexAdditionalBot.
• Fetched when a user asks. ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, Google-Agent, Google-GeminiNotebook and DuckAssistBot.

A crawler is on the list only when its company documents the name on its own site. The crawler reference lists each one with the company's page, and OctoPing re-reads those pages at least every 90 days. The card also shows the shared __usage_ai-training group and the signals the monitored page sends. A crawler that isn't listed isn't judged.

What you need

• A role of Member, Admin or Owner to edit the site and its notifications.
• A site on the address whose rules you want to read. robots.txt rules can differ by folder, and the card judges the address you monitor, so add the page or folder home that matters to the client.
• The email address or channel of whoever manages the client's site.

Turn it on

1. 1. Switch on robots.txt monitoring

   The check is off for new sites. Open the site, choose Edit and switch on the Robots.txt Monitoring row. Check every starts at every 6 hours and offers 1 hour, 6 hours, 12 hours, 1 day, 1 week. Choose Save Changes.

2. 2. Read the search engines first

   After the first check, open the site's Robots.txt and AI access card. The Search engines row shows each of Googlebot, bingbot, Applebot, DuckDuckBot and YandexBot as allowed or blocked, and the site reads Blocked for search when any of them is stopped. A banner at the top of the card says when robots.txt is missing, refused our checker or is answering with a server error. Fix a search block that shouldn't be there before you read the AI section.

3. 3. Read the AI access view

   Companies sit in rows and the three groups in columns. A cell shows whether robots.txt allows or blocks that company's crawler in that group, and is empty where the company has none. Each group has a count of how many of its crawlers are blocked, and the count leaves out the fetchers marked may ignore robots.txt. Those (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, Google-Agent and Google-GeminiNotebook) are fetchers whose company says robots.txt may not apply to them, so a block there is a request. Open a crawler to see what it does, any fallback group it used and the company's page.

4. 4. Check the shared group and the page signals

   The shared __usage_ai-training group reads as disallowed, allowed or not set, with the companies that read it: Anthropic today, Microsoft from February 1, 2027, Google from early 2027. The page signals show what the monitored page sends, such as nosnippet or noarchive, and what each does for the companies that document it. noai and noimageai show as intent only, because no company documents honouring them.

5. 5. Send the alert to the right people

   On the site's Notifications page, open a notification or choose Add Notification. Robots.txt alerts follow the Content changes switch under Alert on, so make sure it's on. The alert goes out through whichever channel the notification uses: email, Slack, Discord, a webhook or browser push.

Test it on a staging site

Use a staging site you control, and let the first check set the baseline. Then publish this file and wait for the next check. The site isn't blocked for search, GPTBot reads as blocked, and the other AI crawlers read as allowed.

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

Remove the second group and a second alert confirms the change back. When a crawler on your own file reads differently from what you expected, open it to see any fallback group it used, and read the changed lines in the alert to see what the check saw.

What the alert says

A change in which crawlers robots.txt lets in sends the Robots.txt Changed alert. It lists each crawler that flipped, with the AI crawlers grouped by category, for example GPTBot going from allowed to blocked, and it shows the changed lines of the file. A change to the shared group gets a line of its own. This is the alert for the test file above, shortened.

Website: https://staging.client.example/
Status: robots.txt content changed

Indexability: Allowed
Page indexing: Allowed

AI crawler access changed:
  AI training
    GPTBot (OpenAI model training): allowed → blocked

Changes:
--- Previous
+++ Current
@@ -1,2 +1,5 @@
 User-agent: *
 Allow: /
+
+User-agent: GPTBot
+Disallow: /

...

View details in OctoPing: <link to the site>

The first reading is the baseline and sends no alert. OctoPing compares each reading with the previous one judged by today's rules, so a change in how OctoPing reads robots.txt, or a crawler added to the list, doesn't raise an alert on its own. If robots.txt starts answering with a server error that lasts past a second read, the alert is headed Crawling blocked: robots.txt server error. Companies treat an unreadable file in different ways, and the card counts the crawlers as blocked while the error lasts, except Amazon's, which use their cached copy from the last 30 days first.

What the card can't tell you

• robots.txt is a request. A CDN or a firewall can refuse a crawler that robots.txt allows, and OctoPing can't see that, because it would take the crawler's real IP addresses. The check doesn't send requests that pretend to be a crawler.
• ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, Google-Agent and Google-GeminiNotebook are fetchers whose companies say robots.txt may not apply to them. The card never counts them as protection.
• Google-Extended, Applebot-Extended, YandexAdditional and YandexAdditionalBot are control tokens. They never fetch anything, so the robots.txt rule is the only signal they carry.
• robots.txt stops crawling, not indexing. A blocked URL can still appear in search results without its content, and the noindex check covers the page side.