Catch a site-wide robots.txt block before it hurts search traffic

A deploy copies the staging robots.txt to production, the site keeps loading and every uptime check stays green. Search engines read the new rule on their next visit, and in our experience the first sign is a drop in traffic days or weeks later. This guide turns on the robots.txt check, routes its alert to the people who deploy, and explains what the alert can and can't see, so you know how far to trust it.

What this check reads

OctoPing fetches the site's /robots.txt and compares it with the previous copy. It reads that file only. It doesn't look at noindex meta tags or at X-Robots-Tag headers on individual pages, so a page that blocks indexing in its own markup isn't caught here. Use it for the site-wide block that a copied file causes.

User-agent: *
Disallow: /

What you need

• A role of Member, Admin or Owner to edit the site and its notifications.
• A site where the robots.txt is correct right now. The first fetch is the baseline and sends no alert, so a file that is already wrong when you switch the check on is treated as normal.
• The email address or channel of whoever ships deploys for that client.

Turn it on

1. 1. Switch on robots.txt monitoring

   The check is off for new sites. Open the site, choose Edit and switch on the Robots.txt Monitoring row. Check every starts at every 6 hours and offers 1 hour, 6 hours, 12 hours, 1 day, 1 week. An alert arrives at the next check after the file changes, so pick an interval shorter than the time you can afford to be blocked. Choose Save Changes.

2. 2. Let the first fetch set the baseline

   After the first check, the site's Robots.txt card shows a Stable badge, an Indexable or Blocked badge and a Content Preview of the file. If it says Blocked and shouldn't, fix the file before you rely on the alert.

3. 3. Send the alert to the right people

   On the site's Notifications page, open a notification or choose Add Notification. Robots.txt alerts follow the Content changes switch under Alert on, so make sure it's on. The alert goes out through whichever channel the notification uses: email, Slack, Discord, a webhook or browser push.

What the alert says

The subject reads Robots.txt Changed with the site's address. The body says whether indexing is Allowed or Blocked, shows a line diff with removed lines marked by a minus and added lines by a plus, and adds a warning when the flag changes. This is the body for the example above, shortened.

Website: https://client.example/
Status: robots.txt content changed

Indexability: Blocked

⚠️ IMPORTANT: Site indexability changed from ALLOWED to BLOCKED.
Search engines may de-index this site.

Changes:
--- Previous
+++ Current
@@ -1,2 +1,2 @@
 User-agent: *
-Disallow:
+Disallow: /

...

View details in OctoPing: <link to the site>

When the file goes back to open, the warning says search engines can now index the site. Any change to the file sends an alert, including a removed sitemap line, and in that case the Indexability line stays at Allowed and there's no warning.

When the Blocked flag appears

The flag is a simple scan, and knowing its edges tells you when to read the diff yourself. We ran each of these through the check on a test site.

The scan looks for the exact text User-agent: * and then reads the 9 lines after it for a Disallow: with nothing but a slash after it. Three things follow. The match is case-sensitive. The scan doesn't stop where the group ends, so a Disallow: / aimed at one crawler can read as Blocked when it sits within those lines. And a block that starts with a lowercase or differently written group still shows up in the diff, which is why the diff matters more than the flag.

Verify it works

• Use a staging site you control. Switch the check on, let the baseline set, then change the file the way a bad deploy would.
• At the next check, the alert arrives with the diff. Restore the file and a second alert confirms it.
• A check that fails with a server error isn't treated as a change. A 404 is read as an empty file, so a file that disappears does alert.