You get an alert with the changed lines at the next check after a client's robots.txt changes, and a plain warning if search engines went from allowed to blocked.
The staging file that shipped to production
Staging sites often carry a robots.txt that blocks every crawler. When a deploy copies that file to production, the site still loads, the uptime check stays green and nothing looks wrong. Search engines read the new rule on their next visit. In our experience the first symptom is a drop in traffic days or weeks later.
User-agent: *
Disallow: /What OctoPing checks
OctoPing fetches /robots.txt from the address of the site you monitor and stores the whole file. The default is one fetch every 6 hours, and you can choose every hour, every 6 hours, every 12 hours, every day or every week. robots.txt checks start switched off, so you turn them on for the sites where search traffic matters.
Each fetch is compared with the previous one. Any change to the file counts, from a new sitemap line to a rewritten rule. OctoPing also flags the site as blocked when it finds Disallow: / in the User-agent: * group, and allowed otherwise. A missing robots.txt is stored as an empty file and reads as allowed, so deleting the file shows up as a change when there was one before. The first fetch sets the baseline.
What the alert says
The subject reads Robots.txt Changed with the site's address. The body states whether indexing is allowed or blocked and shows a line diff of the file, with removed lines marked by a minus and added lines by a plus. When the flag flips, the body adds a warning that search engines may de-index the site, or that they can now index it.
Website: https://example.com
Status: robots.txt content changed
Indexability: Blocked
⚠️ IMPORTANT: Site indexability changed from ALLOWED to BLOCKED.
Search engines may de-index this site.
Changes:
--- Previous
+++ Current
@@ -1,2 +1,2 @@
User-agent: *
-Disallow:
+Disallow: /
View details in OctoPing: <link to the site>Robots.txt alerts use the change-alert switch on each alert rule, so they reach the same email, Slack, Discord, webhook or browser push channels as your other change alerts. If a client is a contact on a status page that shows the site, they get the alert as well. A fetch that fails with a server error isn't treated as a change.
A rewritten robots.txt is one way a page quietly changes. Website change detection watches the page text itself.
How it fits the price
Every site gets robots.txt monitoring along with uptime, SSL and the other checks, and no plan limits which checks a site can use. Plans start at $19 a month for 20 sites. See pricing for the full list.
Every account starts with a 14-day free trial. No card required. Start your 14-day trial, add a client's site, and the checks you turn on start with it.
Last reviewed
Related
- FeatureWebsite change detection with a diff in every alertWatch a page for text changes and get the changed lines in the alert. OctoPing compares readable text instead of raw HTML to keep false alarms down.
- DocsAlert rulesHow a failed check becomes an alert: the retry threshold, the repeat interval, the four triggers, recovery, the default notification, the log and what a maintenance window holds back.
- GuideCatch a site-wide robots.txt block before it hurts search trafficGet an alert when a client's robots.txt changes, with a plain warning if search engines went from allowed to blocked, and learn what the check does not read.