A robots.txt change alerts at the next check that reads the file. A page that picks up a noindex tag alerts once two reads of it agree. The robots.txt alert shows the changed lines, and the noindex alert quotes the directive the page sent.
Changes this covers
A release can change what search engines may index without changing anything a visitor sees. The staging robots.txt is copied to production, a noindex tag in a shared template reaches live pages, or a server rule adds an X-Robots-Tag header. The site keeps loading, so the uptime check stays green. The robots check watches two things for each site you list: the /robots.txt file on its host, and whether the page at the address you added asks search engines not to index it.
URLs to add
Each URL you add is a site, and the page check reads that address only. Add the home page, the landing pages that earn the traffic and one page from each template, such as an article, a product and a category. A redirect is followed to its end, so list the address your visitors land on. Each URL uses one of your plan's sites, and the smallest plan has 20 sites.
The robots.txt file is read from the host of each URL, so URLs on one host share a file. Each listed URL still runs its own check, which means one edit to the file sends one alert for every URL on that host that has the check on. Expect several alerts for one edit, and route them to a channel your team reads. For one alert per edit, turn the check on for one URL per host, so only those URLs get the page reading.
Turn the robots check on
Switch on Robots.txt Monitoring
The check is off for new sites. Edit the site, turn on its Robots.txt Monitoring row and save. The default is a check every 6 hours, and you can choose every hour, every 6 hours, every 12 hours, every day or every week. The interval is roughly the longest you wait for an alert, and in our experience a shorter one suits the pages that earn the traffic.
Read the baseline
After the first check, the site's Robots.txt card shows an Indexable or Blocked badge for the file and a Page Indexing row that reads Allowed, Blocked by noindex or Unknown. The first reading becomes the baseline and sends no alert, so fix anything that reads Blocked and shouldn't before you rely on the alert.
Send the alert to the people who deploy
On the site's Notifications page, open a notification or choose Add Notification and make sure Content changes is on. That switch carries robots.txt and noindex alerts, along with content, DNS and WHOIS change alerts, through whichever channel the notification uses: email, Slack, Discord, a webhook or browser push.
What the alert says
The subject names what changed and the site's address: Robots.txt Changed, Noindex Added, Noindex Removed, or Robots.txt and Page Indexing Changed when one check finds both. For the page, the body gives its indexing state, where the directive was found (the meta tag, the X-Robots-Tag header or both) and the directive text. For the file, it gives the Indexability line, a warning when search engines went from allowed to blocked or back, and a line diff. This is the alert for a check that found both, shortened.
Subject: 🤖 Robots.txt and Page Indexing Changed: https://client.example/pricing
Website: https://client.example/pricing
Status: robots.txt content changed and the page's noindex directive was added
Indexability: Blocked
Page indexing: Blocked by noindex (meta tag)
Directives found:
meta robots: noindex
⚠️ IMPORTANT: This page now carries a noindex directive in its meta tag.
Search engines may drop it from their results.
⚠️ IMPORTANT: Site indexability changed from ALLOWED to BLOCKED.
Search engines may de-index this site.
Changes:
--- Previous
+++ Current
@@ -1,2 +1,2 @@
User-agent: *
-Disallow:
+Disallow: /
...
View details in OctoPing: <link to the site>Webhooks carry the same alert as the robots_changed event. A client who is a contact on a status page that shows the site gets it too, in plainer words and as one email for each URL. See robots.txt monitoring for the check in short form.
When an alert arrives
Read what changed
The subject says whether the file, the page or both changed. For the page, the body quotes the directive the page sent. For the file, the diff shows the lines that changed, and the Blocked flag is only a summary of it that looks for Disallow: / under User-agent: *.
Check the page as served
Open the URL and view its source rather than the rendered page, because OctoPing reads the HTML as the server sends it. To see an X-Robots-Tag header, request the URL with a tool that shows response headers.
Send the evidence to the developer
Forward the alert to whoever shipped the release. It names the site and the change and, for robots.txt, shows the lines that differ.
Wait for the second alert
After the fix, Noindex Removed follows once two reads of the page agree, or a Robots.txt Changed alert whose diff reverses the first. A change you planned alerts too, so the alert also confirms your own change went live.
What the check reads
Related pages
The guide to catching a robots.txt block before it ships tests the alert on a staging site. For an agency running this across client sites, read SEO monitoring for agencies. After a host or platform move, follow website migration monitoring.
Who builds OctoPing
OctoPing is built by Axiolo, which uses it to watch its clients' marketing websites and landing pages. The checks, the client status pages and the monthly summaries are the ones Axiolo relies on for that work.
How it fits the price
Every plan runs the robots check on every site, along with the other checks and unlimited status pages. Plans start at $19 a month for 20 sites. See pricing for the full list.
Every account starts with a 14-day free trial. No card required. Start your 14-day trial and add your home page with the robots check on to see it working.
Last reviewed
Related
- Featurerobots.txt and noindex monitoring with alerts for accidental blocksGet an alert when a site's robots.txt changes or its page picks up a noindex tag, with a line diff and a warning when search engines go from allowed to blocked.
- GuideCatch a site-wide robots.txt or noindex block before it hurts trafficGet an alert when a client's robots.txt changes or the page gets a noindex tag, with a plain warning if search engines went from allowed to blocked, and learn what the check does not read.
- ForSEO monitoring for agencies: noindex, robots.txtGet an alert when a client page picks up noindex, robots.txt changes, the site moves network or nameservers, or a domain nears expiry, with the evidence in it.
- Use caseWebsite migration monitoring checklistConfirm a host or platform move with a maintenance window, hosting and DNS alerts, SSL, robots.txt and noindex checks, and uptime with failure reasons.
- DocsChecks referenceWhat each of the eight checks and heartbeats measures, whether it is on by default, how often it runs and what counts as a failure or a change.