SEO monitoring for agencies: noindex, robots.txt

You want to hear about a noindex tag or a robots.txt block on a client site before it shows in a traffic report. OctoPing watches the pages you list and alerts you when one asks search engines to leave it out, when a robots.txt file changes, when a site moves to another network or CDN or its nameservers or mail records change, and when a domain nears expiry. Robots.txt changes alert at the next check. Noindex alerts wait for two matching reads.

How a release removes a client site from search

A staging template with a noindex tag gets promoted to production, a robots.txt that blocks every crawler is copied across with a release, or a site moves to a new host and the developer doesn't tell you. Each leaves the site loading, so uptime stays green, and in our experience a traffic report is the first signal.

In a 2019 post on Moz's blog, Jeff Baker, then a director at Brafton, described a release on Thursday evening, August 1, 2019 that put a noindex robots meta tag on every page of the Brafton site. He found out almost three days later, when keyword rankings in his SEO tools disappeared. In the first week the site lost about 33.2% of its search traffic, and search visibility returned to baseline after six weeks.

Our read is that a home page already on the list, with the robots check and Content changes alerts on, would have sent Noindex Added within hours of the release, once two reads of the page agreed. The check runs every 6 hours by default.

The noindex check on the pages you list

On each robots check OctoPing reads the page at the address you added and looks for a noindex or none directive in a robots or googlebot meta tag in the head, or in the X-Robots-Tag response header, and ignores directives aimed at other crawlers. When a reading differs from the last one, OctoPing reads the page again straight away and alerts only if both agree, so a single odd response doesn't set off an alert.

A directive that a script adds later doesn't count, because OctoPing reads the HTML the server returns. An error page isn't read and shows as Unknown. The first reading becomes the baseline and sends no alert, so confirm it reads Allowed after the first check.

Pages to add for each client

OctoPing reads the URLs you add and follows no links from them, so your list decides what you see: indexing directives, robots.txt, hosting, DNS, domain and certificate dates, uptime and page text. The robots, hosting, DNS and content checks start switched off, so turn them on per site. In our experience a useful list is the home page, the main landing page, a second landing page, one article and one product or category page. Each URL is a site and uses one of your plan's sites. For example, at 5 URLs per client:

robots.txt changes and the blocked flag

OctoPing fetches /robots.txt from the host of each URL you list and compares it with the last copy every 6 hours by default. Any change sends an alert with the changed lines, and the alert flags the site as blocked when the file has Disallow: / in a group that starts with User-agent: *. The match is case-sensitive and reads the 9 lines after the first User-agent: * line without stopping where that group ends, so a Disallow: / meant for another crawler can set the flag too, and a block written in lowercase shows only in the diff.

One edit sends one alert for each listed URL on that host that has the check on. See SEO change monitoring for the setup.

Host, DNS and domain changes

• Hosting. OctoPing looks up a site's addresses and the network that owns them, and alerts with the old and new network when it changes or a CDN appears or disappears. A move inside one network sends no alert. See hosting change detection and website migration monitoring.
• DNS. OctoPing compares the nameserver and mail (MX) records and alerts with the old and new records. See DNS monitoring.
• Domain and certificate. Both are on for new sites, and warnings start 30 days before a domain expires and 30 days before a certificate does. The domain check reads the public registration record and, for a domain that stays registered, alerts when the registrar, registrant or nameservers change. Some registries, such as .de, publish no expiry date. See domain expiry monitoring.

The content check on client landing pages

The content check sends the changed lines of a page's main text. It checks every 30 minutes by default. Main text can leave out navigation, buttons, footers and, on some page builders, headlines and prices, so change a line on a client page and see whether the next check reports it before you rely on it. See website change detection.

Status pages and contacts for the client

Add a client's email as a contact on a status page that shows their sites. The client gets those sites' alerts, robots.txt and noindex included, in plainer words, plus a monthly summary, and takes no seat. A contact hears about every site on the page, so give each client a page of their own, and gets one email for each URL, so one robots.txt edit sends 5 emails when the page shows 5 URLs. To send one, show a single URL per host on the page, or turn the robots check on for one URL per host, so that only those URLs get the noindex reading. Contacts receive these alerts when Content changes is on in one of the site's notifications. See status pages and the guide to adding a client as a contact. See also marketing agencies and agencies.

Who builds OctoPing

OctoPing is built by Axiolo, which uses it to watch its clients' marketing websites and landing pages. The checks, the client status pages and the monthly summaries are the ones Axiolo relies on for that work.

How it fits the price

Every plan runs these checks on every site, with client contacts up to the plan's subscriber allowance and as many status pages as you need. Plans start at $19 a month for 20 sites. See pricing for the full list.

Every account starts with a 14-day free trial. No card required. Start your 14-day trial and add a client's home page with the robots check on to see it working.