Alert rules

An alert is a decision, and it's made per notification. This page describes what decides it: the failure count, the repeat interval, the four triggers, recovery, the log and what a maintenance window holds back. How each channel delivers a message is on the pages for email, Slack, Discord, browser push and webhooks.

How a down alert is decided

A website notification belongs to one site, and a site can have any number of them. Each one follows the same steps on its own.

1. A scheduled check of the site fails. The site's failure count goes up by one. A successful check sets the count back to zero.
2. If the probe regions can confirm or contradict the failure, OctoPing waits for them, for up to 10 minutes. A failure that every region contradicts sends no down alert. The vantage points page describes the verdicts.
3. The notification compares the failure count with its Retry Threshold. Below the threshold, nothing is sent.
4. At or above the threshold, the notification compares the time since it last sent anything with its Notification Interval. If the interval hasn't passed, nothing is sent.
5. Otherwise the message goes out, and the time is stored as the notification's last sent time.

This runs on every failing check, which is how a long outage produces repeat alerts: each failing check asks the same question again, and the answer changes once the interval has passed.

Delivery settings

Both settings sit under Delivery settings in the notification form, and apply to down alerts and to heartbeat missed alerts.

An example with the defaults

A site checked every 5 minutes, with a Retry Threshold of 3 and a Notification Interval of 10 minutes, goes down and stays down. Minute 0 is the first failed check. Check times are approximate, and the example assumes the probe regions confirm the failure.

The first alert arrives about 10 minutes after the site first fails. A longer check interval or a higher threshold moves it later by the same arithmetic.

The four triggers

Every website notification has four switches. All of them start on. Each alert type belongs to exactly one switch, and a notification with the switch off never sends that alert. The Event column is the value of the event field in a webhook payload and of the Trigger column in the notification log.

The labels are shorter than what they cover. Site goes down also controls expiry and slow-response alerts, and Content changes also controls DNS, robots.txt, WHOIS and status-code alerts. Turn off Site goes down and the notification stops warning about an expiring certificate as well.

When each alert is sent

Which alerts use the threshold and the interval

• down uses both, as described above. primary_blocked uses the threshold, goes out once for a run of failures and doesn't use the interval.
• slow_response goes out at the first slow response and then repeats no more often than the interval while responses stay slow. It needs a Response Time Alert Threshold on the site.
• ssl_expiring goes out at most once in any 24 hours for a site, however many notifications it has.
• The remaining alerts have no threshold and no interval. They go out when the check finds the change or reaches the expiry window.

Recovery alerts

• A recovery alert is sent only after a down alert was sent for the same outage. A site that failed and recovered before any notification reached its threshold sends no recovery.
• It goes to every enabled notification with Site recovers on, and it ignores the interval.
• It's sent when the first successful check arrives.

Heartbeat notifications

A heartbeat notification has two switches, and no others. Both start on.

The Retry Threshold and Notification Interval work as they do for a site, with one difference in what is counted. OctoPing evaluates every heartbeat's deadline every minute and counts each evaluation that finds the ping late. With the default threshold of 3, the alert goes out on the evaluation that makes the count 3. The ping deadline itself is the heartbeat's interval plus its grace period, described on the checks page.

Maintenance windows

While a site or a heartbeat is inside a maintenance window, the alerts that report unavailability and slowness are held back. Alerts about something that planned work doesn't explain still go out.

• Each held-back alert is written to the log as Skipped with the error text maintenance for every notification that would have sent it.
• An outage that already sent a down alert before the window started stays declared. Its repeat alerts are held back, and its recovery alert is sent when the site recovers, even inside the window.
• Failures during a window still count. A site that is still failing when the window ends sends its alert on the first failing check after it, as long as its threshold is already met.

The default notification

Adding a website or a heartbeat also creates a notification named Default Email Notification. It emails every member of the workspace at the moment the site is added, with a Retry Threshold of 3, a Notification Interval of 10 minutes and every switch on. It's an ordinary notification: edit it, disable it or delete it. A member who joins the workspace later isn't added to notifications that already exist.

The notification log

Every attempt is recorded. On a website, open the site, choose Notifications, then View Logs. The page shows the totals for Sent, Failed and Skipped, and a table with the columns Notification, Trigger, Status, Recipients and Error.

The Trigger column holds the event name from the tables above, or test for a test send. Choosing Send Test on a notification bypasses the threshold, the interval and maintenance windows, and records the result the same way. Log rows are monitoring history, so they're trimmed with the workspace's other activity data to its retention period. Heartbeat notifications are logged too, but the app shows the log page for websites only.

Who can change notifications

• Viewers can see notifications and the log.
• Members, Admins and Owners can create, edit, enable, disable, delete and test them.
• Connecting Slack to the workspace takes an Admin or an Owner.

Status page contacts

Contacts added to a status page are a separate list, and they're mailed from the same decisions. A contact hears about an outage only when one of the site's notifications is about to send the down alert, so a retry threshold, a failure the probes contradicted or a maintenance window applies to contacts as well.