Engineering · 10 min read
From one failed request to one page: how SutramX decides a monitor is down
The rules SutramX applies between a failed check and a down alert, and the detection time each of them costs.
Published · By Kumar, QuantumPlug Technologies LLP

An uptime monitor is judged on one moment: the alert that wakes someone up. If it fires on a blip, the team learns to ignore it. If it waits too long, customers find the outage first. In SutramX, the uptime monitor we build and run, the path from a failed HTTP request to a down alert passes through a series of explicit gates. Each one is a small, deterministic function with named outcomes and its own tests. This article walks through those gates in order, using the rules as they are written in the code, and is honest about the detection time each gate adds.
Gate one: re-check, but only when a re-check can change the answer
The first defence against a false page runs inside a single check. A failed attempt is not reported straight away. Instead recheckPlanFor(errorType, httpStatus) decides whether a second attempt from the same location is worth making, and how. The answer depends on the class of failure, not on a fixed delay.
Network-class failures (connection refused or reset, connect timeouts, DNS resolution failures, unreachable hosts) are re-checked after 3 seconds, with a fresh DNS lookup and a connection to the next resolved address, so one bad backend behind round-robin DNS doesn’t count as an outage. Server errors and read timeouts wait 5 seconds, giving a struggling upstream a moment to recover. Anything else that fails, such as a response slower than the monitor’s limit, gets one quick re-check after 2 seconds.
Some failures are never re-checked: an expired or mismatched TLS certificate, a 4xx status, a missing keyword, a redirect loop, a bot-protection block. These are deterministic findings about a response we did receive, and asking the same place again returns the same answer. Other regions confirm them instead.
The re-check also has to respect the schedule. A single attempt may take at most the interval minus 3 seconds, and the whole check, re-check included, must finish within the interval minus 1 second. A timeout is only retried if the full timeout still fits. If the re-check merely timed out because it had a shortened budget, the original failure is reported rather than a weaker second opinion. Heartbeat and browser monitors, and intervals under 15 seconds, skip this step entirely.
Gate two: more than up and down
A monitor with only two outcomes has to treat every failed request as evidence against the target. SutramX stores five statuses: up, degraded, down, blocked and unknown. When an incident vote is counted, up and degraded become up (slow is not down), down stays down, and blocked and unknown abstain.
blocked means the target’s WAF or bot protection refused the check. The edge answered, but we couldn’t see past it. unknown means our own checker failed: a resolver problem, a missing IPv6 route, an overloaded event loop or a region we’ve marked impaired. Neither is evidence of an outage, so neither opens or resolves an incident, and for uptime percentages both count as up. Latency averages leave block pages out, because their timing belongs to the edge, not the site. classifyCheckStatus() will only store blocked or unknown when the matching error type is present. Any other result claiming those statuses is treated as a failure, so a bug cannot quietly turn an outage into an abstention.
Abstention has a limit. When every location has been refused by bot protection for five checks in a row, nobody can see the site at all, and staying silent could hide a real outage behind the firewall. At that point the block is promoted to an ordinary failure, with a message saying so. The opposite direction is closed: a block page never resolves an open incident, and escalation pauses while blocks are the only evidence, because “still down” is not something we can claim.
Before a region’s failure can vote, it also has to persist. The monitor’s failure threshold (1 to 10 consecutive failures, default 1) applies, and new monitors on short intervals get a confirmation period. Under 2 minutes it is twice the interval, clamped between 30 and 60 seconds, so a 15-second monitor needs 30 seconds of continuous failure and a 30- or 60-second monitor needs a minute. Any passing check restarts the clock.
Gate three: a quorum of regions
Each plan has a down quorum, edited in the plan catalogue and seeded at 1 for Free, 2 for Starter, 3 for Growth and 4 for Pro. It is capped at the number of locations the monitor actually uses, so a Pro monitor checked from three chosen locations needs all three. Each region’s vote is its latest result inside a freshness window of roughly twice the interval plus 30 seconds. An older result has no vote.
The decision itself is a pure function, decideQuorum(). It first builds the healthy set: configured regions with a live checker, minus regions whose latest result abstains, minus live regions that have no fresh result for this monitor. That last rule exists because a probe can keep heartbeating while producing nothing for one monitor, and a single stale vote should not decide. Coverage is healthy when the healthy set is at least as large as the quorum. A down decision needs healthy coverage and at least quorum agreeing regions.
One subtle case shows why the rules are written so carefully. The failure threshold applies to every agreeing region, not just the one whose result has just arrived. Early on, a region whose streak was still too short had its vote dropped and was then treated as missing. That shrank the healthy set and let the stricter fallback described below open an incident from two regions on a three-of-three monitor. The fix keeps such a region in the coverage count as a live region that simply doesn’t agree yet.
Quorum has a cost: in principle, the other regions might not check again for a whole interval. So when a region records a fresh failure and no incident is open, every other configured region is asked to check now. The quorum is then decided on fresh votes within seconds rather than at the next scheduled slot.
When the fleet is thin: reduced coverage and second opinions
If enough of a monitor’s regions are offline that the quorum can no longer be reached, decideQuorum() refuses to confirm a down decision and reports blockedByCoverage. A thinned-out fleet must not let one flaky vantage point become a customer-facing outage. A real outage during a probe outage still has to be reported, though, so the caller then tries decideReducedCoverageDown(), which is stricter on purpose. Every remaining healthy region must vote down, each with at least two consecutive failures by default. If the built-in checker has a fresh passing result of its own, the incident does not open. Each refusal has a name: no_healthy_region, missing_vote, region_up, streak_too_short or builtin_up. That makes the logs and tests readable.
When only one region remains, or a monitor has only one location (every Free monitor), a single vantage point’s word is not enough. Up to two other live checkers are asked for a one-off verification check. Their answers are stored as verification votes, never as the monitor’s own check history. One confirmer seeing the failure opens the incident. Confirmers seeing the site up mean a regional problem, not downtime. A down answer from a region that is impaired right now is read as inconclusive. Requests wait up to 120 seconds by default.
The hardest case is a failure seen only by a remote probe whose confirmers never answer. The code treats that as unconfirmed and opens nothing, but only up to a ceiling: once the failure has lasted three consecutive checks and 10 minutes, the incident opens anyway, and the alert says it could not be confirmed from a second location. The code comment states the principle plainly: a missed real outage is worse than a false alert. Every suppression on this path writes a warning and an hourly platform event, so we can see how often it happens.
Recovery is easier than going down, but not too easy
Recovery uses a different threshold: a majority of the live regions, Math.min(quorum, Math.floor(healthy / 2) + 1), never more than the down quorum. That lets an incident resolve while one region is still offline, instead of staying open until that region comes back.
There is a guard on top. Recovery is never confirmed while the down quorum still holds. With many regions, both thresholds equal the plan quorum, so during a partial outage, say 8 regions with 4 down and 4 up, the healthy half alone used to recover the incident. It then reopened as flapping and went quiet for the rest of the outage. Now, if the regions still failing would by themselves confirm an outage, the incident stays open. Recovery has its own consecutive-pass threshold, and new monitors on intervals of 60 seconds or less must stay up for 60 seconds before the incident resolves.
Flapping: one incident, not twelve
A target that oscillates between up and down every half minute should produce one incident, not a page every half minute. If a monitor goes down again within the cooldown (5 minutes by default) after an incident resolved, SutramX reopens that same incident and marks it flapping, rather than creating a new one.
The repeat down alert is suppressed only when the customer has already been told something about this incident. If the earlier alert never reached anyone, for example because it fell inside a maintenance window, the reopen alerts like a new incident, so suppression cannot hide an outage altogether. If a flapping incident then stays down for a full cooldown and the last thing the customer heard was “recovered”, shouldRealertFlappingIncident() sends a fresh alert. Each alert cycle carries its own number in every delivery deduplication key, so the second down alert and its recovery are not dropped as duplicates of the first pair. The flapping flag clears once the incident has stayed resolved for a full cooldown.
The last check before the page
Between an incident opening and its alert going out, the target may already be back: a deploy blip, or one of the deterministic findings that never got an in-check retry. So immediately before dispatch, the built-in checker runs one fresh check. If it still fails, the alert is sent with a “re-verified” line on every channel. If it passes, the alert is deferred and a sweeper tries again in roughly 45 to 105 seconds. After two deferrals or 300 seconds, the alert is sent anyway, because the regular checks still say down and they outvote a single fresh check.
Errors, timeouts, blocked or unknown results and impaired regions all mean send. This step fails open: it can delay an alert by a bounded amount, but it can never cancel one. There is deliberately no language model in this path. As the code notes, an LLM here could only add latency and cost, and a hallucinated “looks fine” would hide a real outage.
What it costs in detection time
None of this is free, and the trade-offs can be read straight from the rules. Take a new HTTP monitor on a 60-second interval that starts returning 503s. The in-check re-check adds 5 seconds. The confirmation period means the failure must persist for a minute before a region’s vote can open anything. Cross-region confirmation gets the other regions’ votes immediately rather than at their next slot, and pre-send verification typically costs one more attempt. On paper, that is around a minute or so between the first failed request and the page, in exchange for not paging on a single bad response. A single-location monitor whose confirmers cannot answer sits at the far end of the trade-off, bounded by the 10-minute ceiling.
These are defaults, not fixed rules. Thresholds, confirmation and recovery periods are per-monitor settings. A team that wants to be paged on the first failure from any region can choose a single location and a failure threshold of 1, accepting the noise that comes with it.
The useful lesson isn’t any one threshold. It is the structure: each decision is a pure function with named outcomes, every suppression has an upper bound, alert-path safeguards fail open, and each trade-off is logged so it can be measured rather than argued about. We apply the same discipline to client systems with consequential automated decisions, such as payments, notifications and access. The rules sit where people can read and test them, and the system can explain why it acted.
Common questions
Does SutramX retry every failed check?
No. Network errors are re-checked after 3 seconds with fresh DNS, and 5xx errors and read timeouts after 5 seconds. Other failures, such as slow responses, get one re-check after 2 seconds. TLS errors, 4xx responses, missing keywords and bot-protection blocks are not re-checked, because the same location would get the same answer. Other regions confirm them instead.
What happens when a monitoring region goes offline?
If too few live regions remain to reach the plan’s quorum, a stricter rule applies. Every remaining region must report the monitor down on consecutive checks, and the built-in checker must not see it up. If only one region is left, up to two other checkers are asked for a one-off verification check before an incident opens.
Why does recovery need fewer regions than an outage?
So that an incident can resolve while a region is offline. Recovery needs a majority of the live regions, capped at the down quorum. It is never confirmed while the regions still failing would themselves meet the down quorum, which keeps partial outages open.