Skip to content

Engineering · 10 min read

Whose outage is it? How SutramX labels an incident “likely external”

How SutramX pools check results across customers, without exposing any of them, to tell a vendor outage from yours, and why it checks its own checker first.

Published · By Kumar, QuantumPlug Technologies LLP

Curved violet ribbon of light

When a payments API or a cloud region starts timing out, every product that depends on it has an incident at the same moment. Each on-call engineer then spends the first stretch of the night proving their own code isn’t broken. An uptime monitor with many customers sees something no single customer can: the same vendor failing for several unrelated teams at once. SutramX uses that view to add a “likely external” verdict to an incident. Doing it responsibly is harder than it sounds. Pooled data must never leak one tenant’s details to another. A failure in our own checker looks exactly like a vendor outage. And telling someone “not your fault” when it is their fault is worse than saying nothing. This article explains how the detection, the verdict and the safeguards work in the code.

Start with a fixed catalogue, not inference

Detection never guesses which hosts are vendors. A catalogue in the code maps hostname patterns to vendors: payments, cloud, hosting and CDN platforms, developer tools and AI APIs, about 80 providers in total. A pattern is either an exact host, such as a vendor’s API hostname, or a wildcard that matches any subdomain of a domain but not the domain itself. The longest matching pattern wins, so a Firebase hostname under a Google domain is attributed to Firebase rather than to Google Cloud. For AWS, detections are kept per cloud region, because us-east-1 failing says nothing about eu-central-1.

Two rules govern what can be added. Every vendor must also have an official status feed, so any detection can be compared with what the vendor itself publishes. And a pattern must identify the vendor’s own infrastructure. A customer’s own domain is never treated as a vendor, however many workspaces monitor it, and that includes a domain the customer points at a vendor through a CNAME. Hosted-app suffixes are included on purpose because they identify the platform, but only the platform’s name is ever shown, never the customer’s subdomain. A test checks every catalogue entry.

What a single check says about a vendor

A monitor being down is not the same as its vendor being down. classifyCheck() sorts every result into one of three verdicts. Failing means the vendor did not answer properly: a timeout, a 5xx, a refused or reset connection, or a TLS failure. Healthy means the vendor answered, even if the monitor failed. A 401 from an expired API key, a missing keyword or a slow response says the vendor is up and the problem is somewhere in the customer’s setup. Abstain covers results that say nothing about the vendor: bot-protection blocks, rate limits (429), DNS failures and our own checker errors.

Older rows without a recorded error type are held to a stricter standard. Only clear evidence counts, such as a 5xx status code or a timeout or connection error in the message. Anything ambiguous abstains. The bias is deliberate: when in doubt, a result does not count towards declaring someone else’s outage.

  • Failing: timeouts, 5xx errors, connection failures, TLS errors
  • Healthy: 2xx and 3xx, 4xx, keyword or assertion failures, slow responses
  • Abstain: bot protection, rate limits, DNS failures, our checker’s errors

Count workspaces, not checks

A detection job runs every minute. It takes the latest result for each monitor and region over a 10-minute window, for every active monitor whose target matches the catalogue. Results from regions whose checker is currently impaired are removed first. A monitor counts as failing on a vendor only when at least half of its conclusive regions report a vendor-class failure, so one flaky vantage point among several healthy ones doesn’t count. A workspace counts as failing when any of its monitors on that vendor is failing.

The unit that matters is the distinct workspace, not the number of checks. One customer with forty monitors pointed at the same API, all failing because of one shared misconfiguration, is still one workspace. Opening a detection needs at least 3 failing workspaces and, by default, at least 30% of the workspaces observing that vendor. The minimum of three is a constant the environment configuration cannot lower. Configuration can make detection stricter, never looser.

A small state machine with hysteresis

A detection moves through a short lifecycle in decideTransition(state, stats, thresholds, checkerOk). With no episode open, meeting the open threshold creates a suspicion, which customers never see. On a later run, at least 45 seconds afterwards and never on a back-to-back run, the suspicion is confirmed if it still meets the open threshold, and dismissed if it doesn’t. Unconfirmed suspicions are deleted after a day.

Once an episode is active, it uses a lower bar to stay open than it needed to open. That is classic hysteresis, to stop it flickering. It continues while at least 3 workspaces are failing and the failing share stays above the clearing share, which defaults to half the opening share, or 15%. Below that, each run counts a clear tick, and three consecutive clear ticks resolve the episode. Runs are serialised across workers with a transaction-scoped advisory lock, so an overlapping run skips rather than counting a clear tick twice or opening a second episode.

The last state is the important one. If our own checking is suspect at the time, the machine does nothing new: a suspicion is dismissed, nothing is opened or confirmed, and an active episode freezes. It neither continues nor counts towards clearing.

Our checker is the first suspect

A monitoring company’s own network trouble looks exactly like the internet breaking. SutramX tracks the health of each checking region in three ways. Canaries probe a few well-known HTTPS endpoints on different providers, where any HTTP answer counts. A region is marked impaired when at least two, and at least half, of its live checkers have failed two consecutive rounds. A failure-rate breaker marks a region suspect when unrelated monitors flip to down at once: at least 5% of the region’s monitors within 2 minutes, at ten times the past hour’s rate, from at least three workspaces, counted per target host and capped at five per workspace. One customer’s shared host going down is their outage, not ours.

The breaker trips only when the other regions disagree. A real outage of a large CDN or cloud provider flips the same monitors in every region at once, which points at the targets, not at us. If more than half the regions would be suspect at the same time, none is marked suspect and the event is logged as a target outage. While a region is impaired or suspect, its down results are recorded as unknown. They abstain, the other regions decide, and uptime figures are untouched. A suspect state expires after 600 seconds by default and cannot trip again for as long, so it cannot hide a real outage beyond that bound.

Fleet-wide, a global alert hold applies when incidents open at an abnormal rate: at least 3% of active monitors in 5 minutes, with each workspace contributing at most five. New down alerts then wait, for at most 300 seconds by default, and an incident that resolves in the meantime never pages anyone. The vendor detector treats this hold as checkerOk = false.

From a detection to a verdict on one incident

An active or recent episode becomes a finding on a specific incident through vendorContributorsFor(). The incident’s vendors come from three sources: the monitor’s own URL, the URLs of monitors it depends on, and vendors the workspace follows on its status board. An episode counts if it overlaps the incident’s window, including episodes that started up to 15 minutes before the incident.

A finding is only marked as the likely cause when three things hold. The vendor is one the monitor actually calls, directly or through a dependency; following a vendor is not enough. The incident’s own failure is the kind a vendor outage produces, which a failed keyword is not. And the failing regions overlap. Without a SutramX detection, an incident reported on the vendor’s official status page that overlaps the window can also be a finding, held to the same relation and failure-type tests. At most three findings are kept, ranked by how direct the link is.

The incident’s fault verdict, from faultOf(), then follows a fixed order. A likely-cause vendor finding, or the separate last-mile rule that spots an Indian ISP failing while data-centre checks pass, makes the verdict external. A checker finding, or a picture made up only of inconclusive results, makes it checker. A confirmed outage with a failure class that points at the target, with no other cause found, makes it yours. Anything else is unknown, and the explanation says which.

The reassuring wording is gated in code. Only one module may write “likely not your fault”, and it returns nothing unless the verdict is external. A build test fails if any other source file contains the phrase. None of this changes whether or when an incident opens or an alert is sent. It only adds context to the explanation and the alert text.

Privacy as a rule in the code

Pooling results across tenants is only acceptable if what comes out is genuinely aggregate. The policy module states the rule at the top. Customer-visible output is limited to the vendor’s name, status, times, probe regions, failure classes and a coarse size, never a workspace, monitor, URL, path or exact count. Size is “several” or “many”, and “many” requires at least 10 failing workspaces. A region or failure class is only named when at least the minimum number of distinct failing workspaces share it, so a detail cannot single out one customer. Each workspace’s timeline note goes only on that workspace’s own incidents.

The same caution applies to good news. The public vendor status page will only call a vendor operational when at least 5 distinct workspaces observed it and our checker was healthy. Otherwise it says there isn’t enough data. A vendor that couldn’t be declared degraded is never declared up either.

Comparing with the vendor’s own status page, conservatively

Each episode is compared with the vendor’s official status page, which we keep checking for a few hours after an episode ends, because providers often post late. firstOfficialReport() takes the earliest official signal between 60 minutes before the episode started and its end. That can be the time an incident first appeared on the page or the provider’s own stated start time. Taking the earliest signal understates our lead time rather than overstating it, so a rarely polled page or a backdated incident can never make SutramX look faster than it was. The result is one plain sentence: the vendor confirmed it a number of minutes later, its page still says all systems operational, or it never reported the incident.

Detection of this kind is easy to make impressive and hard to make trustworthy. In SutramX it rests on four decisions: a fixed catalogue instead of inference, counting distinct workspaces instead of checks, hysteresis with our own checker treated as the first suspect, and privacy limits written as code that tests enforce. Any client system that pools data across tenants or tells users why something happened needs the same care. The facts have to be in code, the conclusions conservative, and the wording tied to a verdict the system can justify.

Common questions

How does SutramX know an outage is a vendor’s and not mine?

Its own check results against known vendor hostnames show the same vendor failing for at least 3 different customer accounts, and a meaningful share of all the accounts checking it. If your monitor calls that vendor and fails in a way a vendor outage would cause, your incident is marked likely external.

Does vendor detection reveal other customers’ data?

No. Only aggregate signals leave the detection module: vendor name, status, times, regions, failure classes and the words “several” or “many”. It never shows accounts, monitors, URLs or exact counts, and a customer’s own domain is never pooled.

Can vendor detection stop an alert from being sent?

No. It adds context to the incident explanation and the alert text. It never changes whether or when an incident opens or an alert is sent.

What if SutramX’s own checker is the problem?

Region health is checked with canaries and a failure-rate breaker. While a region is impaired, its failures are recorded as inconclusive and the other regions decide. While incidents are opening fleet-wide at an abnormal rate, vendor detection opens and confirms nothing.

All insights