Documentation contents

Monitoring failures

Updated

When the measurement itself fails: what it looks like, and how it is kept out of a provider's score.

Sometimes the thing that failed is the measurement rather than the provider. A system that cannot tell the difference publishes outages that never happened, so most of the design here is about telling the difference.

A single worker's failure

A worker whose own network is broken sees everything as unreachable. Its measurements are sincere and completely wrong about the provider.

This is the case consensus exists for. One worker disagreeing with the rest is a fact about that worker, and it is treated as one: see How consensus is calculated. A provider is not marked down because one machine somewhere lost its uplink.

A correlated failure

Harder. If several workers share an upstream and that upstream has a problem, they fail together and agree with each other. Consensus weighted purely by count would read that as a genuine outage.

The defence is independence: measurements from different networks and different regions count for more than measurements from many workers behind one path. A provider's status page names how many independent networks and workers a figure came from, which is what lets a reader judge how much weight to put on it.

ICMP being dropped

Some hosts deliberately never answer pings. Treating silence as an outage would produce constant false alarms against providers who have simply configured their network that way.

This is why a target that never responds looks different from a target that stopped responding, and why a provider whose whole space is silent from the start reads as "not enough data" rather than "total outage".

Platform-side failures

If the platform stops receiving reports, providers move to "no recent data" rather than to an outage. This is the important direction: an ingestion problem on our side must never publish an outage against somebody else's network.

What a reader sees

None of the above is presented as a provider outage. In every case the page either withholds the verdict or states that there is not enough data, which is why "not enough data" appears more often than people expect. It is the honest answer surprisingly often.

If you think a verdict is wrong

Provider managers can see the same breakdown readers can: which regions, which networks, how many workers. A verdict that disagrees with what you are seeing internally is usually a regional or network-specific problem that is genuinely visible from outside and not from inside, which is the case independent monitoring exists to catch. If it is not, the status page's per-network view is the place to start, and support can trace an individual verdict back to the measurements behind it.

Was this page helpful?

Your answer tells us which pages need rewriting.