Threshold Set Without Measuring Real Traffic

What goes wrong

A cap, alert threshold or sanity limit is set from a guess, a typical-day estimate, one environment (a simulator, one test bench) or the wrong time unit rather than from measured live behaviour, so it sits outside the range of normal use: it fires on legitimate peaks, or never fires on the typo it exists to catch.

Example

An alert on 'more than 100 bookings per hour is suspicious' pages the on-call every Friday evening, when 150 is normal.

Not to be confused with

The limit itself is wrong; not a threshold that can never fire by construction (Alert Threshold Above Failure Band), not a time budget that omits inner work (Outer deadline omits inner work), and not a test that fails to probe a limit (Unchallenged Boundary Assertion).

Discovered by

@sks24git GitHub

Field notes · 9

@sks24git2026-11-02

A health probe's timeout was cut to fit a time budget, but the probe is the first call to a sleeping machine. A slow wake now reads as a dead machine, so the main path quietly falls back every time.

Size a probe timeout for the slowest healthy answer, cold start included. Too tight turns a slow yes into a no.

@sks24git2026-10-26beat 1 review round

A liveness threshold was sized from an idle timeout, but the lock holder kept its session busy with keep-alive pings, so the hold had no bound. A quiet node still read as dead and its crashed work never counted as a try.

Bound what you measure, not its neighbour. An idle timeout says nothing about how long a lock is held.
Pulse from outside the critical section. A heartbeat behind a lock dies with the queue.

@sks24git2026-10-25

A new watchdog flagged any change request as stuck after an hour without a commit, but review and automated fix steps in the same pipeline take longer than that by design. Every healthy request would raise a paid alert.

Calibrate a stall threshold against the pipeline's own timeouts. Shorter than a normal step means false alarms.
Exclude known-busy states before timing silence. Quiet is not stuck.

@sks24git2026-10-25beat 2 review rounds

A budget set each run's hard cap equal to the mean price estimate so count times cap matched the monthly total. Every above-average run gets cut, and the benchmark meant to test the estimate can never see past it.

Keep estimate and cap apart. A cap at the mean censors the data that would fix the mean.
Hold the month ceiling at the gate. Check spend before each run, not count × cap.

@sks24git2026-10-13

A connection pool was shrunk to fit a server limit, though metrics showed it was already full. Requests waited longer, and the limit could still be exceeded.

Read the saturation metric before shrinking a pool.
Raise the shared limit instead of starving the client.

@sks24git2026-10-13

A storage cap fit the data of launch day, while retained history kept growing into it. A fixed timeout waited on work that scales with the size of the system.

Size caps for retention times growth.
Scale timeouts with the work they wait on. A fixed limit turns slow into failed.

@sks24git2026-09-10beat 1 review round

A per-request timeout was calibrated from whole-step totals, with no per-request data at all. Downloads run concurrently, so the cap sits below the step's median and could abort an ordinary healthy run.

Do not derive a per-item bound from an aggregate. Measure the unit you intend to bound, or leave it unbounded.

@sks24git2026-09-10beat 1 review round

A timing budget was derived from the runs that finished. The runs killed at the ceiling were dropped, so the quoted maximum is the slowest survivor and the real tail sits above every number in the sample.

Killed runs are data, censored at the limit. A spread over survivors describes the survivors, not the job.

@sks24git2026-09-10beat 1 review round

A budget warning fired at seventy percent of a cap that already carried a twenty-five percent safety margin. The heaviest load the cap was designed for lands above that line, so the normal case warns every run.

Set an alarm against the design point, not the raw ceiling. A margin counted twice cries on healthy runs.
Work one example by hand before shipping a threshold.