observability

Once-only notice repeats per replica

#68rare3 sightingsfirst seen September 2026

What goes wrong

A 'notify once' guard lives in one process's memory, or in a dedup key that is written but never read back, while the work runs on several replicas, tabs or CI matrix legs. Each copy passes the guard on its own, so the notice goes out once per copy.

Example

An alert dedup flag is kept in a worker's memory; with 10 workers the same alert is posted to the channel 10 times.

Not to be confused with

The state exists but in too narrow a scope; not a lock that expires while work is still valid (Lease Before Work Deadline) and not an alert re-firing because the condition flaps.

Discovered by

@sks24git GitHub

Field notes · 3

@sks24git2026-10-20beat 1 review round
A test guard keeps its opt-in flag in module state and wraps the global request function with no unwrap. When files share a worker, one file's opt-in leaks into the next, or a stale wrapper refuses a call nobody reports.
Scope a per-file switch to the file, not the module. A shared worker keeps module state alive.
Install a global wrapper once; test it with shared workers. Default isolation hides the stacking.
@sks24git2026-09-10beat 2 review rounds
A run counted as a smoke test whenever any threshold was supplied by hand. A host genuinely above the production alert level during such a run exits green and is filed as a drill.
Derive 'this was only a drill' from the outcome, not from the presence of an input. Otherwise a real alarm inherits the label.
@sks24git2026-09-05beat 1 review round
A once-per-window alert latch lived in the memory of whichever pod won the scheduler lock, and the winner changes every tick. One incident pages once per replica instead of once.
Keep dedup state where every pod can see it. A latch in one process behind a cluster lock is per-pod.
Key windows to write time, not event time. A backlog lands old and stays unseen.