Outer timeout too short for retries

What goes wrong

An outer timeout is sized for one attempt or one phase, but the code inside performs several — retries, sequential waits, recovery steps — all drawing on the same budget. Legitimate work is cut off before it can finish or even run its own error handling.

Example

A serverless function has a 10-second limit; inside it a client retries three times with a 5-second timeout each, so the second retry is killed mid-flight and the failure handler never runs.

Not to be confused with

Needs a missing term in the time budget (the parts add up to more than the whole); not an ownership lease (a timed claim on a job) running out during valid work, not a deadline that was never passed down, not a poll interval that misses a short state (Poll interval longer than the event), and not work that keeps running after its timeout.

Discovered by

@sks24git GitHub

Field notes · 33

@sks24git2026-11-05

A step moved after a long transfer kept its old deadline, counted from before the transfer. A large transfer then leaves it zero seconds, and every retry pays the transfer again and fails the same way.

Count a moved step's deadline from where it now starts. The work you put in front of it is not its time.
Never let a retry hit the same deadline. It pays again and fails again.

@sks24git2026-11-04

A time budget equals the sum of its steps' worst cases, with nothing left for queueing. One slow runner pickup and the whole job times out though every step stayed in bounds.

Leave slack for waits you do not measure. A budget that adds up exactly fails on the first queue.

@sks24git2026-11-04

A retry budget for a schema change assumed each attempt waits on one lock, but one attempt can chain several short waits. The real worst case overruns the platform's start deadline, and the release is killed mid-change.

Bound a retry loop by measured attempt time, not by one wait. Attempts that chain waits blow the sum.
Pin worst cases with a test that runs them. Arithmetic on constants proves nothing.

@sks24git2026-11-04

A side measurement ran before the main work but after the start clock, so every later deadline counted its seconds too. On a tight plan the work it was meant to observe fails before it begins.

Give a measurement its own budget. If it shares the clock of the work it watches, it can break that work.
Run optional probes where no deadline waits. A slack phase costs nothing to borrow.

@sks24git2026-11-04beat 5 review rounds

A new network call was added to a step whose time budget lists every call it makes, and the list was not updated. When the call is slow, the step outlives the deadline it promised, and a watcher reads live work as dead.

Add the term when you add the call. A budget that sums its steps is only right while the list is.
Pin the budget with an oracle, not the formula. A copied formula moves with the bug.

@sks24git2026-11-02beat 1 review round

A follow-up job runs in a worker thread after the main work, with a fixed time cutoff counted from when the item was taken. The main work alone often eats the cutoff, so the follow-up almost never runs.

Budget follow-up work from what is left. Cap it to the remaining deadline instead of a fixed slice, or it starves.

@sks24git2026-10-30

A fit check let a batch through when its timed waves filled the window exactly, leaving zero time to prepare any part. The run passed the check, started nothing and failed as an outage instead of a clear refusal.

Count setup time inside every fit check. A check that budgets only the work admits jobs with no room to start.
Test a budget at window-1, window, window+1. Off-by-one hides at equality.

@sks24git2026-10-26beat 2 review rounds

A retry loop gave each request a timeout equal to the whole remaining budget. So the first hung request ate the full budget, and the retry it was built for could never run.

Cut each attempt well short of the budget. Otherwise one hang leaves no time to retry.
Retry the error class, not one message. A body cut mid-read throws differently.

@sks24git2026-10-25beat 1 review round

A list check ran one slow remote call per item, in a row, under one short deadline, and a timed-out check read as "no access". Big lists lost the very items the user could open, and the page called itself complete.

Keep "said no" apart from "did not answer". Folding a timeout into a refusal hides data and lies about it.
Price the loop against its deadline. Serial calls times items must fit the budget.

@sks24git2026-10-16beat 1 review round

A size rule let a work item grow to the exact time limit of the job that runs it, while setup and checks spend the same clock. An item sized at the top is killed before it ends, and every retry dies the same way.

Set size limits below hard kills, not at them. Setup and checks eat the same clock.
Check each new number against every limit in the doc. A rule that fits one bound can break another.

@sks24git2026-10-13beat 1 review round

A grouped job waits a fixed window after its first run ends, but its other runs queue behind all normal work. They miss the window, so the result ships without them and their part of the input goes unread.

Size a wait window from queue time plus a full run. A shorter window drops work already paid for.
Never latch a result while its parts still sit in a queue. Late parts have nowhere to land.

@sks24git2026-10-12beat 1 review round

Two steps in a fallback chain shared one time limit, and the first step had no limit of its own. When the first step stalled, time ran out before the backup could start, and every request failed.

Give each step in a fallback chain its own time limit. One shared clock lets a slow first step starve the backup.

@sks24git2026-09-13beat 1 review round

A five-second cap was added so one stalled fetch could not hang the run, and the comment argued it in terms of the whole job. The cap is paid per read, and the run makes 184 of them, so the stall still ends in a timeout.

Price a timeout against the whole run. Multiply it by the call count before you believe it.
A bound argued at job level must be job level. Otherwise it only looks like a fix.

@sks24git2026-09-10beat 1 review round

The per-pod ceiling behind a deploy budget is measured from container start, so image pull and scheduling fall outside it. Every deploy pulls a fresh tag, and the margin over that ceiling was three minutes.

A probe budget starts at container start, not at scheduling. Add pull time before calling it the per-unit cost.
Three derivations that each miss a term mean the number must be measured.

@sks24git2026-09-10beat 2 review rounds

A rollout budget was derived from per-pod limits, while the wait covers the whole release and replicas are replaced one at a time. With three replicas the documented worst case understates reality two- to threefold.

A total budget is not a per-unit limit. Multiply by the units replaced in sequence before calling it a floor.
A deadline that restarts on progress bounds one unit, not the run.

@sks24git2026-09-10

A deploy step waits for readiness and, on timeout, rolls back with the same wait, costing up to twice its budget. The job cap was left at exactly that double, and a cancellation is not a failure, so no alert fires.

A step with an atomic rollback costs 2x its wait budget. Size the job cap from the double.
A timeout-cancelled job skips failure-only diagnostics. Its silence then reads as success.

@sks24git2026-09-10beat 1 review round

A retry budget counted four attempts and ignored the sleeps between them. The tool's default backoff adds over two minutes, so the documented worst case is short by half and the timeout was sized from the wrong number.

Add the waits, not just the attempts. Retry arithmetic that ignores backoff always understates the ceiling.

@sks24git2026-09-10beat 2 review rounds

A coverage check summed its matches across all jobs and passed when the total was non-zero, which held while one job carried the whole corpus. Split it into three and two of them lost the check silently.

When you shard a subject, re-derive every check that aggregated over it. A union test survives the split and stops testing.
Plant a fixture in each shard, not in the first.

@sks24git2026-09-10beat 1 review round

A per-step timeout was justified by a floor counting only the work, while the clock starts when the step launches and includes setup and one runtime start per invocation. A real hang outruns the cap it was sized for.

Size a timeout from the clock that enforces it, not from the work you measured.
State which zero a budget counts from.

@sks24git2026-09-10

Which module runs in which leg was written twice: once in the job matrix, once in a case statement. Drop a matrix row and every surviving leg decides the module belongs elsewhere, so it runs nowhere and all legs pass.

Two lists that must agree need a check reading both. Neither half can notice the other is gone.
Assert what ran, not that nothing failed.