Test Depends On Machine Environment

What goes wrong

A test silently depends on the machine it runs on — time zone, locale, an inherited environment variable, the installed toolchain — without pinning it. On a differently configured but perfectly valid developer or CI machine the test means something else: it goes red on correct code or green without checking anything.

Example

A date-formatting test passes on a laptop in UTC+3 and fails on the CI runner in UTC because the expected string was written in local time.

Not to be confused with

Limited to isolating the test environment; parsing the environment in production, or general date arithmetic bugs, are not this.

Discovered by

@sks24git GitHub

Field notes · 16

@sks24git2026-11-04

Test setup switches mode on the generic CI flag, which many tools set. A developer whose shell exports it gets the CI path locally, and a setup step with no timeout can stall the job to its hard limit.

Key behaviour on a variable you own, not a generic CI flag. Give every setup step its own timeout.

@sks24git2026-10-30beat 2 review rounds

Each fix to make one test runner act like another showed a new gap: imported classes, empty files, test ids. Three rounds later the bridge was still leaking, so we cut it and kept the part that worked.

Run a file with the runner it was written for. Faking one runner's rules inside another adds a new gap per rule.

@sks24git2026-10-29

A test guarded a query plan by matching the exact wording newer database versions print. The build runs an older version that words it differently, so the check could never fail and a revert would pass.

Run a plan-text check on the version CI runs. A pattern written on your laptop can be blind in the gate.
Watch every guard fail once under CI. A guard never seen red guards nothing.

@sks24git2026-10-26

A test pins 'now' to a calendar date but the rows it writes get their timestamps from the real clock. The day the calendar catches up with the pin, the old rows fall into the window and the suite goes red for good.

One clock per test. If 'now' is pinned, every timestamp the test writes comes from the pin.
Bound every time window on both ends. An open-ended window drifts with the calendar.

@sks24git2026-10-15beat 3 review rounds

A test helper skips rewriting golden files whenever a CI flag is set, and the agent sandbox now sets that flag. The agent follows the test's own hint, nothing is written, and the test stays red.

Grep every reader of an env var before you export it. One flag often gates more than one thing.
Make fix-it hints fail loud when they cannot work. A silent no-op costs a whole run.

@sks24git2026-10-09beat 1 review round

A test pulls in a script that imports a package declared only in the root manifest, and locally it resolves by walking up the tree. CI installs just the subproject, so the required typecheck fails.

Reproduce CI's install, not your checkout. A parent folder's packages hide a missing dependency.
Read a cost after the work ends. A total taken mid-flight counts running jobs as zero.

@sks24git2026-09-15beat 2 review rounds

A layout that clips overflow reports no horizontal scroll at all, so a width check reading scroll size against client size sees zero. Content is cut off on a narrow screen and the check stays green.

Clipping and scrolling are different failures. A metric that sees one is not a width check.
Overflow past a box is not overflow past the screen — say which one a number means.

@sks24git2026-09-13beat 4 review rounds

The judgement was split from the reading so tests could drive it, but one predicate stayed behind in the page. Delete it and nothing changes: no test can run that code, and every real page answers it with an empty list.

Put the decision where a test can reach it. Readings collect numbers; predicates belong in the pure half.
A silent branch is not covered by its neighbours. Check each one alone.

@sks24git2026-09-13beat 2 review rounds

A check started failing whenever a third-party font host was unreachable, inside the gate that guards deploys. One outage turns every page red, and an offline developer gets the same wall instead of a result.

Keep third parties out of required checks. Their outage becomes your release freeze.
Bound every wait you add. An unbounded one turns a stall into a timeout with no culprit named.

@sks24git2026-09-13beat 1 review round

A guard covered the one field its author thought of, while the neighbouring fields fail comparisons just as quietly. Drop one and every box check compares against a value, so the pin reports nothing.

Validate the whole shape, not the field you remember. The forgotten one is silent by the same mechanism.
A pin blind in one environment is no pin. Ask which regime a green run measured.

@sks24git2026-09-11beat 1 review round

A 14-day sum was computed inline beside its 30-day twin, and the test only checked that a 20-day-old row was absent. Any window from 14 to 19 days passed, and the ≥ marker and the unknown branch ran under no test at all.

Pin a window on both edges. A row on the last covered day and one on the first uncovered day kill every off-by-N.
Extract a rule used twice and test it pure. Inline copies drift and hide.

@sks24git2026-09-10beat 1 review round

A refusal was added to a shared rule, but the list feeding it still offered the input it refuses. Every tick re-offered it, retries ate the batch budget, and the armed feature did nothing with healthy counters.

Add a refusal, trim every producer that can still supply the refused input. Else the guard is right and the system is stuck.
A pass that never runs looks like one with nothing to do.

@sks24git2026-09-10

A test of a day-boundary fix tells right from wrong for only three hours out of twenty-four. Reinstate the old timezone-derived bound and it passes at every other hour, with no sibling test pinning it.

Pin the clock, never inherit it. An assertion falsifiable in a narrow window is green by schedule, not by correctness.

@sks24git2026-09-10beat 1 review round

A test case relied on a binary being absent from the runner image and asserted nothing about it. It passes today and turns a required check red the day that image starts shipping the binary.

Pin the environment a test depends on instead of inheriting it. An unstated assumption fails on somebody else's upgrade schedule.

@sks24git2026-09-10

An oracle test anchored its window to the wall clock, so whether a midnight seam fell inside the window depended on the hour the test ran. It was green on every run except the ones that mattered.

Anchor test windows to a fixed local time, not a function. A test green 22 hours a day is a coin with a schedule.
Seed rows one second either side of every seam.

@sks24git2026-09-06beat 3 review rounds

A chart drew numbers inside tall bars at a fixed offset, and its collision test only covered numbers above the bar. When one series trailed the other, the marker painted last sat on the digits and the check stayed green.

One collision test for every placement. The branch the test skips ships covered.
Let the pin measure what the guard measures. Two windows for one rule turn a font swap into a red build.