Run aborts before the named check

What goes wrong

A test stops at an earlier step - an unprepared dependency, a missing input, a setup error - and never executes the behaviour its name claims to pin. The verdict still reads as a pass, or as the expected failure, so the reader relies on a judgement about code that never ran.

Example

A test named for a refusal on unreadable settings dies two calls earlier on an unprepared dependency, so deleting the refusal leaves it green.

Not to be confused with

Not Test name promises more than checked, where the named behaviour does run and is only partly asserted; here the named code is never reached at all.

Discovered by

@sks24git GitHub

Field notes · 10

@sks24git2026-11-10

A test case reused the stub left by the case above it, so it failed on that stub's data, not on the rule it named. Break the rule and the test stays green, because the stub still makes it fail.

Reset every stub a case relies on. A leftover stub makes the case fail for someone else's reason.
Make each negative case fail on one assert only. Then you know which rule it checks.

@sks24git2026-11-09

A service began refusing a request type outside its active work window, and the post-deploy self-check sent exactly that request while idle. Every deploy now fails its own check, though nothing is broken.

Run your health checks in the state they run in. A new state rule can quietly break the check that guards every deploy.

@sks24git2026-11-04

A test plants a forged cookie and expects it rejected. After a signing change, the forged value fails its signature check whatever name is read, so the test passes even if the name check is gone.

Re-run a guard test against the bug it guards after any nearby change. A new check can make it pass for the wrong reason.
Fail one check at a time. Two failures hide each other.

@sks24git2026-10-13beat 1 review round

A test fixture pastes the checkout path into a quoted literal, so an apostrophe in the path breaks it. The pipeline still exits zero, and checks that only assert absence pass without running the hook.

Make a test pipeline fail when any stage fails. Otherwise the last stage hides the crash.
Pair every absence check with a positive control. Silence also means nothing ran.

@sks24git2026-09-22beat 1 review round

A sanitizer grown from its failing fixture, not the provider's rule: each round fixed one symptom and exposed the next. Written out, the rule had an uncoded clause: same-role turns merge, so the checked tail is a run.

Write the provider's rule before the fix. Branches then derive from the rule, not the fixture.
Prove a branch-named test reaches its branch. An earlier stage may eat the input first.

@sks24git2026-09-10

The assertions that a job is wired into CI run inside that same job. Narrow its trigger and they never execute, the umbrella accepts a skip, and nothing reports that the guard is gone.

A check on a job's own wiring must live in a job that always runs. Self-guarding is a circle, not a control.

@sks24git2026-09-10beat 3 review rounds

A check that verifies it has not been disabled is executed by the very step it polices, so telling that step to tolerate errors greens the job and the verdict with it. Self-audit cannot outlive its own host.

Anchor an anti-disarm check outside the thing it guards. A witness that dies with the accused proves only that both are gone.

@sks24git2026-09-09beat 4 review rounds

The end-to-end test built a broken fixture and asserted the run refused it, but an unrelated rule rejected that fixture first. Downgrade the refusal under test to a log and the whole thing stays green.

Make the fixture fail for exactly one reason. Another rule refusing it first means you graded nothing.
Assert the count of complaints, not only their text.

@sks24git2026-09-09

A self-test step landed in a CI job that installs nothing, while the script imported a browser library at module top. Static imports resolve before the first line, so the required check died before reading its own flag.

Check what the job installs before adding a step to it. A required check that cannot start blocks every merge.
Import heavy dependencies where you use them.

@sks24git2026-09-05beat 1 review round

A test pinned "unreadable settings refuse the request" with a bare a method on a strict mock. An unstubbed read two calls earlier threw first, so the read the pin named never ran and changing it stays green.

Assert the reason, never only the type. An earlier failure of the same class keeps the test green while the guarded check never runs.
Trace the path up to the assert. Strict mocks fail first.