Check passes over an empty set

What goes wrong

A check loops over the items it is meant to judge, and the set it collects is empty, so no comparison ever runs and the check reports success. The reader takes the green result as evidence that the rule held, when it only means the rule was never applied to anything.

Example

A rule walks every entry in a folder and complains about bad ones; a cleanup step moved the entries elsewhere, so the rule now walks nothing and reports that everything is fine.

Not to be confused with

Not Fixture never creates the triggering condition, where a case exists but misses the branch, and not Test Data Too Small To Tell, where data is present but too thin; here there is nothing to judge at all.

Discovered by

@sks24git GitHub

Field notes · 8

@sks24git2026-11-04

A flaky test got a wait-until-done loop that just stops trying after two seconds. On a slow box the old race comes back, and the test fails as a wrong answer instead of a timeout.

Assert the wait reached its state. A poll loop that falls through hides the race it was written to fix.
Clean up in finally. One leaked row turns one red test into two.

@sks24git2026-11-04beat 3 review rounds

A test skips its checks when its fixtures are absent, to stay green off the build server. The local gate treats any skip as a silenced test, so every local run now fails.

Read the gate's skip rule before adding a skip. A skip that is fine to you can be a failure to it.
Keep local behaviour unchanged when fixing CI. Each local tweak opened a new edge.

@sks24git2026-11-03

A test helper ran a probe that hides its own errors, and every negative case then expected an empty answer. A broken probe returns empty too, so all those cases pass on code that reads nothing.

Make a quiet helper fail loud on impossible output. Empty from a broken tool looks like a pass.
Keep the unit's own guard when you add an end-to-end path. The outer layer hides the inner one.

@sks24git2026-10-26

A CI guard step piped its checker through a formatter and a tool without a flag, so the step's exit status was a tool's zero. A time bomb found in a test landed in the job summary while the job stayed green.

Pipe a gate's output only under a flag. A pipeline returns the last command's status, not the checker's.
Test the red path of every new CI step. A guard that cannot fail is decoration.

@sks24git2026-09-13

Two new readings entered a checker as parameters defaulting to the passing values. Drop either from the code that collects them and the gate goes silent, green over the very defects it was added to catch.

Never default a measurement to "fine". A missing reading must throw, not read as a pass.
Check the whole shape in one place. The key you forget is the one that goes silent.

@sks24git2026-09-12beat 1 review round

Tests pinned a disabled feature with "this call never happens" assertions, on a path that could never make the call anyway. They pass for free, so the whole gate could be deleted and the suite would stay green.

An assertion that nothing happened must run the arm that could make it happen. Otherwise it pins the path you chose, not the rule you meant.

@sks24git2026-09-10

The self-test never drove the enforcement path, and at this commit the list it computes is legitimately empty. Replace that computation with an empty list and everything stays green while enforcement never fires.

A rule whose live input is empty needs a synthetic non-empty case. Real data that never reaches the branch tests nothing.
Exercise the exit code, not only the report.

@sks24git2026-09-08

A cleanup moved tombstones out of a directory, and the one row that exercised the filter for them now saw an empty folder. Deleting the filter stays green and reopens the nag the change had fixed.

When you move data, move the test that reads it. A row that passes on an empty input pins nothing.
Reread comments next to code you changed.