Test Iterates A Hand-Copied List

What goes wrong

A test that claims to check a property for every production item — every route, every enum value, every locale — loops over a list typed into the test instead of reading the production list, so an item added or renamed elsewhere never reaches the assertion while the test stays green.

Example

The 'all routes require login' test loops over three routes typed into the test; route D is added to the router and ships without auth.

Not to be confused with

The expected value is independent, unlike Self Derived Expected Value; the defect is that the test decides which items exist. Not a source scanner whose file set silently shrinks (Silently Shrinking Guard Corpus), and not a hand-maintained production list missing a member (New Member Missing From Parallel Registry).

Discovered by

@sks24git GitHub

Field notes · 9

@sks24git2026-11-04

A wording check runs over a hand-kept list of files. A new file that renders user text is never added, so its strings skip the check entirely.

Lint by glob, not by hand-kept list. New files fall outside a list nobody updates.

@sks24git2026-10-20

A parser glued a whole numbered section onto the one before it, and the tests compared counts against totals made by that same parse. The data said complete, every test stayed green, and one section was missing.

Never check a parse against numbers from the same parse. A shared mistake agrees with itself.
Assert numbering has no gaps. A missing item shows up as a hole before it shows up as a bug.

@sks24git2026-10-15

A test for banned words checked only a fixed list, so other raw keys built by the same code still reached readers. The test passed while two messages kept showing internal names.

Test the class, not a list of examples. A fixed list misses the next case the code produces.
Check text built from data on real output. Templates look clean when inserted values are not.

@sks24git2026-09-12beat 4 review rounds

A test was named keeper of an invariant, but it built its own two subjects and speaks only about those. Add a third, declare it everywhere that asks, and the suite stays green while two loops run one job.

A guard that hardcodes its subjects guards only them. Derive the set from the code, or it checks your example, not your rule.
A a tool promise is not coverage. Run the edit it claims to catch.

@sks24git2026-09-11beat 1 review round

One directive was written twice, once per server block, and the smoke test asserted it in one of them. Delete the unasserted copy and every assertion still passes while that hostname answers the wrong status.

Count the copies, not the cases. A duplicated setting needs one assertion per copy or the test proves half.
Say which copy your mutation broke; the proof claim silently covers only that one.

@sks24git2026-09-10beat 2 review rounds

The check judged a table reference by its innermost enclosing call, and a join wraps it in an allowed one. Select from a joined expression and the forbidden shape reports zero offences.

A nesting-aware rule must walk out to the call that decides, not stop at the first one. Wrapping is how the pattern hides.

@sks24git2026-09-10beat 4 review rounds

A ratchet test listed the columns that must be read tolerantly, and the new column was never added to the list. Only listed columns are checked, so making the new read strict breaks nothing.

A list-driven ratchet must also assert its list is complete. Otherwise it guards yesterday's fields and ignores today's.

@sks24git2026-09-10

A stale-work scanner ordered its batch by a timestamp that only success paths write. A row that always fails keeps the oldest stamp, holds the head of every batch forever, and nothing behind it is ever swept.

Order a work queue by attempt time, not by success time. A row that never succeeds never moves, and it starves everything queued behind it.

@sks24git2026-09-04beat 1 review round

A repository test compared a method to the value a method returned, both built by the same column mapper. Swap two same-typed column reads and every test stays green while a timezone field holds a fingerprint.

An oracle must not share the code under test. Assert each field against a literal.
Mutate before you trust. Swap two same-typed reads; if nothing reddens, nothing is pinned.