tests/ci

Test Checks Type Not Value

#89rare22 sightingsfirst seen September 2026

What goes wrong

A test asserts only the type, format or set membership of a result computed from specific inputs — 'is a string', 'is 64 hex chars', 'is a valid date', 'throws some exception' — so a value computed from the wrong input, in the wrong unit, or a constant that always has that shape passes.

Example

The test only asserts that formatting the order's creation date returns a string; the function formats the wrong field and the test passes.

Not to be confused with

The value is produced and reached but never tied to its input or to what consumers depend on; not an expected value copied from the implementation (Self Derived Expected Value) and not a code path never reached (Unreached Branch Test).

Discovered by

@sks24git GitHub

Field notes · 3

@sks24git2026-11-04beat 2 review rounds
A keyed hash's test only checked that the keyed output differs from the bare one and repeats for the same key. Swap the real secret for a constant in the code and every test stays green while the outputs turn guessable.
Pin a keyed hash to its key. Assert the exact output under one secret and a different one under another, or a constant key passes.
@sks24git2026-09-12
A new typed error was asserted only by its class, never by the status it maps to. Change its parent and every test stays green while callers keyed on the old code quietly stop retrying.
Assert the wire contract, not the class name. The status is what callers branch on.
Route one test through the real handler. A constructor call proves nothing about the response.
@sks24git2026-09-10beat 1 review round
Nothing pinned the converted amount or the rate on the new journal legs, so mutating the rate to zero left the suite green. The pointer in the intent doc had also drifted into a neighbouring test.
Assert the money fields, not the row count. A leg with a zero rate is a valid row and a wrong one.
A cited line number is a claim; re-run it before trusting it.