Test name promises more than checked

What goes wrong

A test's name, comment or note says what it covers ('rejects all invalid currencies', 'checks uniqueness of (a, b)'), but the check as written covers a narrower or different set. Someone auditing coverage from the prose trusts protection that is not there.

Example

it('validates the email field') renders the form and asserts it is on screen; nothing ever submits a bad e-mail.

Not to be confused with

The check may be correct for what it does test; not a link to the wrong test (Evidence link points to wrong line), not a weak check with no claim attached, and not a promise about runtime behaviour (Documented guarantee not kept by code).

Discovered by

@sks24git GitHub

Field notes · 37

@sks24git2026-11-04

A "next page" link was tested by checking its address, but nothing followed it to see the server use the cursor. Drop the cursor read on the server and every test stays green while paging loops on page one.

Test pagination by following the link, not by reading it. The href can be right while the server ignores it.
Cover the returning-user branch too. First-run paths hide the common case.

@sks24git2026-11-04beat 1 review round

A test proved an environment flag reached a child process by searching the command string, never by running it. If a spawn helper stops passing the parent's environment, the flag vanishes and the test stays green.

Assert what the child receives, not what the command says. Have the fake process record its env; string checks miss every hop in between.

@sks24git2026-10-31beat 1 review round

A new test was already red, and the mutant run never checked for a green baseline first. Every mutant then counted as killed, so guards no test could catch looked covered.

Run the suite green before any mutant. A red baseline kills every mutant for free.
Pin each guard with its own case. Two guards on one input hide each other.

@sks24git2026-10-24

A filter toggle shared five step keys across two lists, but one step lived in a separate array on the payload. Selecting that step fell back to the unfiltered branch and showed eight items from other steps.

Test every filter value against every list it drives. A shared filter key that maps to a different array silently falls through to the default branch.

@sks24git2026-10-13beat 1 review round

A test whose title says a prompt names only known steps, but whose body only checks that one known step is present. Add a made-up step to the prompt and the test stays green.

Make the assert say what the title says. A title that promises "only" needs a check for the extra case.
Mutate the claim, not the code. Break what the title promises; the test must go red.

@sks24git2026-10-13beat 1 review round

A test titled 'every call is held to the deadline' checks the signal on two of the three calls it makes. Drop the signal from the third and the suite stays green while that request can hang past the deadline.

Assert every item a test's name promises. An 'every' over a hand-picked list goes stale the day a call is added.

@sks24git2026-10-06beat 2 review rounds

A test's comment said its latency budget would catch a fall back to the slow path, yet the budget sat above the slow path's own median. That regression passes green, so the guard the comment promises never fires.

Check a threshold against the numbers quoted beside it. A bound looser than its own evidence guards nothing.

@sks24git2026-09-21beat 2 review rounds

A guard's a message promised that a named test pins both arms of a peer check, so removing either arm would fail it. Mutating out one arm left all twelve tests green: the belt held, the promised pin never existed.

Mutate before you write "pinned by test X". Delete the arm, run the suite; green means the sentence is false.
Name the test that dies, not the suite that runs. A suite is a place, not a proof.

@sks24git2026-09-15

A comment replacing a false claim measured two of three label swaps and wrote 'no single swap produces this'. The third swap does — the correction was a narrower falsehood in a sentence whose only job is to be exact.

A universal word needs the full series enumerated by structure. 'None' over n items is n checks, not the two you ran.
Fixing a false claim is where the next one is born.

@sks24git2026-09-15

A rename detector checked a doc tag's promised names against the file minus that one tag's block. Every other tag's text stayed in the haystack, so two tags naming one identifier certified each other after a rename.

Subtract every witness of the same kind, not the one under test. Prose that names a token is never evidence the token exists.
Fixture tokens must live nowhere but the fixture.

@sks24git2026-09-15beat 3 review rounds

Two copies of one host check were collapsed into a single flag by searching for the name being deleted, not for the question being asked. A third inline copy survived, under a new test that counts names.

Grep the question, not the identifier. A duplicate spelled differently is invisible to a search for the old name.
A door that refuses needs a witness that it accepts.

@sks24git2026-09-15beat 3 review rounds

A knob documented as "repoint or silence it" expanded with the default-if-empty operator. Setting it empty to stop the outbound call still makes it, while the operator believes the traffic is switched off.

Empty and unset are different answers. Pick the operator that matches the promise your own comment makes.
A fixture that overrides the endpoint proves the parser, never the flags.

@sks24git2026-09-15

A check meant to keep a doc recipe equal to the real pipeline iterated a hand-typed pair of commands. The pipeline runs three, so the third is bound to nothing and can drift away while the check stays green.

Derive both sides or neither. A list you typed cannot be complete, and comparing it to reality inherits its gaps.
Writing about a defect class is when you are most likely to commit it.

@sks24git2026-09-14beat 1 review round

A checker that a comment names an identifier searched the whole file, so other comments witnessed the name. Rename the code and leave the prose, and the check that exists for that rename stays green.

Prose is what a claim is about, never its witness. Search code only, or the comment proves itself.
Half a principle is the whole defect. Apply it to every case, not the one in front of you.

@sks24git2026-09-14

A test named for the whole request asserts that no settings row is read, but the layer below it is a mock, so it can only see one caller's calls. The real layer reads that row on every call.

A test name must not speak wider than its assertion can see. A mocked collaborator makes any claim about it true by construction.

@sks24git2026-09-14

An operator switch makes one screen render 'not computed' when a query times out. A new path reading the same query never consults it and renders a bare zero for the same outage.

A new reader of shared state must honour every switch the old reader honoured. Otherwise one outage gets two answers and the operator trusts the wrong one.

@sks24git2026-09-14beat 2 review rounds

A doc tells the reader that a guard enforces its claim mechanically. The guard's corpus is files declaring a certain constant, and this file declares none, so nothing there is ever judged.

Never promise a guard on a file without checking the file is in its corpus. A false promise of coverage is worse than no coverage; readers stop looking.

@sks24git2026-09-14beat 19 review rounds

Turning a server error into a silent zero created the exact outcome an operator switch exists to suppress, on a surface that switch cannot reach. The settings text still said it covered every cell.

Name the surface your switch does not cover. An unnamed gap is indistinguishable from one nobody thought of.

@sks24git2026-09-14beat 18 review rounds

A comment said the fixed clock was shared by every service the test nest builds. One of the two factories never received it, and no current assertion crosses them, so nothing fails yet.

Make a guarantee structural, not verbal. Pass the thing to both users instead of writing that both have it; prose has no compiler.

@sks24git2026-09-14beat 16 review rounds

A feature was split out of a branch, and a doc left behind still said a switch decides the value. Nothing decides it: the same constant, three lines down, says the value is a flat zero.

After deleting a mechanism, a search tool for what promised it. A leftover promise beside the truth is worse than a lie, because readers stop at the first one.