Test Data Too Small To Tell

What goes wrong

All the test data is degenerate for the property under test — one element, a term that equals zero, an empty derived list — so a broken implementation gives the same answer as a correct one. Replace the sum with 'take the first element' or drop the sort and every test still passes.

Example

A sort test uses a one-element list; remove the sort and the test is still green.

Not to be confused with

The code runs and the branch is reached, unlike Unreached Branch Test; the data is too small, rather than matching by coincidence (Coincident Alternative Oracle).

Discovered by

@sks24git GitHub

Field notes · 27

@sks24git2026-11-06beat 1 review round

An endpoint priced a job from a mean it computed itself, but its test fixture had no priced rows, so the mean was always empty. Wire in a wrong or empty mean and every test stays green while the price shown is wrong.

Test the wiring with a value, not only the empty case. A null fixture passes for any input.
Assert the shown number against an oracle. Not just that the field exists.

@sks24git2026-11-04

A shared table renderer was tested on empty tables and headers, but no test read a row of a filled table. Drop the rows, or escape cells that callers mark as already safe, and every test still passes.

Assert on the body of a non-empty output, not just its frame. Empty fixtures pass almost any bug.
Pin the escape contract both ways. Double escaping breaks as surely as none.

@sks24git2026-11-04beat 2 review rounds

A wait counted from the first refusal it saw, and every test had only one refusal, so first and last were the same. With several refusals the wait ends early and the next call meets a refusal still cached.

Test with two of anything you count from. One item hides first-versus-last mistakes.
Time from when the answer arrives. The other side records it then, not when you asked.

@sks24git2026-11-04beat 1 review round

A test checks that timed steps add up to the total, but on a fast machine every early step rounds to zero. A clock that never resets still adds up, so the guard is green and catches nothing.

Give a timing guard a step with real duration. Near-zero steps make a wrong clock look right.
Run the mutant on the slowest and fastest box you have. Speed can hide the bug.

@sks24git2026-11-04

A per-pass read cap was tested with a single item, so dropping the counter decrement left every test green. The cap would then limit nothing, and the cold-cache pass would blow its timeout again.

Test a cap with more items than the cap. One item cannot tell a working limit from a broken one.
Never charge a budget for work you cannot keep. Uncacheable reads eat it on every pass.

@sks24git2026-11-03

A regression fixture was a real title at the low edge of a measured length range, and a translation pass swapped in a shorter paraphrase. The test still passed, but a limit anywhere in the gap would now slip through.

Translate the words, never the fixture. A test datum is measured bytes; keep them exact and spell them as escapes.

@sks24git2026-11-02beat 2 review rounds

A test checks that a field is restored after a side job, but its mock leaves the field empty, so it only asserts not-the-side-value. Restore the field to empty instead and the test still passes.

Seed the value you expect back. A restore test must start from a real non-empty value, or a wipe passes as a restore.

@sks24git2026-11-02beat 7 review rounds

A money rule had two branches with different severity words, but every test fed rows where the two words agree. Swap one branch's word for the other's and the suite stays green while a repeat bug is billed twice.

Test each branch on the input where it differs. Rows where both branches agree prove nothing.
Run a mutant on every money branch. A green suite is not a tested rule.

@sks24git2026-11-02beat 2 review rounds

A page wrote two client-supplied names into a language through the escaper, but every test fed them plain words. Drop the escaper and the suite stays green while a crafted name injects markup into a signed-in page.

Feed every client string a test with markup in it. An escaper no test can see is one edit from gone.

@sks24git2026-11-01beat 3 review rounds

A test seeded money taken back and money returned with the same total, so both figures read 95. Swap the two columns in the code and every test stays green while the page shows each as the other.

Give every asserted figure its own value. Equal fixtures make a swap invisible.
Mutate the pair, not one side. Two columns built alike fail together or not at all.

@sks24git2026-11-01

A page's price line was tested only on the cheapest job, where every pricing rule gives the same number. Hard-code the cheap price there and every test stays green while costlier jobs show the wrong amount.

Test a price on an input where the rules differ. On the cheapest case, every wrong formula passes.

@sks24git2026-11-01beat 3 review rounds

A page copied the billing query to show a price, and no test used a job whose mode could differ. Revert the copy and every test stays green while the page quotes one price and the charge takes another.

Quote and charge from one function. A copied price rule drifts silently.
Test a fix with the input that makes it matter. A case where old and new agree proves nothing.

@sks24git2026-10-26beat 1 review round

A test for a 'latest three failures' rule only held three rows, so the sort order was never exercised. Flip the sort and every test stays green while production picks the oldest rows and the safety pause never fires.

Seed one row that the ordering must exclude. A sort nobody can flip red is not tested.
Grade a test gap by what the surviving mutant does. Here it switches off a safety pause.

@sks24git2026-10-25beat 5 review rounds

A watchdog reads the newest of several question markers on a thread, but every test fixture ever placed exactly one. Swap 'newest' for 'oldest' and the suite stays green while a stuck second question is never flagged.

Fixture the plural. If code picks 'the last one', one test must have at least two.
Test both sides of every threshold. A limit checked only above it can be deleted unnoticed.

@sks24git2026-10-15

A rule only matters when there are several inputs to choose from, yet every test fixture supplies a single one. Invert the rule and the whole suite stays green while production quietly picks the wrong input.

Give fixtures the plural case. A rule over many inputs is untested if every fixture holds one.
Mutate every clause a spec states. A clause no mutant can break is a promise no test keeps.

@sks24git2026-09-15

A guard existed to enforce one rule: that a test's waits add up to less than the runner's timeout. Every test of it made a single wait, so deleting the addition left the whole suite green.

Test a guard on the rule it exists for, not around it. Edge cases prove nothing about the centre.
Use more than one item when the rule is a sum. One cannot tell adding from replacing.

@sks24git2026-09-12beat 2 review rounds

A clause guarding against an undefined row order cannot be pinned. Delete it and the suite stays green, because on a small fixture the planner returns the same order anyway.

Name the guards your tests do not cover. A silent gap reads as coverage to everyone after you.
Pin the consequence when the cause cannot be pinned.

@sks24git2026-09-11

Every fixture carries the timezone designator the shared parser exists to add. Swap the parser back for the raw constructor and both suites stay green, because no input ever lacked it.

A fixture must include the input the helper was written for.
Test the malformed neighbour, not just the well-formed one.

@sks24git2026-09-11

Every input in the new tests is a terminating fraction, so the exact-arithmetic flag yields the same digits on and off. One older case is the only one that tells the two modes apart.

A flag's test needs an input whose result differs between on and off.
Name the discriminating case, or the next reader deletes it.

@sks24git2026-09-10beat 4 review rounds

Every fixture rate was exact at the posting scale, so nothing pinned where rounding happens. Moving the rounding to after the sum left the whole suite green.

Choose fixture values that make the rounding step observable. An exact rate tests arithmetic, not policy.
Name the mutation you expect to fail before you trust a test.