Assertion passes with the change reverted

What goes wrong

A test written after a change is never run against the code as it was. It asserts properties the earlier version already satisfied, so the one behaviour being shipped is pinned by nothing. The reader treats a green new test as proof that the new behaviour is protected, when it only records what stayed the same.

Example

A test added alongside a rewritten read asserts the rows that come back, and the old read returned those same rows, so the property the rewrite was for is unguarded.

Not to be confused with

Not Assertion Cannot Fail: this check does redden for some edits and would fail against unrelated code; it is blind only to the behaviour the change introduced.

Discovered by

@sks24git GitHub

Field notes · 10

@sks24git2026-11-08

A page change made pages follow the dark theme, but the only rendered test checked the one case forced to stay light. Pin every page light, or restore the old fixed colors, and every test still passes.

Test the headline case, not only the exception. A guard on the side path says nothing about the main one.

@sks24git2026-11-07beat 1 review round

A script ran its main block only when its own path matched the path it was started with, and a linked folder made them differ. The check then did nothing and exited clean, so the build gate passed unchecked.

Never let a gate exit clean without doing its work. A skipped check must fail loudly.
Do not guess the entry point from paths. A link and the real path are two strings.

@sks24git2026-11-04

A documentation page removes banned internal terms by hand without adding a test to scan for them. The next copy edit can reintroduce the dropped terms while every existing check stays green.

Guard every banned-phrase cleanup with a test. Prose without a check drifts back on the next edit.
Pair each rule change with an assertion. A static rule keeps the fix alive.

@sks24git2026-10-30

A race fix strips a background-added field from the positive checks but leaves the one negative check comparing the raw record. When the background write wins, that check can never match, so its bug passes.

Normalise both sides of every check on raced data. A not-equal on a raw record passes for free once a field gets added.

@sks24git2026-10-15

A layout fix restores a breakpoint, but no test checks the rule, and the only guard on that page scans for leaked data. Delete or invert the rule and every suite stays green while the phone view breaks again.

Guard each fix with a check that fails without it. An untested fix can vanish in any edit.
Read what a guard asserts, not what it loads. A test that opens a page may skip its layout.

@sks24git2026-10-02beat 1 review round

A test checks a guard's exit code and that stderr contains its message, but each case also trips a later check with the same exit code. Make the guard only warn and the test stays green.

Assert the whole stderr, not a substring. A later check's message slips past a contains-match.
Mutate the exit line, not just the condition. A warn-only guard is the likeliest edit to ship.

@sks24git2026-09-12beat 3 review rounds

A helper translated one code and returned every other unchanged. Every assertion about it used a code where input and output match, so replacing the helper with the raw value broke no test.

Assert where the two sides differ, or the assertion is decoration. An identity case cannot witness a mapping.
Arm the side that can violate the property. Name it before the expectation.

@sks24git2026-09-11

The only test reaching the branch accepted either of two exceptions. When the two coroutines serialise, the loser takes the other path, so replacing the throw with an early return survives untouched.

An assertion that accepts two outcomes tests neither.
Force the schedule you mean to test, then assert one exception.

@sks24git2026-09-10beat 7 review rounds

The new test asserted the counts the rewritten query returns, and the old query returned exactly those too. The one property the change shipped was pinned by nothing.

Run your new test against the old code. If it passes, it pins what you kept, not what you changed.
Name the mutation the test must redden before you write it.

@sks24git2026-09-10

A plan test asserted that a helper CTE appeared in the plan, but it only explained a query where that CTE was an unused join. The assertion passed on nothing while the readers that depend on it went unchecked.

Assert on the query that reads the thing. A node present in a query that ignores it proves only that the join exists.
Name the queries a test does not cover in its own doc.