Guard scans less than it claims

What goes wrong

A check runs over a subset of the material it is described as covering, because of where it starts reading, what it filters out, or which files the tooling actually looks at. Anything outside that subset is never examined, yet the pass is read as a verdict on the whole.

Example

A check that forbids a heavy dependency reads only the top of each file, so the same dependency declared further down passes unnoticed.

Not to be confused with

Not Only Some Listed Cases Tested, where a short list is honestly worked through; here the check claims the whole input and quietly reads part of it.

Discovered by

@sks24git GitHub

Field notes · 18

@sks24git2026-11-04

A strict shell script ran a bare command substitution, so when it failed the script stopped with no message. Its self-check looked at one branch of an if, so moving the step to the other branch stayed green.

Give every command that can fail in a strict shell script an explicit refusal. Silent exits look like success.
Check both branches of a guarded step. One-sided checks miss moves.

@sks24git2026-11-03beat 1 review round

A check that a document names only known steps was narrowed to fix false alarms, and it stopped reading one of the two words the document uses for a step. A typo in that wording now passes the check unseen.

List every form the old check caught before you narrow it. A fix for noise can drop real catches.
Keep one case per form in the check's own test.

@sks24git2026-11-03beat 1 review round

A drift test pulled each name only from the word right beside "pass", so in a list only the last name was checked. Put a wrong name first in "the A and B passes" and the test stays green.

Test a drift guard with a list, not one item. Prose names things in lists, and a matcher that reads one neighbour skips the rest.

@sks24git2026-11-02beat 2 review rounds

A rename sweep is guarded by scans that check old addresses and links, but nothing checks the old name as plain words on the site. The next edit types the name back into a page and every check stays green.

Guard the claim, not the edit. Scan for the plain word, not just the links you replaced.
Fix the guide agents read first. A stale name there comes back in their next edit.

@sks24git2026-11-01beat 1 review round

A test pinned a page block by checking a list of fragments it should contain, and each review round found one more line the list left out. Any sentence not on the list could change or vanish with every check still green.

Pin generated text whole, against independent sources. A list of required fragments is only as good as what you remembered.

@sks24git2026-10-30beat 5 review rounds

A status check read only the last few hundred bytes of a log line, but the error body comes after the status key. A long provider error page pushes the key out of view, so the run's failure is never read.

Size a tail window by the longest field after the key you need. A short window works for one line shape and goes blind on the next.

@sks24git2026-10-28beat 2 review rounds

A guard that failed loudly when a skip-list entry vanished was relaxed to fire only for modules still in the run. Rename the module and the heavy test quietly returns to every pull request.

Keep the loud failure when the list is empty. A guard that checks only what is present cannot see what was renamed away.

@sks24git2026-10-27beat 1 review round

A CI fast path picks tests by import graph after a source change. One guard test scans the whole source tree from disk and imports nothing, so a bad change passes the fast path and fails only in the merge queue.

Imports are not the only dependency. A test that reads sources from disk must always run; list them by grep, not by hand.
Mutate the guard, run the fast path. It must go red.

@sks24git2026-10-27

An affected-tests selector found no test for a schema migration: migrations load by path, and the fixture running them lives in test setup. Empty selection counted as nothing to run, so the pull request passed green.

Empty selection means unknown, not safe. Zero tests picked for a real change means run everything.
Files read by path are off the import graph. Give them an explicit rule.

@sks24git2026-10-27

A CI path filter called one directory front-end only and skipped every back-end test when nothing else changed. The back-end contract tests read fixtures from that very directory, so a broken contract merged green.

Classify paths by their readers, not their folder. Grep who opens a directory first.
Keep the full suite on the merge path. A filter in the queue is a hole.

@sks24git2026-10-18

A clean split of one big test file into topic files kept every test, but the CI gate still named only the old file. Merged alone, most of the moved tests stop running in CI until someone edits the gate list.

Count tests where CI runs them, not where they load. Same count and hash can hide a gate that skips them.
Ship the gate edit with the move. Or schedule it for the minute after merge.

@sks24git2026-10-17

A size check skips whole folders it lists as data, but two of those folders also hold hand-written page code. That code is never measured, so it can grow past the limit while the check stays green.

Exclude files by type, not by folder. A folder named for data still collects code.
Expand every skip glob once and read the list. A glob you never expanded hides what it drops.

@sks24git2026-09-12beat 2 review rounds

A prop type was narrowed so the compiler would enumerate every caller. The test fixture kept the old wider type and nothing complained: test files sit in no build config project, so no compiler reads them.

Check your test files are inside a build config project before trusting a type as a gate. Unchecked fixtures keep advertising contracts the code has dropped.

@sks24git2026-09-10beat 7 review rounds

Half the labels are looked up with keys built by string interpolation, which the missing-key checker skips by design. Rename one entry in the catalogue and nothing anywhere reports the gap.

A key your checker cannot see needs a test that reads the catalogue. Interpolated lookups are invisible to static scans.

@sks24git2026-09-10

A census mapped over constants and returned nothing for any shape it did not recognise, so unmatched entries silently vanished. Add an admitted route in a new shape and the guard checks nothing about it.

Make an unrecognised shape fail loudly instead of mapping to nothing. A silent skip shrinks the corpus without shrinking the claim.

@sks24git2026-09-10beat 4 review rounds

The scanner drops whole lines it treats as noise, so a suite invoked after an echo on the same line leaves the corpus unseen. The per-path zero-count control cannot notice while other lines still match.

Drop tokens, not lines. A line-granular filter also hides whatever shares a line with the thing you meant to skip.

@sks24git2026-09-10beat 2 review rounds

The wiring check read only the job's condition and dependencies, so appending a swallow-the-error suffix to the command left it green. Dropping the flag that invokes the check did the same.

A disarming edit must be a diff in something pinned. Pin the exact command line and the error-tolerance keys, or the check switches off untouched.

@sks24git2026-09-09beat 2 review rounds

A module imports a heavy package only where it is used, and a guard checks that nothing above the entry point mentions it. Static imports load from anywhere in the file, so rewriting that one line in place stays green.

Grade the kind of a dependency, not its position. The loader does not care where you wrote the line.
Strip the allowed form, then refuse whatever still mentions it.