The Gate That Measures Nothing

A nightly database backup failed sixty times out of sixty runs, and the alert that existed to say so had been firing into a page nobody was ever going to read. That is one instrument. An audit of latent defects across the customer-installed agent product we build found the same shape eleven times over, and the sweep that followed it turned up five more across our own tooling and that product’s repository. Every one of them had been green, or silent, for its entire life. None of them could have gone red, because none of them could observe the layer it claimed to check.

The rule that came out of it is short. A green check is evidence only if the check can observe the layer it claims to check. Until you have established that, the green is not a fact about the system. It is a fact about the check.

Five instruments, none of which could fail

The nightly backup had never once produced an artifact, going back to its first run in mid-June. The cause was a platform limitation, not a misconfiguration: export is unsupported for any database containing a virtual table, and a full-text search index had landed in that database in a migration. The chosen mechanism was structurally incapable of ever succeeding. The job’s failure handler wrote an error annotation onto the log page of the run that had just failed. It fired sixty times and reached nobody.

A fleet CI health check asked for the single most recent workflow run across all workflows on the default branch and used it as the verdict for the whole repository. Whichever workflow finished last decided the result, so any workflow that completed more recently masked every failure beneath it. On the repository where we found it, the probe reported healthy while four workflows were failing on the same branch, two of them with streaks that filled the entire inspection window. The old check also produced a non-deterministic answer: rerun it after any workflow completed and the same repository could flip.

A security workflow audited a minority of the npm projects the dependency configuration declares. The unaudited ones carried high-severity advisories the whole time. Worse, one of them was a workspace member, so an audit run there walked up and audited the root tree instead: five hundred and two dependencies rather than the two hundred and seventy its own lockfile describes. A matrix leg pointed at it would have been a second copy of the root leg wearing a different name, green for as long as it existed while the advisories in its own lockfile went unobserved. The same workflow had permission to read the repository and nothing else, so it could not have opened an issue or notified anyone even in principle. It was red on the main branch for nine consecutive days in August and thirteen days in July, and the only record either time was a summary page.

Four shell test suites, one hundred and three assertions between them, had never been run by any workflow or any package script. Sixty-nine of those assertions cover secret-leak detection. They all passed when we finally ran them, but nothing knew that, and nothing would have noticed them going red.

In the repository for the customer-installed agent product, a regression monitor recorded 2,059 startup failures out of 2,059 runs since it landed in early May, with zero successes ever. It had never posted the comment it exists to post. The cause was a permissions declaration: the calling workflow declared none, the repository default is read-only, and the called workflow needs write access to issues. A workflow may only grant permissions its caller already holds, so the platform refused the run before any job existed.

Why this class is silent by construction

A monitor that produces a wrong answer is a bug you can find. A monitor that produces no answer is harder, because the only evidence is an absence, and absences are not something anyone counts.

The regression monitor is the clearest case. A startup failure produces no job and no check run, so the CI notification sink emitted nothing for it. Over the same window that sink recorded 6,214 events for that repository, successes included, and not one of them was a run of the dead workflow. The sink was working. It had simply never been given anything to say. Nobody looks at a list of six thousand notifications and notices which one is missing.

The backup is the same story with a different mechanism. The job went red every night for two months, and the handler that was supposed to escalate it wrote into the log of the failed run. Checking that the job had gone green would not have helped either, because the previous handler had been “working” by that standard the whole time.

The audit that surfaced the class in the product repository turned up instruments with the same property in more inventive forms: a database fake that ignored SQL entirely, so a wrong WHERE clause was structurally invisible to every test that used it; a parity test that pinned a hand-written transcription of a shell script rather than the script itself, so mutating the real derivation, which changes the bearer credential on every deployed seat, still passed ten times out of ten; a workflow step whose findings were killed by shell error handling before they could print, so the control fired and reported nothing; a mock that echoed a fixed payload and acknowledged nothing the caller sent, so every assertion about what had been applied passed regardless.

This is not a feature that was never wired

We have written before about built-but-unwired features: code that merged, passed its tests, and never entered the path that real output travels. The audit discipline there is to start from a guarantee the product makes and trace backward to where it is enforced.

This is the layer above that. Here the feature under inspection is the instrument, and the instrument is the thing you would normally use to answer the question. When a safety filter is unwired, a working test suite can still catch it. When the test suite is the unwired thing, the ordinary method of finding out has already failed. The two classes compound: an unwired feature guarded by a blind check is invisible twice.

The practical consequence is an ordering rule. When a check says the code is fine and the system says otherwise, the check is a suspect, not a witness. That sentence is now the third rule in a debugging skill we wrote off the back of the audit, sitting alongside Agans’ nine rules and the reproduce-before-theorising discipline from Zeller’s Why Programs Fail. It is there rather than in a document of advice because the alternative ordering, which is to trust the check and go hunting in the code, is what kept eleven instruments alive.

An instrument has to be proven able to fail

Every fix in this batch had to carry a demonstration that the new check could go red for the right reason. Not an argument that it would. A run.

For the backup, the alert path was exercised deliberately: a forced failure opened a deduplicated issue, a second forced failure added a comment to the same issue rather than opening a second one, and the test commits were reverted. An alert whose arrival you have not personally watched is a claim, not a check.

For the fleet health probe, the new test suite was run against two deliberately mutated implementations. Reintroducing the original defect, inspecting only the first workflow, failed four of thirteen cases by name. Treating skipped runs as decisive failed one. Restored, all thirteen passed. A suite that has only ever been green is indistinguishable from a suite that cannot go red.

For the security audit, each matrix leg now runs against an isolated copy of its own manifest and lockfile and asserts that the package count npm audited equals the count its lockfile describes. That assertion was itself tested against a known-bad case, with the isolation deliberately removed, and it failed with the message it was written to produce.

For the shell suites, the runner discovers suites by pattern rather than from a hardcoded list, because a hardcoded list re-creates the defect the moment someone adds a fifth suite and forgets to register it. Finding zero suites is a failure, because a discovery step that silently matches nothing is indistinguishable from a passing run. Both modes were exercised against the shipped script.

There is a weaker move that looks like the same thing and is not. Deleting the code under test and watching the suite go red proves less than it appears to, because a suite can fail on a missing import. The stronger test is to mutate the comparison so it always passes and watch the suite fail anyway. Absence-failures are weak evidence. Mutation-failures are the real thing.

The positive control

The last piece is the one that generalizes furthest, and it comes from a different lane: reviews that report finding nothing.

An absence is not a low-confidence finding. It is a differently evidenced one, and it needs its own admission standard. A claim that a repository contains zero instances of a dangerous pattern means something only if you can show the instrument was capable of returning a non-zero result against that repository. A sibling pattern that returned matches, a planted fixture, or a self-test inside the probe will do. One of our reviews reported zero unsafe subprocess invocations, and that claim was worth something because the same scan located thirty-six subprocess call sites. The same report’s finding of zero security headers shipped with no positive control at all, so nothing showed the probe could have returned a header had one been there.

Every instrument in this article would have been caught by that one question, asked once, at the time it was built: show me this thing failing. Not the system failing. The instrument.

The backup job would have failed the question on day one. So would the health probe, the audit matrix, the unrun suites, and the regression monitor that never started. The cost of asking is one deliberate breakage and one run. The cost of not asking was sixty missed backups, two months of undetected red builds, a set of unaudited dependency trees, a hundred and three assertions rotting in place, and 2,059 runs of a check that had never once executed.