Trust You Can't Verify

When a change goes in, the tooling reports success, and the report is true as far as it goes: the change was accepted. What I need to know is usually a harder shape — whether the old thing stopped, whether there's data at the far end of a job that says it ran. Those are claims about an absence, and nothing in the stack is instrumented to report one.

The signing-key version of this — a rotation that applied clean and left the retired key live in the JWKS — I've written up separately. The habit came out of the backups.

What the drill compares

Completed on a backup job means the job finished. It is no claim about restorability.

So every layer gets drilled against a baseline captured before the restore: row counts, object counts, known keys with known values, written down while the system is healthy. Restore, then diff against that. Cluster state comes back in about 1.5 seconds, a namespace's volumes in about 50, a full VM in about 2.5 minutes — and the times are the least interesting output. The pass condition is captured data, not the restore tool's exit code, because the tool is the thing under test and it doesn't get to grade itself.

The nastiest failure here is silent by construction. A database dump piped into an encrypt step: the dump fails, the pipe hands the encrypt step an empty stream, and encrypting nothing yields a completely valid artefact — right extension, right headers, decrypts fine, exits 0. About 200 bytes. Next to a real dump the only thing separating it from success is the size column, which nothing in the pipeline reads. The job now aborts if its output is under 1 KB, which is crude and has a limit worth stating: it catches an empty dump and misses a truncated one. A dump that dies eighty percent through is megabytes of plausible SQL. Only the restore-and-diff catches that.

The negative control

A restore that runs while the original infrastructure is still in reach proves the data came back. It doesn't prove where from. A mount still attached, a cached credential, a snapshot repository that still resolves — any of them can quietly carry the restore, and the output looks identical either way. So the off-site drill runs as an isolation control: a clean machine that was never part of the cluster, no route to the source, nothing cached, holding the encrypted bucket and the key. Cutting every alternate path is what makes it a control.

Not hypothetical. An object-store lifecycle rule I'd set for cost reasons expired the wrong prefix and destroyed a backup repository. The scheduled jobs carried on reporting success, writing into a repository whose history was gone. The drill found it; nothing on the monitoring side moved.

Someone else's version of the gap

In November 2025, Nikon revoked the certificate for a Z6 III. A June 2026 security analysis of C2PA — researchers at UMBC, Hacker Factor and the NSA — inspected a manifest signed with that certificate more than six months later. Adobe's Content Credentials Inspect reported the signature valid. Verifieddit reported it invalid. In the paper's words: "Neither conforming validator reports the revocation."

Both conform. The spec makes revocation checking optional out of privacy concern, permits it only over OCSP, and expressly forbids CRLs — so a standard built to answer is this media what it claims to be leaves is this signer still trusted to implementer preference.

The one I can't drill

I have no equivalent for a model. Nothing captured beforehand to diff against, no isolation control cutting the alternate paths, no probe that comes back rejected when the answer is wrong — the failure and the success are the same fluent paragraph. I still use them daily; the output just goes through something that fails loudly first.

The drill runs monthly: baseline capture, restore onto a clean target, diff. The 1 KB floor sits upstream of all of it, in the nightly job.