A green test suite is a rumourA green test suite is a rumour

I have a line I repeat to my agents until it is boring. A green test suite is a rumour. A live URL is a fact.

It sounds like a slogan. It is a summary of three bills I paid.

Green while production was 100% broken

One of my products stores credentials encrypted in Postgres. The storage code had a unit test. The test mocked the database client and checked that the ciphertext went in as a Node Buffer. It passed. It was also the bug.

The real client serialised that Buffer to JSON text, and the database stored the text as bytes. Every ciphertext was corrupted on the way in, so every decrypt failed. No feature that needed a credential could ever have worked. The test did not miss the bug. It asserted it.

Nobody found it by reading code. It surfaced on the first real write, read and decrypt against the real database.

The function nobody called

Another product parks a post as a preview and waits for a human to confirm it. One function turns that preview into a real post. It had thorough unit tests. All green.

It had zero callers in production. The UI never rendered its button. The inbox query never fetched posts in the state it handled. A user who pressed Try again re-ran the dry run forever. Every layer was tested. The connection between two green layers was not.

231 green tests and a 500

The third one is the shortest story. 231 tests passed. The endpoint they covered returned 500 on every call.

I do not need more detail than that to learn from it. If 231 checks can pass while the one thing a user does fails every time, the checks measure something other than the product.

What a test actually proves

A unit test proves that a function works when it is called with what the test passes it. That is useful. It is not the same claim as “the feature works”.

The gap sits in three places. The seam between two layers, where nobody asserted that one calls the other. The boundary with a real dependency, where the mock answers the way you imagined and the driver answers the way it actually does. And the deploy, where the code in git is not the code that runs.

Agents are very good at closing gaps inside their own tests. They are not curious about the other places. One night twelve lane tasks merged with green suites, and all twelve needed a fix against the running system. One of them had a service fix committed to git and never deployed.

The rules my agents follow now

• Trace the call chain for every new capability. Who calls it in production? Is it imported, rendered, queried?
• Give storage that is binary, encrypted or serialised a real round trip: write, read, decode, compare.
• Make a new test fail against the unfixed code first. A test that never failed has not proved anything yet.
• After a merge, run the suite with the real environment, then use the actual surface as a logged-in user.
• Ship, then load the live page and read what it says.

None of this replaces tests. Tests are how you change code without fear. They are just not evidence that the product works. They are evidence that the code matches the tests.

So the last question in my verification step is not “are the tests green”. It is: did a real request go through the real path and come back right? The orchestrator answers it by running the request, not by reading the report. How often that answer differs from the lane’s own report is on Research.