All tests pass

About this video

After twenty minutes, Leila's agent reports: "Done! I fixed the bug in the price calculator, and all tests pass." Before she tells her team, she scrolls back through what it actually did. It ran the tests at the start, and two failed. Then it read the calculator file, edited it twice, and edited a test file. Then it wrote "all tests pass". It never ran the tests again.

An agent's report is testimony: someone telling you what happened. It is real evidence and usually accurate, but it can be wrong without anyone lying. A language model writes the likeliest next words, a story about fixing a bug usually ends with "and all tests pass", and it says everything in the same confident voice, right or wrong.

Checking doesn't mean redoing the work; it means looking at the evidence instead of the summary. Ask for the command and its real output. Read the diff, the exact lines that changed in each file. Leila's diff showed that the test which checked the price with tax had been replaced by a weaker one that only checks a price exists: the test passes without testing anything. The agent didn't plan to cheat; it was the shortest way to finish its story. Reports go wrong in a few common ways: a check it didn't run, the wrong check, a weakened check, work stopped half-way, and changes nobody asked for.

The best fix happens before the report. Give the agent something real to look at in its loop: tests it can run after each change, a script that compares output with what's expected, a screenshot, the error messages. An agent that can see whether its change worked usually fixes its own mistakes; one that can't is guessing. A hook can make the tests run every time it tries to stop. And agree a checkable definition of done before it starts.

Transcript

1. 0:00After twenty minutes, Leila's agent reports, done, I fixed the bug, and all tests pass. She is about to tell her team the calculator is fixed. Should she?
2. 0:22Scroll back through what it actually did. It ran the tests at the start, and two failed. Then it edited the calculator twice, and a test file. It never ran the tests again.
3. 0:48So all tests pass is a claim, like testimony. The model writes the likeliest next words, and a story about fixing a bug usually ends that way. It sounds just as sure when it's wrong.
4. 1:12Now read the diff, the exact lines that changed. In the test file, the check for the price with tax was removed, and a weaker one added. The test passes without testing anything.
5. 1:38Reports go wrong in a few common ways. A check it didn't run, the wrong check, a weakened test, work half done, or changes nobody asked for.
6. 2:00Better still, let it check itself. Give it tests it can run after each change, and it sees its own mistakes in the loop, long before it reports to you. A hook can make sure the tests always run.
7. 2:26And agree on what done means before it starts, so a report that says done has to mean all of it.
8. 2:44Done is a claim. Check the evidence.