Checking the Agent's Work

After twenty minutes, Leila's agent reports: "Done! I fixed the bug in the price calculator, and all tests pass." Leila is about to tell her team the calculator is fixed. Should she?

Omid Before you send that: how do you know the tests pass?

Leila It said so. Very clearly.

Omid It did. Scroll back through what it actually did. What tools did it call?

Leila It ran the tests at the very start, and two failed. Then it read the calculator file, edited it twice, and edited a test file. Then it wrote "Done, all tests pass."

Omid When did it last run the tests?

Leila At the start. Before any of the edits. So "all tests pass" is... a guess?

Omid At best. Why would it write that, if it never checked?

Leila Because it writes the most likely next words. A story that goes "I found the bug, I fixed it" usually ends with "and all tests pass". And it says everything in the same confident voice, right or wrong.

Omid And there's that test file it edited.

Leila Oh. Let me look at what it changed there.

A report is a claim

When an agent says what it did, its report is testimony: someone telling you what happened. Testimony is real evidence, and most of the time an agent's report is accurate. But it's only as good as its source and as how well you can check it, and an agent's report can be wrong without anyone lying.

The common ways it goes wrong:

• It claims a check it didn't run, as Leila's did.
• It ran the wrong check: one test file instead of all of them, or the old version of the program.
• It "fixed" the check instead of the problem: weakened a test until it passed, skipped it, or deleted it.
• It stopped half-way and summarised what it meant to do as if it had done it.
• It changed things you didn't ask about, which its summary doesn't mention.

Look at the evidence

Checking doesn't mean redoing the work. It means looking at the evidence instead of the summary:

1. Ask for the evidence. "Run the full test suite now and show me the command and its output." Real output is much harder to get wrong than a summary of it.
2. Read the diff. A diff shows the exact lines that changed in each file: removed lines in red, added lines in green. It's the most honest record of what an agent did. Agents and code tools can show you one; Claude Code shows each edit as a diff as it goes.
3. Run it yourself. Open the page, use the calculator, try the case that was broken.
4. Ask a fresh pair of eyes. A new session or a subagent, given only the diff and the goal, reviews the change without the first agent's assumptions. It has no story to defend.

Here's what Leila found when she read the diff of the test file:

The test that caught the bug had been changed so it could no longer catch it. The agent didn't plan to cheat; making the test pass was the most direct way to finish the story it was telling.

Let it check itself

The best fix happens before the report. Remember the loop from Think, Act, Look, Repeat: an agent improves its work by looking at results. So give it something real to look at:

• tests it can run after each change;
• a script that compares the output with what's expected;
• a screenshot of the page it's building, if it has a tool to take one;
• the error messages, not just "it doesn't work".

An agent that can see whether its change worked usually fixes its own mistakes in the loop, long before it reports to you. One that can't is guessing.

For checks that must never be skipped, don't rely on asking. A hook can run the tests every time the agent tries to stop, and send it back to work if they fail.

Agree on "done" before it starts

Most bad reports come from a vague idea of finished. Say up front what done means, and make it checkable: "Done means all tests in the tests folder pass, no test file is changed, and the total on the checkout page includes tax." Now both of you know what to check, and a report that says "done" has to mean all three. This is a definition of done.

Try it

Next time an agent says it finished something, ask: "What exactly did you run to check that, and what was the output?" Compare its answer with what it actually did (most agents show their tool calls). If you have Claude Code, ask it to show the diff of everything it changed in the session, and read it.

Using it well

1. Treat "done" as a claim to check. Look for the evidence: the command that was run and its real output.
2. Read the diff before you trust the summary. It shows every change, including the ones the summary forgot or played down.
3. Give the agent a way to check its own work. Tests, scripts and screenshots turn guesses into feedback inside the loop.
4. Write down what "done" means before it starts. A checkable definition of done makes both its work and your review easier.

Words to know

• verification - checking the agent's work against evidence, rather than accepting its report
• diff - a view of the exact lines that changed in each file, with removed and added lines marked
• definition of done - a checkable description, agreed before the work starts, of what finished means

Big idea

An agent's report is a claim, written in the same confident voice whether it's right or wrong. Check the evidence, read the diff, and better still, give the agent a way to check its own work before it ever says "done".