Testing and Shipping34 of 37
For readers who know a little Python.
In the lesson Testing an Agent you saw why an eval runs each task several times: the same task can pass on one run and fail on the next. Here you write the part that turns a pile of run records into a report you can act on.
The input. One run per line: the task's name, pass or fail, and what the run cost in dollars. Blank lines and lines starting with # are skipped (that's done for you).
fix-test pass 0.40
fix-test fail 0.52
add-readme pass 0.10
What to print. Write grade(records), which returns the report's lines:
- One line per task, in the order each task first appears: how many runs passed, as a fraction and a whole percentage, the task's total cost, and its cost per pass (total cost divided by the number of passes), or no passes. For example fix-test: 4/5 passed (80%), 2.04 dollars, 0.51 per pass.
- flaky: and the tasks that passed some runs and failed others, separated by , , or none.
- always failing: and the tasks that never passed, or none.
- total: the number of runs, how many passed with the percentage, and the total cost: total: 10 runs, 9 passed (90%), 2.54 dollars.
Percentages have no decimals (f"{x:.0f}") and dollars two (f"{x:.2f}").
Why per pass? A cheap task that fails half the time can cost more per success than an expensive one that always works. The cost you actually care about is what one finished job costs, failures included.
Why flaky matters. An always-failing task tells you something is missing: a tool, a permission, an instruction. A flaky one is worse in a way, because a single run can't tell you whether a change helped. Those are the tasks to run more often, and the first place to look when you change the system prompt or the model.
Sign in to try this exercise.