Compaction in Code

mini now prints the size of every request, and on Leila's big project she watches it climb: 40,000, 90,000, 140,000 tokens. Somewhere past the budget, something has to give. Claude Code would say the conversation was "compacted". Now mini has to do it itself, which means deciding exactly what to keep, and what the next request will look like.

Leila The easy way is to throw away the oldest messages until it fits.

Omid What's in the oldest message?

Leila The task. Fine, keep message 0 and throw away from message 1 on.

Omid Message 1 is the model's first reply: a tool_use, say, reading a file. Message 2 is its tool_result. What happens if you drop message 1 and keep message 2?

Leila There's a result for a call that isn't there any more.

Omid And the API refuses that request: a tool_result has to answer a tool_use in the message just before it. The other way round fails too: a tool_use with no result after it. So whatever you drop, a call and its result go together.

Leila And even if I drop them in pairs, I lose what those results said. If mini read the config file at turn two, that's just gone.

Omid Which is why most agents don't only drop things. They ask the model to write down what matters first.

The summary request

Compaction in mini is one extra request: everything so far, plus a note asking the model to summarise the session for itself. Then mini replaces the whole history with the task and that summary.

def compact(messages, system, tools):
    """Replace everything but the task with a summary written by the model.
    Called right after tool results were added, so the last message is the user's."""
    task = messages[0]
    ask = {"type": "text", "text": (
        "Before going on, summarise this session for yourself: the task, what is done, what is decided, "
        "which files changed, and what is left. Be specific. Don't use any tools in this reply.")}
    last = {"role": "user", "content": messages[-1]["content"] + [ask]}
    reply = client.messages.create(model=MODEL, max_tokens=4000, system=system, tools=tools,
                                   messages=messages[:-1] + [last], cache_control={"type": "ephemeral"})
    summary = "".join(b.text for b in reply.content if b.type == "text")
    if not summary:  # it asked for a tool instead: keep going, try again next turn
        return messages
    return [{"role": "user", "content": f"{task['content']}\n\nProgress so far (a summary):\n{summary}"}]

What to notice:

• The note goes into the last user message, next to the tool results that are already there. The results still answer the model's last tool calls, and the roles still take turns.
• The summary request uses the same system and tools as every other request. So it starts exactly the same way, and with the cache_control line it can read the long history from the cache instead of paying for all of it again (the last lesson in this module explains both). For the same reason it keeps the same model and settings: changing the effort here, to save a little, would make it pay full price for the whole history.
• The tools are there, but the note asks for none. If the model asks for a tool anyway, the reply has no text, and mini simply carries on uncompacted and tries again next turn.
• What the summary asks for is the real design. The task, what's done, what's decided, which files changed, what's left: that's what the model needs to carry on. A summary that just says "we worked on the report" would leave it lost.

When to compact

mini checks after every turn, using the request size from the last lesson:

        u = response.usage
        if u.input_tokens + u.cache_read_input_tokens + u.cache_creation_input_tokens > CONTEXT_BUDGET:
            messages = compact(messages, system, tools)

It runs after the tool results are added, which is exactly when the last message is a user message, as compact expects. And it uses the size of the request just sent, which is a good guess at the next one.

Two things are lost, and you should know which. The details of everything before, as in context management: whatever the summary leaves out is gone for good. And the newest tool results: the file mini read a moment ago survives only as the summary's description of it, so the model may read it again. That's another reason to keep decisions in files: a plan written to PLAN.md survives any compaction, because the model can always read it back.

Trimming instead

There's a cheaper way, with no extra request: drop the oldest turns and keep the rest word for word. It loses more (nothing is summarised), but it costs nothing and keeps the recent results exact. Some agents trim first and compact only when trimming isn't enough.

If you trim, the pairing rule is the whole job:

• Keep message 0, the task.
• Drop whole turns from the oldest: an assistant message and the user message after it, together. A tool_use then never loses its tool_result, and the roles still take turns.
• Keep the newest turn, which the model is about to answer.

The exercise after this lesson is exactly that function.

Try it

Set CONTEXT_BUDGET = 20_000 in a copy of mini and give it a task that reads several files. Watch for the turn where it compacts, and print the summary it wrote. Is it the summary you'd have written? What's missing? No key? Write, in five lines, the summary you'd want mini to keep after the session in the first picture: task, done, decided, changed, left.

Using it well

1. Decide what the summary must contain, and ask for it by name. The summary is all the model will know.
2. Use the same system prompt, tools and model for the summary request. It's the cheapest request to make, and the most consistent.
3. Never split a tool call from its result. Drop or keep them together, and keep the roles taking turns.
4. Write the plan to a file. A file is the one thing compaction can't lose.

Words to know

• compaction - replacing an agent's history with the task and a summary the model writes, to make room in the context
• trimming - dropping the oldest whole turns of the history, with no summary; cheaper, and loses more
• turn pair - an assistant message and the user message after it, which hold a tool call and its result and must be kept or dropped together

Big idea

Compaction in your own agent is one extra request - "summarise this for yourself" - and a new, short history built from the answer. Whether you summarise or trim, a tool call and its result always go together, and what you ask the summary to keep is what the agent will know.