The growing staircase

About this video

Leila pays for the AI she uses by the amount. One long chat, twenty short messages, cost far more than all her short chats together. Where did the tokens go?

AI companies charge for two kinds of tokens. Input tokens are everything sent to the model in a request. Output tokens are everything it writes back, including its thinking tokens, even the ones you never see. Prices are given per million tokens, and output costs more, about five times as much for Claude's models, because every output token needs its own full step. In late 2026 Claude's largest everyday model cost about 4 dollars per million input tokens and 20 per million output tokens, and its smallest about 40 times less. One request of 12,000 input tokens and 1,500 output tokens comes to about 8 cents.

The surprise is how a bill grows. The model keeps nothing between requests, so to let it "remember", the app sends the whole conversation so far with every new message. If each message and its reply is about 500 tokens, the first request is 500 tokens, the second 1,000, the third 1,500, up to 10,000 for the twentieth. Added up, that's 105,000 input tokens for a conversation only 10,000 tokens long. You pay for the whole staircase, not the top step.

An agent does the same with every tool call: the result goes into the conversation and the whole thing is sent again. A task of a hundred steps can send millions of input tokens. Time works the same way, since output is written one token at a time. So start a new conversation for a new task, use a smaller model for simple jobs, keep big pastes out of long chats, and lower the effort on easy work.

Transcript

1. 0:00Twenty short messages, about five hundred tokens each. That should be ten thousand tokens. So why is the bill so much bigger?
2. 0:16You pay for tokens going in and tokens coming out, priced per million. Output, thinking included, costs about five times as much, because every output token is its own step.
3. 0:40One request, twelve thousand tokens in and fifteen hundred out, comes to about eight cents on the largest model. A small model costs about forty times less. So how does a bill get big?
4. 1:02The model remembers nothing between requests. So every message sends the whole conversation again. Five hundred, then a thousand, then fifteen hundred, up to ten thousand for the twentieth.
5. 1:32Added up, that's a hundred and five thousand input tokens, more than ten times what the chat looks like. You pay for the whole staircase, not the top step.
6. 1:54An agent sends everything again after every tool call. A hundred steps can send millions of tokens. And output is written one token at a time, so time is tokens too.
7. 2:18So start a new conversation for a new task. Use a small model for simple jobs. Keep big pastes out of long chats, and lower the effort on easy work.
8. 2:40You pay for the whole staircase, not the top step.