One Token at a Time

When you ask a chat assistant something, the answer doesn't appear all at once. It flows onto the screen a few words at a time, as if someone were typing. Is that just a nice effect, or is the model really writing it bit by bit?

Leila I always assumed the answer was ready and the app shows it slowly so it feels friendly.

Omid A reasonable guess, and some apps do smooth it out a little. But the model really does produce it piece by piece. Remember what it learned to do in training?

Leila Guess what comes next.

Omid Exactly that, and only that. You give it some text, and it gives you its guess for the next token. One token.

Leila One token? Then how does it write a whole page?

Omid What would you do with one token, if you wanted a page?

Leila Stick it on the end of the text... and ask again? "Now what comes next?"

Omid That's how it works. The new token is added to the end, the whole text goes back in, and the model guesses the next one. Then again, and again, hundreds of times, until it guesses a special token that means "I'm finished", or it reaches a length limit.

Leila So the answer appears bit by bit because it's being made bit by bit.

Omid Right. And it's not exactly a single guess each time. It's more interesting than that.

A probability for every token

At each step, the model doesn't just name one next token. It gives every token in its vocabulary a probability: how likely it is to come next. Most get almost nothing. A few get most of the chance.

Then one token is picked from these chances. This step is called sampling. Usually it picks one of the likely tokens, but not always the single most likely one. That's on purpose: always picking the top token tends to make text that is dull and repetitive.

How adventurous the pick is can be set with a dial called temperature: low means "nearly always pick the top tokens", high means "give the less likely ones a real chance". Chat assistants and agents set it for you.

Put together, writing an answer is a loop:

This is next-token prediction, and it is the only thing a language model does. Everything else, from a poem to a plan for fixing a bug, is built out of these single steps.

What this explains

Why answers stream in. Each token needs a full step of work by the model. A long answer is many steps, so it takes time, and apps show each token as soon as it's ready.

Why the same question gets different answers. At each step a token is picked from a range of likely ones. Pick a different token early on, and the rest of the answer follows a different path. Ask the same question in two new chats and you'll often get two different, equally reasonable answers.

Leila Here's something I've noticed. Once it starts on a wrong idea, it often keeps going with it, even piling up reasons.

Omid Why might that happen, if it writes one token at a time?

Leila Because... the wrong part is now part of the text. And it's guessing what comes next after that text.

Omid Yes. A token, once written, is never taken back. Every later step continues the text as it now stands, including the mistake. What usually comes after a confident wrong claim in a piece of writing?

Leila More confident claims backing it up. Oh, that's not great.

Omid It's why the way an answer starts matters so much. And it's one of the reasons models now often think before they answer, which is the next lesson.

Try it

Open two new chats with the same assistant and ask both exactly the same open question, like "Give me a name for a bakery." Compare the answers. Then ask a question with one right answer, like "What is 12 times 12?", in both. Notice where the answers differ and where they don't.

Using it well

1. If an answer heads the wrong way early, start it again rather than arguing. Everything it writes later continues the wrong start. Many apps let you regenerate or edit your message; use that.
2. Ask again for a second opinion. Because each answer is one path among many, a fresh try at an important question can show you whether the first answer was solid or just one possibility.
3. For anything that must be exact, ask for something you can check: the calculation written out, the source, the command it ran. A fluent answer is what the model is best at; a correct one needs checking.

Words to know

• next-token prediction - what a language model does: given some text, give a chance to every possible next token
• sampling - picking the next token from those chances, usually one of the likely ones but not always the top one
• temperature - a setting for how adventurous the pick is: low sticks to the most likely tokens, high gives unlikely ones more of a chance

Big idea

A model writes one token at a time: it gives every possible next token a chance, one is picked, added to the text, and the whole thing goes back in. That's why answers stream in, why the same question can get different answers, and why an early mistake tends to stay.