One token at a time

About this video

When you ask a chat assistant something, the answer flows onto the screen a few words at a time. That isn't an effect: the model really does make it piece by piece. All a language model learned to do is guess what comes next, so given some text, it gives its guess for the next token. One token.

At each step it gives every token in its vocabulary a probability, a chance of coming next. After "The cat sat on the", mat might get 41 percent, floor 17, sofa 9, bed 6, and thousands of others almost nothing. Then one token is picked from these chances. This step is called sampling: usually it picks a likely token, but not always the top one, since always picking the top tends to make dull, repetitive text. A setting called temperature decides how adventurous the pick is. The picked token is added to the end of the text, the whole text goes back in, and the model gives chances for the next one, again and again, until it picks a special end token or reaches a length limit. This is next-token prediction, and it is the only thing a language model does.

It explains three things. Answers stream in, because each token is a full step of work. The same question can get different answers, because picking a different token early sends the rest down a different path. And an early mistake tends to stay: a written token is never taken back, and every later step continues the text as it stands, so after a confident wrong claim come more confident claims. If an answer heads the wrong way early, start again rather than arguing; ask again for a second opinion; and for anything exact, ask for something you can check.

Transcript

1. 0:00An AI's answer flows onto the screen a few words at a time. Is it ready and shown slowly, or really being written as you watch?
2. 0:18It's written as you watch. For the text so far, the model gives every possible next token a chance. After "the cat sat on the", mat gets forty-one percent, floor seventeen, and thousands of others almost nothing.
3. 0:44Then one token is picked from those chances. Usually a likely one, but not always the top one. A setting called temperature decides how adventurous the pick is.
4. 1:06The picked token is added to the end, and the whole text goes back in. Chances, pick, add, and round again, hundreds of times, until an end token or a length limit. That is all a language model does.
5. 1:36So the same question can get different answers. Pick a different token early, and the rest of the answer follows a different path.
6. 2:02And a token, once written, is never taken back. Every later step continues the text as it stands, mistake included. After a confident wrong claim come more confident claims.
7. 2:26If an answer starts the wrong way, start again rather than arguing. Ask again for a second opinion. And for anything exact, ask for something you can check.
8. 2:44One token at a time. Chances, pick, add, again.