One token at a time

Two answers

Ask ChatGPT to describe the sea in one sentence. Then ask it again. The same question, word for word, and a different answer. So it cannot be looking the answer up. Each answer is built from tokens. A token can be a word, a piece of a word, or just a comma. The two answers match, token for token, up to here. To see why they part, go back to that moment.

The next token

Four tokens are on the page, and the model has one job: choose the fifth. It reads everything written so far, and gives a score to every token it knows, more than a hundred thousand of them. The scores become chances. Vast is the favourite, at about one in three, but nothing is certain.

Then it rolls the dice, with those chances as the odds. Last time, the roll landed on vast. This time: restless. Restless joins the text. Now the whole thing runs again: five tokens in, new chances, another roll. And again, and again, one token at a time, until it draws a full stop. No finished sentence was ever waiting. Each token was chosen only when its turn came. How bold the dice are is a setting, called temperature. Turn it down, and the favourite almost always wins. Turn it up, and long shots get through. ChatGPT usually runs somewhere in between.

One path of many

So every answer is one path through a tree of possible sentences, picked one roll at a time. ChatGPT is not looking your answer up, or reading it from a finished draft. It asks one question, over and over: given everything so far, what comes next?