AI School › How it’s built inside · Lesson 63 of 130

🧩 What tokens are

Text gets chopped into little pieces called tokens.

When you eat a chocolate bar, you do not swallow it whole: you break it into squares. AI does the same with words. Before it can work on a sentence, it cuts it into little pieces called tokens. A token can be a whole word like "cat," or a piece of a word like "kitt" and "en" for "kitten," or even a single symbol like a period. It is like snipping a sentence with scissors into many small tiles, so the AI can handle them one at a time instead of gulping everything in one bite.

Why chop things up? Because this way the AI can handle even words it has never seen. If it meets "supermegacool," it does not freeze: it splits it into known pieces and manages. Counting tokens matters for another reason too: AIs have a limit on how many tokens they can read and write at once, a bit like a backpack that fits only so many books. Longer sentences use more tokens, and each token costs the computer a little work. So next time you write to an AI, remember: it does not see "words," it sees a row of tiles to fit together.

🎒 Try it in class (or at home)

Take a long sentence and split it with a pencil into pieces of 3-4 letters: that is how AI sees tokens.

← Lesson 62: The layers of the network · Lesson 64: Words become numbers →

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.