AI Dictionary › AI Fundamentals

Tokenizer

Tokenizzatore

The tokenizer is the component that splits text into minimal units called tokens before the model can process them. Think of a cook who, before cooking, cuts ingredients into standard pieces: not a whole carrot, but predictably sized slices. Likewise the tokenizer turns "intelligence" into fragments like "intelli" + "gence", mapping a common word to one token and a rare one to several. Each token becomes a number the model can handle. How the text is cut (by words, sub-words or characters) shapes how much text fits in memory and how much it costs to process.

Definition

Understanding tokenization matters because API costs and length limits are measured in tokens, not words. Non-English text, emoji and code can consume far more tokens than expected, making a request costlier or truncating it.

Related terms

• Token
• Context Window
• Embedding
• Perplexity (metric)
• Context caching

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.