AI Dictionary › Fondamenti AI

Tokenization

Tokenizzazione

Tokenization is the process of splitting text into minimal units called tokens that a model can process numerically. A token isn't always a whole word: it can be a word, a syllable, a single character, or a fragment like a suffix. Picture cutting a necklace into beads: each bead is a token, and the model counts and orders beads rather than reading the continuous thread. Methods like Byte-Pair Encoding build a vocabulary of recurring fragments, so even rare or invented words are composed from known pieces. Each token becomes an ID number, the bridge between human language and computation.

Definition

Understanding tokenization is practical: API costs and context limits are measured in tokens, not words. Languages with long compound words consume more tokens, and text heavy with symbols or code can fragment unexpectedly, affecting both price and performance.

Related terms

• Token
• Context Window
• Embedding
• Softmax

More in Fondamenti AI

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.