AI Dictionary › Fondamenti AI

Knowledge Distillation

Distillazione

Knowledge distillation is the process where a large, capable model, the teacher, trains a smaller model, the student, to imitate its behavior. Instead of learning only from correct labels, the student learns from the teacher's soft probabilities, which carry rich information about how plausible an answer is versus the alternatives. It's like a master craftsman who doesn't just say 'right or wrong' but explains the nuances of reasoning, transferring intuition hard to grasp from raw examples alone. The result is a compact model that retains much of the original's performance.

Definition

Distillation is crucial for bringing the intelligence of huge models into real applications with cost and speed constraints. Many fast, cheap production models are distilled versions of larger ones, answering nearly as well while using a fraction of the resources.

Related terms

• Quantization
• Foundation Model
• Inference
• Transfer Learning
• Fine-tuning

More in Fondamenti AI

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.