AI Dictionary › AI Fundamentals

Mixed Precision Training

Addestramento a precisione mista

Mixed precision training is a technique that uses reduced-precision numerical formats for part of the computations during training, while keeping higher precision where more stability is needed. The goal is to speed up training and reduce the memory required, without significantly compromising the quality of the final model. It is now a standard practice when training large-scale models.

Definition

How it works

Instead of using 32-bit numbers for every calculation, most operations are carried out with 16-bit numbers, which are faster to process and lighter to store. Some critical steps, such as gradient accumulation, are kept at higher precision to avoid numerical errors that would harm training convergence.

Applications

It is widely used when training large language models and deep networks on specialized hardware such as GPUs and TPUs, which are optimized to run reduced-precision calculations much faster. It allows larger models to be trained with the same available hardware resources.

History & etymology

It became common practice as hardware capable of efficiently running reduced-precision calculations spread, helping make the training of today's large-scale models more sustainable.

Related terms

• GPU and TPU
• Compute
• Quantization
• FLOPS
• Gradient Clipping

More in AI Fundamentals

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.