AI Dictionary › Fondamenti AI

Learning Rate Warm-up

Warm-up (del learning rate)

Warm-up is a training technique in which the learning rate starts at a very low value and is gradually increased in the early stages of training, before following its normal schedule. It prevents the model from receiving overly abrupt updates while its weights are still random and unstable. It is a common practice when training deep neural networks and large language models.

Definition

How it works

At the start of training, computed gradients can be very noisy, since the model has not yet learned anything. A high learning rate at this stage risks causing unstable updates or divergence. Warm-up increases the learning rate linearly or gradually over a set number of initial steps, giving the model time to stabilize before proceeding at full speed.

Applications

It is widely used when training transformers and other deep architectures, often combined with a later learning-rate decay phase. It is one of the hyperparameters configured in the training schedule, along with the number of warm-up steps and the maximum learning rate value.

History & etymology

It is an established engineering practice in deep neural network training, becoming particularly relevant as model sizes grew and adaptive optimizers became widely adopted.

Related terms

• Learning Rate
• Adam Optimizer
• Gradient Clipping
• Hyperparameter
• Gradient Descent

More in Fondamenti AI

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.