AI Dictionary › Fondamenti AI

Momentum (in optimization)

Momentum (nell'ottimizzazione)

Momentum is a technique used in optimization algorithms to accelerate and stabilize gradient descent, taking previous updates into account in addition to the current gradient. The idea is inspired by physics: just as a moving object tends to keep its direction, weight updates accumulate a kind of inertia that makes them less sensitive to local oscillations in the loss function.

Definition

How it works

Instead of updating weights using only the gradient computed at the current step, momentum keeps a moving average of past gradients and uses it to determine the direction of the update. This allows faster movement through flat regions of the loss function and reduces oscillations in directions with high curvature.

Applications

It is a core component of many optimizers used in training deep neural networks, including more sophisticated variants that combine momentum with per-parameter learning-rate adaptation. It helps make training faster and more stable, especially on complex loss surfaces.

History & etymology

It is a concept borrowed from classical physics and applied to numerical optimization, now one of the standard building blocks of neural network training algorithms.

Related terms

• Gradient Descent
• Adam Optimizer
• Learning Rate
• Gradient Clipping
• Loss Function

More in Fondamenti AI

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.