AI Dictionary › AI Models
Gradient clipping is a technique that limits the maximum value gradients computed during neural network training can take, cutting them when they exceed a set threshold. It prevents excessively large parameter updates, caused by anomalous gradients, from destabilizing training and making the model diverge instead of improving.
During training, the gradient descent algorithm computes, for each parameter, how much and in which direction to adjust it to reduce error. In some situations, particularly in very deep or recurrent networks, these gradients can grow explosively, a phenomenon known as gradient explosion. Gradient clipping intervenes by rescaling or truncating gradients that exceed a certain maximum norm, before they are used to update the weights.
It is standard practice in training large language models and recurrent networks, where it helps keep training stable over long computation periods that can last weeks and involve an enormous number of update steps, reducing the risk of having to restart training from an earlier checkpoint due to a sudden divergence.
The name literally describes the operation: gradients are "clipped" when they exceed a threshold, a practical safeguard that has become increasingly relevant as trained neural networks have grown in depth and complexity.
From our network
INDACO TMS: Transport Management for European Logistics
Shipment tracking, multi-carrier EDI and automated invoicing in one cloud platform. Invoices generated in under 10 seconds.
Visit indacotms.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.