AI Dictionary › Fondamenti AI

Regularization

Regolarizzazione

Regularization is a set of techniques that discourage a machine learning model from becoming too complex, by adding an explicit cost for complexity during training. The goal is no longer just to minimize error on the training data, but to find a balance between fitting known data well and staying simple enough to generalize to new data, directly countering the risk of overfitting.

Definition

How it works

In linear models the most common techniques are L1 and L2 regularization: both add a term to the error function proportional to the size of the model's coefficients, penalizing overly large weights. L2, also called ridge, shrinks coefficients gradually without ever setting them exactly to zero; L1, called lasso, tends instead to zero out the least useful coefficients entirely, implicitly performing a selection of the most relevant features. In neural networks there are different forms of regularization, such as dropout, which randomly deactivates a portion of neurons during training to prevent the model from relying too heavily on individual pathways.

Applications

It is a practically ubiquitous technique in applied machine learning: from regression models used in finance and insurance to large neural networks for computer vision and language, regularization is almost always present in one form or another. The degree of regularization applied is normally a hyperparameter to be chosen through cross-validation, seeking the point where the model is flexible enough to learn, but not so much that it learns by heart.

History & etymology

The term derives from the Latin regula, rule, and literally describes the idea of imposing an additional rule that keeps the model's freedom in check. Regularization techniques originate in applied mathematics and twentieth-century statistics, particularly in the ill-posed inverse problems studied by the Soviet mathematician Andrey Tikhonov in the 1940s and 50s, whose formulation, known today as Tikhonov regularization, underlies the modern ridge regression used in machine learning.

Related terms

• Overfitting
• Bias-variance tradeoff
• Feature selection
• Hyperparameter tuning
• Linear regression

More in Fondamenti AI

• LLM (Large Language Model)
• Token
• Context Window
• Temperature
• AI Hallucination
• Fine-tuning
• RAG (Retrieval Augmented Generation)
• Embedding

Put it into practice

• Real scenario: apply this concept with an instant AI verdict.
• Daily gym: short drills to train dictionary concepts.
• Playoff game: challenge other players, elimination-style.

📱 Download the Android app (beta) iOS coming soon

Say what you mean. Get what you need.

Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.