AI Dictionary › AI Fundamentals
Ensemble learning is the strategy of combining the predictions of multiple machine learning models to achieve an overall result more accurate and stable than any single model would achieve alone. It rests on an intuition as simple as it is powerful: different models often make different mistakes, and combining them appropriately tends to make individual errors cancel out.
There are several strategies for building an ensemble. Bagging trains many identical models on different random samples of the same data and averages their results, as random forest does with its decision trees. Boosting trains models in sequence, each focused on correcting the mistakes of the previous ones, as in gradient boosting. Stacking combines even very different models, for example a decision tree, a logistic regression and a neural network, using a further model that learns how to best weigh each one's predictions. In every case, diversity among the combined models is what makes the ensemble effective: identical models that all make the same mistakes add no value by voting together.
Ensembles dominate data science competitions on tabular data and are widely used in production wherever reliable predictions are needed: credit scoring, demand forecasting, fraud detection, recommendation systems. The price to pay is lower interpretability compared to a single simple model, and higher computational cost both in training and in production, since every prediction requires consulting multiple models instead of just one.
The idea of combining multiple estimates or judgments to achieve a result better than any single one is ancient, rooted in collective decision theory and classical statistics. In machine learning, ensemble learning developed as a systematic discipline starting in the 1990s, with foundational work on bagging by Leo Breiman in 1996 and on boosting by Robert Schapire and Yoav Freund in the mid-decade, which formally demonstrated how combining weak models could produce much stronger predictions.
From our network
Kaimaki Web: Websites That Win Customers
Custom websites, web apps and digital marketing for growing businesses.
Visit kaimakiweb.com →From the Agora Intelligence blog
📱 Download the Android app (beta) iOS coming soon
Say what you mean. Get what you need.
Grace Certified, the AI coach that trains and certifies your prompt engineering, by Agora Intelligence.