What is Ensemble Learning?

Ensemble Learning is a subfield of Machine Learning in which multiple models (called weak learners) are trained to solve the same problem and combined to produce better results. The logic in ensemble learning follows the phenomenon of "the wisdom of crowds", which describes that the aggregate decision of a group of individuals is superior to the decision of a single individual. Derived from this phenomenon is the basic idea that combining multiple weak learners produces more accurate and robust models. Weak learners are models that usually perform poorly on their own due to high bias or high variance. In ensemble learning, they form the building blocks for developing more complex models.

Ensemble Learning provides a way to overcome the typical challenges of training a single model (e.g., high variance, low accuracy, or bias).

There are different ensemble learning methods. Among the best known are bagging, boosting and stacking.

Bootstrap aggregation (bagging) combines different homogeneous weak learners with high variance in prediction. In this process, different weak learners are trained in parallel with a different part of the training dataset. The predictions of each weak learner are then combined by taking an average or by majority voting.

In boosting, different homogeneous weak learners are trained sequentially (i.e., one after the other). In this process, the models are adapted step by step, so that the state of a model in a given step depends on the models in the previous steps.

Stacking often combines different heterogeneous weak learners. A so-called meta-model is then trained based on the predictions of the individual weak learners.

In simplified terms, bagging aims to obtain a model with a lower variance than its individual components (weak learners), while boosting and stacking aim to produce models that are less biased than their components (weak learners).

In summary, Ensemble Learning is a promising extension of classical Machine Learning methods with a lot of potential.

Source (translated): Towards Data Science