AdaBoost

Each new model in the chain pays extra attention to whatever the last one got wrong.

Key Points

• Trains weak learners sequentially, not in parallel like bagging.
• Misclassified points get higher weight before the next learner is trained — the model 'adapts' to its own mistakes.
• Each learner's vote is weighted by how accurate it was — better learners matter more in the final decision.
• Base learner is typically a decision stump (a tree with just one split).
• Sensitive to noisy data and outliers because it keeps upweighting hard-to-classify points.

What it is

AdaBoost (Adaptive Boosting) builds a sequence of weak learners — usually shallow decision trees called stumps — where each new learner is trained to focus more on the examples the previous learners misclassified. The final prediction is a weighted vote of all learners, with more accurate learners given more say.

Why it exists

A single weak learner (like a one-split decision stump) barely beats random guessing on its own. AdaBoost exists to prove — and exploit — the idea that a sequence of weak learners, each nudged to fix the last one's blind spots, can be combined into a strong overall classifier.

Intuition

How it works

1. Initialize equal weights

   Every training example starts with the same weight/importance.

2. Train a weak learner

   Fit a simple model (e.g. a decision stump) on the weighted data.

3. Score and reweight

   Increase the weight of misclassified points so the next learner focuses on them; decrease weight of correctly classified points.

4. Repeat

   Train the next weak learner on the updated weights, repeating for a set number of rounds.

5. Combine with weighted vote

   Final prediction is a weighted vote across all learners, where more accurate learners get more influence.

When to use

• Use on relatively clean datasets where you want a strong classifier built from very simple base learners.
• Use when you want a boosting model that's simpler and faster to reason about than Gradient Boosting.

When not to use

• Do not use on data with significant label noise or many outliers — AdaBoost will keep upweighting them and can overfit badly.
• Do not expect it to outperform modern gradient boosting libraries (XGBoost, LightGBM) on large tabular datasets.

Common mistakes

• Using deep trees as the base learner instead of shallow stumps — defeats the purpose and increases overfitting risk.
• Not realizing outliers get disproportionate weight over successive rounds, distorting the final model.

In practice

Largely superseded in practice by Gradient Boosting and XGBoost, but still taught as the conceptual foundation of boosting and occasionally used for face detection-style problems with simple, fast base learners.

Related concepts