Random Forest

A forest of imperfect decision trees that together make surprisingly few mistakes.

Key Points

• It's bagging applied specifically to decision trees, plus random feature selection at each split.
• Each tree sees a bootstrapped (random-with-replacement) sample of the training data.
• Each split only considers a random subset of features — this decorrelates the trees.
• Final prediction = majority vote (classification) or average (regression) across all trees.
• Much less prone to overfitting than a single deep decision tree.

What it is

Random Forest builds many decision trees, each trained on a random sample of the data and a random subset of features, then averages their predictions (regression) or takes a majority vote (classification). This randomness makes each tree different, so their combined errors cancel out.

Why it exists

A single decision tree grown deep enough will overfit — it memorizes the training data. Random Forest was built to keep the trees' flexibility while cancelling out their individual overfitting through averaging many decorrelated trees.

Intuition

How it works

1. Bootstrap sampling

   For each tree, randomly sample the training data with replacement to create a slightly different dataset.

2. Random feature subset

   At every split in every tree, only consider a random subset of features rather than all of them.

3. Grow trees independently

   Grow each tree fully (or to a max depth) without pruning — individual trees are allowed to overfit.

4. Aggregate predictions

   Combine all trees' outputs by majority vote (classification) or averaging (regression).

When to use

• Use as a strong, low-effort baseline for tabular classification or regression problems.
• Use when you need feature importance estimates without much manual work.
• Use when you want a model that's robust to outliers and doesn't need feature scaling.

When not to use

• Do not use when you need a fully interpretable, single decision path (use a single decision tree instead).
• Do not expect it to extrapolate well on data far outside the training range — trees can only predict within seen value ranges.
• Do not use for very high-dimensional sparse data like raw text — linear models or boosting often do better there.

Common mistakes

• Setting n_estimators (number of trees) very high assuming it always helps — returns diminish and training slows down.
• Not tuning max_depth or min_samples_leaf, leading to needlessly slow, memory-heavy trees.
• Treating feature importance from Random Forest as causal — it only reflects predictive usefulness, not cause and effect.

In practice

One of the most common baseline models for structured/tabular data — used for churn prediction, credit scoring, fraud detection, and as a quick feature-importance check before building anything fancier.

Related concepts