Random Forests — Tutorial

Wisdom of the crowd — bagging

A single decision tree is unstable: small changes in training data produce completely different trees. Random Forest fixes this by training many trees on bootstrapped subsets of the training data (sampling with replacement) and averaging their predictions. This is called bagging (bootstrap aggregating).

By averaging 100+ trees, the variance of the ensemble is much lower than any single tree, while bias remains low — the core tradeoff that makes random forests strong out-of-the-box performers.

Feature randomness — the forest part

The 'random' in Random Forest: at each split, only a random subset of features is considered (typically √n_features for classifiers, n_features/3 for regressors). This decorrelates the trees — if one strong feature dominates every tree, the trees will be correlated and averaging won't reduce variance.

With random feature subsets, each tree is a different expert. Their errors are less correlated, so averaging genuinely reduces variance.

Feature importance and out-of-bag evaluation

Random forests give two useful extras: (1) Feature importances: each tree's impurity-based importance is averaged across the forest. More reliable than a single tree's importances. Use permutation importance for a more rigorous estimate.

(2) Out-of-bag (OOB) score: each tree was trained without the ~37% of samples not in its bootstrap sample. Those samples can be used as a free validation set. Set oob_score=True to get an estimate of generalization error without a separate validation split.

Continue your research

Use these connected ModelRefs sections to compare alternatives, inspect implementation paths, and review the evidence and governance boundaries relevant to Random Forests — Tutorial.

• Browse models
• Review providers
• Inspect benchmarks
• Explore workflows
• Learn AI concepts
• Read decision guides
• Review trust and methodology
• Start a decision path