Theory & Mathematics

TL;DR

Random Forests build many decision trees, each trained on a slightly different subset of your data, then combine their predictions for a more robust and accurate result. This "wisdom of the crowd" approach helps overcome the weaknesses of individual trees, like overfitting. Key concepts involve bagging, feature randomness, and how they reduce variance while maintaining low bias.

1. The Mental Model

Imagine you're trying to predict something, and you ask a hundred different experts for their opinion. Instead of picking one expert, you combine all their answers. That's essentially what Random Forests do, but with decision trees as the "experts."

2. The Core Material

Random Forests are an ensemble learning method that leverage the power of multiple decision trees. The magic comes from two main techniques: bagging (Bootstrap Aggregating) and feature randomness.

How Bagging Works

Photo by Masud Allahverdizade on Pexels

When you train a single decision tree, it can easily overfit your training data, meaning it learns the noise and performs poorly on new, unseen data. Bagging combats this by training multiple trees on different subsets of the original training data.

Here's how it goes:

1. Bootstrap Sampling: For each tree in the forest, you create a new training dataset by randomly sampling with replacement from your original data. This means some data points might appear multiple times in a subset, and others might not appear at all. This creates slightly different datasets for each tree.
2. Independent Tree Training: Each tree is then grown independently on its unique bootstrap sample. These trees are typically grown to full depth, making them individually prone to overfitting.
3. Aggregation:
   • For classification tasks, the forest takes a majority vote from all trees. If 60 out of 100 trees predict "yes," the forest predicts "yes."
   • For regression tasks, the forest calculates the average of all tree predictions.

The Role of Feature Randomness

Photo by Zsófia Fehér on Pexels

Beyond bagging, Random Forests introduce another layer of randomness: feature subsampling. When each tree is being built, at each split point, it doesn't consider all available features. Instead, it randomly selects a subset of features to choose from.

This is crucial because:

• It further decorrelates the trees. If there's one very strong predictor in your dataset, every tree might pick it as the first split, making them all very similar. Feature randomness forces trees to explore other predictors, leading to a more diverse forest.
• This diversity is key to reducing the variance of the model without significantly increasing its bias. Each tree might be high-variance (overfitting), but their combined "average" or "vote" reduces this variance significantly.

graph TD
    A["Original Training Data"] --> B{Bootstrapped Sample 1};
    A --> C{Bootstrapped Sample 2};
    A --> D{Bootstrapped Sample N};

    B --> E["Train Decision Tree 1 (with feature randomness)"];
    C --> F["Train Decision Tree 2 (with feature randomness)"];
    D --> G["Train Decision Tree N (with feature randomness)"];

    E --> H["Prediction 1"];
    F --> I["Prediction 2"];
    G --> J["Prediction N"];

    H & I & J --> K{"Aggregate Predictions"};
    K --> L["Final Random Forest Prediction"];

Bias-Variance Trade-off

Photo by Stefan on Pexels

• Individual Decision Trees: High variance (prone to overfitting), low bias (can capture complex relationships).
• Random Forests: By averaging many high-variance, low-bias trees, Random Forests achieve low variance and low bias, making them a powerful and robust model. The randomness ensures that errors from individual trees don't all point in the same direction, effectively cancelling each other out when aggregated.

Out-of-Bag (OOB) Score

Photo by Erik Mclean on Pexels

Since each tree is trained on a bootstrap sample, roughly one-third of the original data points are not included in that tree's training set. These are called out-of-bag (OOB) samples. You can use these OOB samples to evaluate the performance of each tree without the need for a separate validation set. The OOB score is essentially an internal, unbiased estimate of the model's generalization error.

3. Worked Example

Let's say we have a dataset of 10 samples (A-J) and 3 features (X, Y, Z) to predict whether a customer will buy a product (Yes/No).

1. Original Data: [A, B, C, D, E, F, G, H, I, J]

2. Tree 1 Training:

   • Bootstrap Sample: Let's say it randomly samples [A, A, C, D, F, G, H, I, I, J]. (Note: B, E are OOB for Tree 1).
   • Feature Randomness: When building Tree 1, at each split, it might only consider 2 out of the 3 features (e.g., X and Z for the first split).
   • Result: Tree 1 predicts "Yes" for a new customer.

3. Tree 2 Training:

   • Bootstrap Sample: Let's say it samples [B, C, C, D, E, E, G, H, J, J]. (Note: A, F, I are OOB for Tree 2).
   • Feature Randomness: At each split, it might consider 2 out of the 3 features (e.g., Y and Z).
   • Result: Tree 2 predicts "No" for the same new customer.

4. ...and so on for many trees (e.g., 100 trees).

5. Aggregation (Classification):

   • Suppose 70 trees predict "Yes" and 30 trees predict "No".
   • The Random Forest's final prediction for the new customer is "Yes" (majority vote).

4. Key Takeaways

• Random Forests are an ensemble method that combines many decision trees to make predictions.
• They use bagging (bootstrap sampling) to create diverse training sets for each tree.
• Feature randomness at each split point further decorrelates the individual trees, enhancing diversity.
• This combination effectively reduces variance without significantly increasing bias, leading to robust models.
• Random Forests inherently handle high-dimensional data and are less prone to overfitting than single decision trees.
• The Out-of-Bag (OOB) score provides an internal, unbiased estimate of model performance.

5. Now Try It

Think about a classification problem you're familiar with (e.g., predicting customer churn). Imagine you're building a Random Forest to solve it. List three ways the inherent randomness of the Random Forest algorithm (both bagging and feature randomness) helps it make a better prediction than a single, fully-grown decision tree. What success looks like: You've clearly articulated how each type of randomness contributes to a more robust and accurate prediction in the context of your chosen problem.