Random forest

Overview

A random forest averages many trees, and each tree is grown on a resampled copy of the training data 2208.04112 §2.1. That design assumes each row is an independent observation. With repeated measures, rows from one subject are correlated. Then observation-level bootstrap samples contain rows from nearly every subject, the trees become similar to each other, and the out-of-bag (OOB) error looks better than it is, because OOB rows resemble in-bag rows from the same subject 2208.04112 §2.1 2208.04112 fig. 1.

Every variant below changes one of three things: how the training sample is drawn, what the trees see as input (history summaries or random effects), or what the split criterion optimises. The standard forest needs none of these changes for independent rows, and its sensitivity to feature scaling is low 2212.12343 §5.1.

Thirteen cards on wikillm list this concept. The first is a review of longitudinal extensions, which runs no experiments of its own, so its variants are reported from the cited work and not measured here 2208.04112 §5. The second is an empirical study of feature scaling, in which a random forest is one of 20 classifiers, tested on binary tabular datasets 2212.12343 §4.1. The third uses forests as regressors for a single image-quality score, with features that are not repeated measures 1612.05890 §4.4. The fourth is a survey of text classification that lists forests among its classifiers and reports no measured results for them 1904.08067 §4.8. The fifth is an email spam and phishing classifier that compares a forest with linear models on text features 2405.11619 tab. 2. The sixth is a card-fraud comparison of seven standard classifiers, one of them a forest, on transaction features 2608.07471 fig. 4. The seventh applies forests to ModSecurity rule-firing features for SQL injection 2406.13547 §3. The eighth is a predictive-maintenance study on rare machine failures, where a forest is the classifier and also the objective that tunes the data generators 2607.19153 §3.3.2 tab. 4. The ninth is a discrete-choice paper, in which a forest is one of several benchmarks for a neural choice model 2207.12877 tab. 1. The tenth is a classifier comparison on spiral drawings for Parkinson's disease, where a forest is one of nine default classifiers 2609.07408 p. 11. The eleventh is a systematic review of machine learning for disease diagnosis. It names forests among the methods of the studies it reviews, but it trains no models and gives no forest results of its own 2112.15538 §6. The twelfth is a review of machine learning for mineral mapping from remote sensing. It describes forests as bagged trees that vote or average, and lists them among the classifiers used in cited studies, without results of its own 2103.07678 §5.2.4 tab. 2. The thirteenth is a dataset paper on burned-area mapping from satellite time series, in which a forest is the only classifier and is trained on one row per grid cell and year 2609.23687 §2.2.15 §2.2.16.

Variants

Standard forest (baseline)

Each tree is grown on a bootstrap sample of rows, and the trees are averaged. This is the reference that the variants below modify. On 82 binary tabular datasets with five-fold cross-validation, a forest's scores barely change when the input features are rescaled. Friedman tests do not reject "no difference" across the six scaling variants (all datasets: F1 p = 0.5169, G-Mean p = 0.4980) 2212.12343 tab. 4. The paper reports decision trees and their ensembles as nearly insensitive to the scaler 2212.12343 §5.1 2212.12343 §5.3. The paper configures the forest with 100 trees and the Gini criterion 2212.12343 tab. 1. Its scores are for classification only, and repeated measures are not tested 2212.12343 §4.1.

Subject-level resampling (RF++ and the two-stage bootstrap)

For clustered data without a time effect, the review sorts the fixes into three groups. First is averaging, which collapses each subject's rows into one before growing a standard forest. It loses within-subject variation and was reported to degrade more than standard RF as the number of subjects falls 2208.04112 §2.2.1. Second is the subject-level bootstrap, RF++: resample subjects, and keep every row of each drawn subject in-bag 2208.04112 §2.2.2 2208.04112 fig. 2. Third is the two-stage bootstrap of Adler et al.: resample subjects, then keep one randomly chosen row per drawn subject. In the simulations the review summarises, this scheme predicted best against standard RF, averaging and RF++, but the review notes that results depend on the setting 2208.04112 §2.2.2.

Subject-level resampling only adjusts for clustering. There are no random effects and the time order of observations is not used 2208.04112 §2.2.2.

Historical summaries for time-varying predictors

For a predictor measured over time, the historical RF first reduces each subject's past to a summary. One example counts past values that fall below a threshold inside a look-back window 2208.04112 eq. 2. Other summaries are a relative frequency above a threshold and a mean, which the review says give similar accuracy in a personal communication 2208.04112 §2.3.1. Only the frequency-based summaries keep a forest invariant to monotone transforms of the predictor 2208.04112 §2.3.1.

The window length and threshold are searched inside each split, which is costly when there are many time-varying predictors. The htree authors search on a random sub-sample instead 2208.04112 §2.3.1. The search grid and sub-sample size are not stated. Variable importance is computed by deleting each variable and refitting, so it can hide variables that are correlated with others 2208.04112 §2.3.1.

Mixed-effects hybrids

The hybrids replace the fixed part of a linear mixed model with a tree or a forest, and keep subject-specific random effects. The model is y_i = f(X_i) + Z_i b_i + \epsilon_i with normal random effects and errors 2208.04112 eq. 3. MERT and MERF fit it by alternating a tree or forest on the adjusted outcome with a mixed-model fit of the random effects 2208.04112 §2.3.2. Observation-level resampling is kept in MERF, which is valid only if the random effects absorb all within-subject correlation 2208.04112 §2.3.2.

RE-EM trees use the tree only to define leaves, then fit a mixed model with a leaf indicator as fixed effect. Unlike MERT they allow a non-diagonal error covariance, for example autocorrelated errors 2208.04112 §2.3.2. REEMforest averages several such trees. The stochastic variants add a subject-specific Gaussian process 2208.04112 §2.3.2.

For a binary outcome, GMERT, GMERF and the Bayesian BiMM forest extend the same idea through a generalized mixed model 2208.04112 §2.3.2 2208.04112 §2.3.3. GMET and GMERF initialise the linear predictor with a plain GLM on all predictors, so they cannot run when there are more predictors than observations 2208.04112 §2.3.2 2208.04112 §5.

Prediction differs by case. For a new subject, use only the fixed part. For a training subject, add its estimated random effect 2208.04112 §2.3.2.

Multivariate split criteria

When a subject's repeated outcomes are treated as one vector, the node impurity changes. Segal's generalized sum of squares uses a node covariance 2208.04112 eq. 4. Its covariance is estimated in the parent and reused for both children, and all subjects must share observation times 2208.04112 §3.1.1. The independence case reduces to a multivariate sum of squares used in mvpart and in multivariate forests 2208.04112 §3.1.1. Distance-based criteria (db-MRT and related) need only a distance, so responses can differ in length or timing. Prediction within a leaf then needs care, because a plain average may not exist 2208.04112 §3.1.1. Likelihood-based criteria exist for multiple binary responses and for multivariate normal responses 2208.04112 §3.1.2.

These criteria apply directly only to time-invariant predictors. Time-varying predictors need one of the history or mixed-effects techniques as well 2208.04112 §3.

Regression forest with a log-determinant split objective

This variant is not a repeated-measures fix. The forests predict one quality score per image from hand-built features, and the training targets are human ratings 1612.05890 §3. The features are split into three groups, one forest per group, and a linear layer combines the three outputs. The weights are fitted by least squares 1612.05890 §4.4 eq. 7. Each forest prediction is the mean over its trees eq. 6. Node splits maximise a log-determinant objective built on a conditional covariance from probabilistic linear fitting eq. 4 eq. 5. The authors used T = 2000 trees per forest, chosen by sweeping 100 to 5000 1612.05890 §5.1. Depth, leaf size and the feature-subset size are not stated 1612.05890 §4.4. Author-reported Spearman correlation with human scores is 0.931 under 5-fold cross-validation, against 0.921 for one forest on concatenated features 1612.05890 tab. 7 tab. 6. No code was checked for this variant 1612.05890 §5.

The paper does not say whether the forest outputs used to fit the linear layer are out-of-fold 1612.05890 eq. 7. In-sample outputs would inflate that fit, so use out-of-fold predictions when reimplementing.

Applied uses of the standard forest

Seven of the cards use an unmodified forest as one classifier or baseline in a comparison, and three surveys name forests without measuring them. The cards do not record a repeated-measures design for any of these studies, and each reported number comes from one author-run evaluation: a single split, one cross-validation scheme, or repeated random draws from one dataset.

On email spam and phishing, the forest has 100 trees, each grown on a random subset of features at every split, with a majority vote 2405.11619 p. 9. On a merged corpus of 82,486 emails, with one 80/20 split, the forest reaches 0.984 accuracy on TF-IDF features. A linear SVC reaches 0.991 on the same split 2405.11619 tab. 2. With Word2Vec features the order reverses: the forest scores 0.838 and the linear SVC 0.821 2405.11619 tab. 2. Tree depth, features per split and bootstrap settings are not stated 2405.11619 p. 9. Other forest rows in the same table use different subsets of the data and are not comparable with the proposed model 2405.11619 tab. 2.

On card fraud, the forest is one of seven candidates. The training split is randomly undersampled to equal class counts, and Time and Amount are standardized 2608.07471 §3.4. The forest's author-reported ROC AUC is 0.911, against 0.946 for logistic regression, the best single model, and 0.954 for a stacked ensemble 2608.07471 fig. 4 §4.2 §5. The paper uses one Kaggle dataset, reports no run-to-run variance, and does not state the forest's trees, depth or features per split 2608.07471 §3.1 §4.1. The paper also does not say whether model selection used the validation or the test split 2608.07471 fig. 3 §4.2, so the AUC values may be optimistic.

On web-attack detection, each feature is a binary flag for one ModSecurity SQL-injection rule that fired on a query 2406.13547 §3. The forest is the top author-reported model at paranoia levels 3 and 4, with TPR at 1% FPR of 99.45, against 99.35 for L1 logistic regression at level 4 2406.13547 tab. 1. That gap is 0.1 points on one balanced test set of 10,000 samples 2406.13547 §4.1. The 1% FPR is measured on that balanced mix, so it is not a direct estimate of false positives in production 2406.13547 §4.1. The paper says a non-linear model is harder to put into ModSecurity than linear weights 2406.13547 §3. A forest has no penalty term, so it cannot set rule weights to zero the way the L1 models do 2406.13547 §3 2406.13547 §4.4.

On rare machine failures, the forest is the classifier in every scenario. The baseline gives it class weights balanced for the imbalance, with no resampling 2607.19153 §3.2.1. Its author-reported PR-AUC is 0.835 with precision 1.0 and recall 0.588 2607.19153 tab. 4. Random undersampling and SMOTE lower its PR-AUC to 0.724 and 0.7525 2607.19153 tab. 4. The best row adds synthetic failures from four WGAN-GP generators, one per failure type, and reaches PR-AUC 0.857 and recall 0.882 at precision 0.570 2607.19153 tab. 4 §3.2.5. The forest's hyperparameters are searched on a shallow grid that the paper does not list, and a deeper search is said to have crashed the system 2607.19153 §5.2. The GAN objective is the PR-AUC of a forest on the validation split, so the same classifier that is compared across scenarios also sets the GAN hyperparameters 2607.19153 §3.3.2. The paper runs one seed per configuration and does not test whether the 0.022 PR-AUC gap over the baseline is larger than run-to-run noise 2607.19153 §5.1. Compute is also large: about 14 hours for the multi-GAN scenario 2607.19153 §5.1.

In the discrete-choice paper the forest is a benchmark with a grid of 50 to 400 trees and depths of 5 to 20, chosen on validation loss 2207.12877 §6.2. On Swissmetro its log-likelihood loss is the lowest in the table, 0.523, and its accuracy is 0.776, below the 0.793 of the proposed network 2207.12877 tab. 1. On Expedia its loss is 2.712, worse than the 2.482 of multinomial logit 2207.12877 tab. 1. The authors tie the forest's overfitting on Expedia to its larger input dimension 2207.12877 §6.3 §6.4. The paper also says training the proposed network takes about an order of magnitude longer than a forest, and that it is unclear whether this is inherent 2207.12877 app. E. All of these numbers are author-reported on one 10-fold scheme.

On spiral drawings for Parkinson's disease, the forest is one of nine classifiers with library default settings and no tuned values 2609.07408 p. 11. Its inputs are eleven statistics per image 2609.07408 p. 9. The paper reports no held-out test set, and every score comes from stratified 10-fold cross-validation on 72 training images, repeated ten times 2609.07408 p. 8 p. 11. The forest's AUC is 0.90 on the plain drawings and 0.97 on the augmented set, and the edge-preprocessed sets give 0.85 and 0.89 2609.07408 fig. 12 fig. 13. The card does not record how augmented copies of one image are assigned to folds, so the 0.97 may be optimistic. Check that before relying on it.

On burned-area mapping, the forest is the only classifier. Each row is one 500 m grid cell in one year, built from daily values of the four BGR-NIR bands 2609.23687 §2.2.15. The grid search covers 100, 200 and 300 trees and maximum depths of 3, 5 and 7 2609.23687 §2.2.16. Each of the three relabelling approaches is repeated 200 times with a new random draw, and 30% of the samples are held out 2609.23687 §2.2.16 §2.2.17. The author-reported mean test F1 for the binary labels is 0.92 to 0.95 2609.23687 tab. 7. Non-burned cells are 86% to 88% of the grid 2609.23687 §3.3, so overall IoU stays high even when burned-area IoU is low. In the best validation setup, burned-area IoU is 0.49 to 0.71 2609.23687 tab. 8. The grid is spatially adjacent and the paper does not discuss spatial autocorrelation, so a random cell split may be optimistic 2609.23687 §2.2.9.

The text survey names forests as one of its classifiers and describes them in its section on classifiers, with no measured results 1904.08067 §4.8. Use it for the vocabulary, not for a ranking. The mineral-mapping review does the same for remote sensing. It describes the forest as suited to large, complicated datasets without a parametric form 2103.07678 §5.2.4 and reports no forest result. Its two numeric findings come from other studies and do not involve forests 2103.07678 §5.

The disease-diagnosis review lists a forest alongside Gaussian and Bernoulli naive Bayes for coronary heart disease, citing one study 2112.15538 §6. It gives no accuracy or other metric for that forest, and it reports the reviewed accuracies only as the authors' own 2112.15538 §7. Use it as a pointer to the primary studies, not as evidence for the forest's performance.

When to use which

Implementation sketch

The shared core is a per-tree training sample. Clustered data needs a subject-level draw, and the forest uses each drawn subject's rows as the review describes 2208.04112 §2.2.2 2208.04112 fig. 2.

# Per-tree training sample for clustered data  [arxiv:2208.04112 §2.2.2]
def tree_sample(rows_by_subject, two_stage):
    subjects = list(rows_by_subject.keys())
    # Draw subjects with replacement, not rows  [arxiv:2208.04112 §2.2.2]
    drawn = sample_with_replacement(subjects, size=len(subjects))
    in_bag = []
    for subject in drawn:
        rows = rows_by_subject[subject]
        if two_stage:
            # Adler et al.: one random row per drawn subject  [§2.2.2]
            in_bag.append(random_choice(rows))
        else:
            # RF++: every row of the drawn subject is in-bag  [§2.2.2]
            in_bag.extend(rows)
    # Whole subjects that were never drawn are out-of-bag  [§2.2.2]
    out_of_bag = [
        row for subject in subjects if subject not in drawn
        for row in rows_by_subject[subject]
    ]
    return in_bag, out_of_bag

The review does not spell out the OOB rule above; it follows from subject-level resampling 2208.04112 §2.2.2. Per-subject OOB is the estimate to use 2208.04112 §2.2.2.

For independent rows the sample is a plain row bootstrap, and no scaling step is needed for the forest itself 2212.12343 §5.1. Scalers are fitted on the training part of each fold only, as the scaling study does 2212.12343 §4.2.

Contradictions

The spam paper ranks a linear SVC above the forest on TF-IDF features, while the ModSecurity paper ranks the forest above the linear models at paranoia levels 3 and 4 2405.11619 tab. 2 2406.13547 tab. 1. The fraud paper ranks logistic regression above the forest on AUC 2608.07471 fig. 4. The tasks, features and test sets differ, so these orderings are reported side by side and not resolved. The spam paper also ranks the forest above the SVC on Word2Vec features 2405.11619 tab. 2.

The disease-diagnosis review's text and its tables credit the heart-disease methods to different studies. The text credits Sun et al. with a forest among the heart-disease methods 2112.15538 §6, while its table credits Bemando et al. 2112.15538 tab. 3. The card does not resolve which attribution is right, so this page does not use either one.

In the discrete-choice paper the forest has the lowest Swissmetro loss, while the proposed network has the higher accuracy on the same split. The two metrics point to different winners, and the paper does not settle which one matters 2207.12877 tab. 1. On Expedia the same table ranks the forest below multinomial logit on loss, so the forest's rank changes with the dataset.

The predictive-maintenance paper's text and table disagree on the forest's F1. Its §4.1 calls the multi-GAN forest F1 (0.6923) the highest across conditions, while the baseline forest F1 in tab. 4 is 0.740, and the same section later says the multi-GAN F1 is slightly lower 2607.19153 §4.1 tab. 4. Its text and figures also differ on the RUS forest precision (0.2156 against 0.216) 2607.19153 §4.1 fig. 10. Use the table values 2607.19153 tab. 4. The paper's two rows labelled "Wide RF grid" repeat values close to the SVM rows, which looks like a copy error, so they are not used here 2607.19153 tab. 4.

Within the longitudinal review, the cited simulations rank the two-stage bootstrap first, but the review says results depend on the setting 2208.04112 §2.2.2. The scaling study and the review do not overlap 2212.12343 §4.1.

Papers

• Learning a No-Reference Quality Metric for Single-Image Super-Resolution: uses three random forests, one per feature group, whose outputs are linearly combined to score super-resolved images.
• A review of machine learning in processing remote sensing data for mineral exploration: surveys ML for mineral mapping from satellite and airborne data, describes random forests among its classifiers, and runs no experiments.
• A multi-temporal dataset for mapping burned areas in the Brazilian Cerrado using time series of remote sensing imagery: builds a labelled WFI cell dataset and uses a random forest as its only classifier; reports binary F1 of 0.92 to 0.95 and burned-area IoU of 0.49 to 0.71 in the best setup.
• Text Classification Algorithms: A Survey: maps the text-classification pipeline and lists random forests among its classifiers, with no experiments.
• Machine learning based disease diagnosis: A comprehensive review: a systematic review of ML for diagnosis that names forests among the reviewed methods, with no experiments of its own.
• Longitudinal random-forest review: maps CART-based forest extensions for repeated measures, with equations and a software table, and runs no experiments.
• The choice of scaling technique matters for classification performance: shows that a random forest is nearly insensitive to the choice of feature scaler across 82 binary datasets, while scale-sensitive classifiers react strongly.
• Novel Interpretable and Robust Web-based AI Platform for Phishing Email Detection: compares a 100-tree forest with linear SVC and Naive Bayes on TF-IDF and Word2Vec email features, and reports the linear SVC as best.
• Application of Artificial Intelligence for Fraudulent Banking Operations Recognition: compares seven standard classifiers, including a 0.911-AUC forest, on undersampled card transactions, and reports a stacked ensemble as best.
• ModSec-Learn: Boosting ModSecurity with Machine Learning: learns weights over ModSecurity SQL-injection rules with a linear SVM, logistic regression or a random forest, and reports the forest as best at paranoia levels 3 and 4.
• Breaking the Homogeneity Assumption: Specialized Multi-Generator Adversarial Learning for Rare Failure Detection in Predictive Maintenance: trains one WGAN-GP per failure type and pools the synthetic failures; a random forest is the classifier and reports PR-AUC 0.857 against 0.835 for the cost-sensitive baseline, on one run.
• Representing Random Utility Choice Models with Neural Networks: benchmarks a tuned random forest against its neural choice model; the forest has the lowest Swissmetro loss, and the proposed model has the higher accuracy on both datasets.
• Impact of canny edge detection preprocessing on performance of machine learning models for Parkinson's disease classification: trains nine default classifiers, including a forest, on spiral drawings with and without edge preprocessing and augmentation; the forest's AUC falls after edge preprocessing and reaches 0.97 on the augmented plain set.