Longitudinal data analysis with tree ensembles

Overview

Longitudinal (repeated-measures) data has several rows per subject: a patient measured every few months, or several samples taken at one visit. Rows from the same subject are correlated, and visits can be irregularly spaced or missing. Standard random forests assume independent rows, so two things break: observation-level bootstrap samples contain rows from nearly every subject, which makes the trees similar, and out-of-bag error looks better than it should 2208.04112 §1, §2.1.

The invariant shared by every variant is that the method must account for the repeated rows, in one of three ways:

1. Resample by subject. Draw whole subjects into each tree, so the trees differ more and OOB rows are not twins of in-bag rows 2208.04112 §2.2.
2. Add random effects. Keep a per-subject offset in a mixed-effects model and let a tree or forest model the fixed part 2208.04112 §2.3.
3. Model the trajectory as one response. Treat a subject's outcomes as a vector or matrix and change the split criterion to compare whole trajectories 2208.04112 §3.

The review that this page is compiled from covers all three. For multivariate responses, time-varying predictors also need the time-effect methods 2208.04112 §3. It runs no experiments of its own, and it reports that the packages have never been benchmarked against each other 2208.04112 §5. Use the notes below as a map, then go to the primary papers before implementing.

Variants

Clustered rows, no time effect

Subject-level resampling only. The model is a subject mean plus independent noise 2208.04112 eq. 1, so the method corrects for clustering but uses neither time order nor random effects 2208.04112 §2.2.2.

• Averaging. Collapse each subject's rows to one row and run a standard forest. This discards within-subject variation, hides unequal numbers of rows per subject, and cannot average categorical values that differ between visits. The cited study reports that it degrades more than a standard forest as the number of subjects shrinks 2208.04112 §2.2.1.
• Subject-level bootstrap (RF++). For each tree, bootstrap subjects and keep all of their rows in-bag. A subject-level prediction is a majority vote over its rows, and the same vote gives a subject-level OOB error 2208.04112 §2.2.2. Card: Hu and Szymczak's review.
• Two-stage bootstrap (Adler et al.). Resample subjects, then keep one randomly chosen row per selected subject. In the simulations the review cites, this beat a standard forest, averaging and RF++, but the review notes that results depend on the setting 2208.04112 §2.2.2.

Time effect, univariate outcome

Historical random forest. Predict each observation from its concurrent predictors plus a summary of the same subject's earlier observations. Time-varying predictors are summarised over a look-back window. The example summary counts past values below a threshold; the review also lists frequency and mean summaries. Only the count and frequency summaries are invariant to monotone transforms 2208.04112 §2.3.1, eq. 2.

Each split searches the window and threshold along with the predictor. This is costly when there are many time-varying predictors, so the htree authors tune the cut-off on a random sub-sample 2208.04112 §2.3.1. Variable importance is available 2208.04112 tab. 3.

Mixed-effects hybrids, continuous outcome. The model is a tree or forest for the fixed part plus a subject-level random effect, fitted by an alternating loop 2208.04112 §2.3.2, eq. 3:

• MERT / MERF (Hajjem et al.) use a tree or forest for the fixed part and assume residuals with a common variance. They resample observations rather than subjects, which the review says is justified only when the random effects capture all within-subject correlation 2208.04112 §2.3.2.
• RE-EM tree / REEMforest. The tree only defines leaves. The leaf effects are then fitted in a linear mixed model, which allows a non-diagonal residual covariance such as autocorrelation 2208.04112 §2.3.2. Public R packages exist 2208.04112 tab. 3.
• Stochastic variants (SMERT, SMERF, SREEMtree, SREEMforest). Add a subject-specific Gaussian process to capture smooth within-subject deviations 2208.04112 §2.3.2.

Prediction differs by subject status. For a new subject, use only the fixed part. For a subject seen in training, add its estimated random effect 2208.04112 §2.3.2.

Mixed-effects hybrids, non-Gaussian outcome. Two routes for binary outcomes:

• GMERT / GMERF (generalized linear mixed model with a tree or forest in the fixed part) use pseudo-likelihood iterations that need starting values. The GMET / GMERF variant starts from a plain GLM on all predictors, so it cannot be used when there are more predictors than observations 2208.04112 §2.3.2.
• BiMM tree / forest (Bayesian mixed model for binary outcomes). The fixed part is a tree's leaf indicators, fitted by alternating a tree step with a Bayesian GLMM step until the posterior log-likelihood stabilises 2208.04112 §2.3.3.

Multivariate response: whole trajectories

These methods treat a subject's repeated outcomes as one object and change the impurity used to pick splits. They apply directly only to time-invariant predictors 2208.04112 §3.

• Distance-based impurity. Segal's criterion uses a generalized sum of squares with a node covariance, estimated once in the parent and shared by both children. It requires the same observation times for all subjects 2208.04112 §3.1.1, eq. 4. With an independence covariance it reduces to the multivariate sum of squares used in several existing packages 2208.04112 §3.1.1.
• db-MRT. Impurity is built from a distance matrix over pairs of observations. Responses may then differ in length or be irregularly timed, provided a distance is defined. Prediction inside a leaf needs care, because a plain average may not exist 2208.04112 §3.1.1.
• Likelihood-based impurity. For several binary responses, Zhang's exponential-family model uses the maximised log-likelihood in each node. For multivariate normal responses, Abdolell et al. use a deviance that equals a Mahalanobis distance 2208.04112 §3.1.2.
• Spline coefficients. Fit a spline to each subject's trajectory and regress the basis coefficients with a multivariate tree. This handles irregular times 2208.04112 §3.2.

When to use which

The review gives no benchmark ranking, so the choices below are drawn from the constraints each method's description and software support imply.

Implementation sketch

Language-agnostic pseudocode for the common core of the subject-level route. Only the sampling step is shared by all variants here; the mixed-effects loops are in the review and in the original papers 2208.04112 §2.2.2 §2.3.2.

# Per-tree training sample for clustered rows  [arxiv:2208.04112 §2.2.2]
def tree_sample(rows_by_subject, two_stage):
    subjects = list(rows_by_subject.keys())
    # Resample subjects with replacement, not rows  [arxiv:2208.04112 §2.2.2]
    drawn = sample_with_replacement(subjects, size=len(subjects))
    in_bag = []
    for subject in drawn:
        rows = rows_by_subject[subject]
        if two_stage:
            # Adler et al.: keep one random row per drawn subject
            in_bag.append(random_choice(rows))
        else:
            # RF++: keep every row of the drawn subject
            in_bag.extend(rows)
    # OOB is measured per subject, not per row  [arxiv:2208.04112 §2.2.2]
    out_of_bag = [
        row
        for subject in subjects if subject not in drawn
        for row in rows_by_subject[subject]
    ]
    return in_bag, out_of_bag

Where the variants differ:

• Two-stage vs RF++: the only change is how many rows per drawn subject enter the tree 2208.04112 §2.2.2.
• Historical RF: each time-varying predictor is replaced by its windowed summary before the split search, so the window and threshold must be tuned 2208.04112 §2.3.1. The review does not state the candidate grid or sub-sample size.
• Mixed-effects hybrids: wrap the forest in an alternating loop that refits the random effects and checks a generalized log-likelihood. The review does not state the convergence tolerance 2208.04112 §2.3.2.
• Distance-based criteria: the node impurity is a sum of pairwise or Mahalanobis terms. The review's text gives the split gain with a sign that would select the worst split if taken literally, and its formula is not normalised by node size. Check the original sources before coding it 2208.04112 §3.1.1.

Contradictions

• Is MERT / MERF code available? The software table lists an R program for MERT and MERF from the original author 2208.04112 tab. 3. The text says the MERT / MERF code is not publicly available 2208.04112 §4, §5.
• High-dimensional evidence. The review reports that MERT / RE-EM based trees and forests work on high-dimensional data and beat both the linear mixed model and a standard forest, citing a 2021 study 2208.04112 §2.3.2. Its later summary says that, except for BiMM and REEMforest, the methods have not been evaluated on high-dimensional data 2208.04112 §5.
• Variable importance for historical RF. The method section describes importance by deleting a variable and refitting 2208.04112 §2.3.1. The software table labels the historical RF importance as permutation importance 2208.04112 tab. 3. These are different procedures. Check which one the software computes.
• Resampling unit for mixed-effects hybrids. MERF resamples observations and relies on its random effects to absorb within-subject correlation 2208.04112 §2.3.2. The subject-level methods argue for resampling whole subjects to avoid that correlation 2208.04112 §2.2.2.
• Two-stage bootstrap. Reported as the best option in simulations, with the caveat that results depend on the setting 2208.04112 §2.2.2. The review does not claim it generalises.

Papers

• Hu and Szymczak's review: narrative review of tree-ensemble methods for repeated measurements, with model equations, fitting loops and a software table. No experiments and no code of its own.