Mixed-effects models

Overview

A linear mixed model (LMM) writes the outcomes of subject i as a fixed part, shared by all subjects, plus a random part that gives each subject its own offset and error structure: y_i = X_i\beta + Z_ib_i + \epsilon_i, with b_i\sim N(0,D) and \epsilon_i\sim N(0,R_i). It is fitted by maximum likelihood or REML 2208.04112 §2.3.2. The random effects absorb the correlation between repeated measurements of one subject, which a standard random forest ignores 2208.04112 §2.1 2208.04112 §2.3.2.

The hybrids covered here keep the random part and replace the linear fixed part X_i\beta by a function f(X_i) learned by a tree or a forest 2208.04112 eq. 3. The generalised version (GLMM) does the same for non-Gaussian outcomes through a link function. For binary outcomes, a Bayesian variant also exists 2208.04112 §2.3.3.

Two rules hold across the hybrids. For a subject seen in training, prediction adds the estimated random effect \hat b_i. For a new subject, only the fixed part is used 2208.04112 §2.3.2. Do not add random effects you do not have.

Variants

The tree and forest variants below come from one review. None has been compared with the others in a single benchmark, and the review notes that the software packages have never been benchmarked against each other 2208.04112 §5. The last subsection is a plain linear mixed model, used as an analysis step in an applied study rather than as a hybrid method.

MERT and MERF (Gaussian outcome)

The mixed-effects regression tree (MERT) and forest (MERF) fit the model by an EM-style loop. Each round removes the current random effects from the outcome, grows a tree or forest on the adjusted outcome, and then re-estimates the random effects and variances from a linear mixed model. The loop stops when the generalised log-likelihood stops changing 2208.04112 §2.3.2. The review assumes R_i=\sigma^2 I_{n_i} for these methods [arxiv:2208.04112 §2.3.2]. MERF draws observation-level bootstrap samples and uses out-of-bag predictions to limit overfitting. This is valid only if the random effects capture all within-subject correlation 2208.04112 §2.3.2. Its starting values are zero random effects, \sigma^2=1 and D=I_q, described as slightly modified from the original papers 2208.04112 §2.3.2. The code is not publicly available, according to the review 2208.04112 §4 2208.04112 §5.

RE-EM tree and REEMforest (Gaussian outcome)

The RE-EM tree uses the same first step as MERT. In the second step, the tree only defines a partition of observations into leaves. A linear mixed model then uses one fixed effect per leaf, fitted with nlme::lme. Because of that fitter, R_i may be non-diagonal, for example autocorrelated [arxiv:2208.04112 §2.3.2]. REEMforest averages K RE-EM trees 2208.04112 §2.3.2. Both are available as R packages on CRAN 2208.04112 tab. 3. REEMforest is one of only two methods the review reports as evaluated on high-dimensional data 2208.04112 §5.

Stochastic variants (SMERT, SMERF, SREEMtree, SREEMforest)

These add a subject-specific centred Gaussian process \omega_i, which models deviations that are correlated over time within a subject. The covariance is \mathrm{Cov}(\omega_i(s),\omega_i(t))=\gamma^2\Gamma(s,t), and \omega_i is independent of b_i and \epsilon_i 2208.04112 §2.3.2. The subject-level residual used for the next tree step subtracts the estimated \omega_i as well as Z_i\hat b_i 2208.04112 §2.3.2.

GLMM hybrids (non-Gaussian outcome)

• GMERT runs penalised quasi-likelihood. A weighted MERT pseudo-model replaces the weighted LMM pseudo-model in each round. It needs starting means \hat\mu_i. The GMERT simulation used 0.25 for binary zeros and 0.75 for binary ones, with no justification given 2208.04112 §2.3.2.
• GMET and GMERF are the GLMM version of RE-EM. The leaf indicators enter as fixed effects, and the tree target is the linear predictor minus the random part. The starting linear predictor comes from a GLM on all predictors. This is impossible when the number of predictors exceeds the number of observations. GMERF replaces the tree with a forest 2208.04112 §2.3.2.

BiMM tree and forest (binary outcome, Bayesian)

The model is \mathrm{logit}(\mu_{it})=\mathrm{CART}(X_{it})\beta+Z_{it}b_{it}, where the CART terms are leaf-membership indicators. The method alternates between growing a tree or forest and fitting a Bayesian GLMM, and stops when the change in posterior log-likelihood falls below a threshold 2208.04112 §2.3.3. The threshold and the rule for the working outcome are not stated 2208.04112 §2.3.3. The review reports BiMM as one of the two methods evaluated on high-dimensional data 2208.04112 §5.

Plain linear mixed model in an applied study (no learned fixed part)

The fixed part stays linear here. An exploratory human-robot study of LLM-guided exercises fitted a linear mixed model to per-session anxiety scores from three delivery modes. The fitting used lme4 with lmerTest, REML, Satterthwaite degrees of freedom, and full-information maximum likelihood for missing data 2402.17937 §2.6.3 tab. 4. Pre-session anxiety was included as a covariate, with condition-specific slopes 2402.17937 §3.2.2 tab. 4. Pre-session anxiety differed between conditions on day 1, and the mixed model is the only adjustment for that difference 2402.17937 §3.1 §3.2.2.

Estimates reported against the worksheet reference: robot b = -6.26 (SE 1.80, p = .001) and chatbot b = -6.74 (SE 2.04, p = .002). The day effect was b = -0.33 (SE 0.21, p = .111) with 38.7% power, so the paper treats it as inconclusive rather than as no change 2402.17937 §3.2.2 tab. 4. Sample size was 38 completers, but the study is exploratory and small 2402.17937 §2.3.3.

When to use which

Implementation access also matters. The CRAN packages LongituRF and REEMtree cover RE-EM trees, REEMforest and the stochastic variants. The MERT and MERF programs are listed as an R program from Dr. Ahlem Hajjem. The GMERT and BiMM code are in the paper supplements 2208.04112 tab. 3.

Implementation sketch

The shared core of the Gaussian hybrids is one loop. The variants differ in how the fixed part is grown and which random terms are subtracted.

# Shared loop for tree/forest + random-effects hybrids  [arxiv:2208.04112 §2.3.2]
def fit_hybrid(y, X, Z, subjects, grow_fixed_part):
    # Start random effects at zero, sigma^2 = 1, D = identity  [§2.3.2]
    b = {i: zeros(q) for i in subjects}
    sigma2, D = 1.0, identity(q)
    while not converged(y, b, sigma2, D):
        # Remove the current random effects from the outcome  [§2.3.2]
        y_star = {i: y[i] - Z[i] @ b[i] for i in subjects}
        # Learn the fixed part with a tree or forest  [§2.3.2]
        f_hat = grow_fixed_part(y_star, X)
        # Re-estimate random effects and variances given f_hat  [§2.3.2]
        b, sigma2, D = fit_lmm(y, f_hat(X), Z)
    return f_hat, b, sigma2, D

Where variants differ:

• RE-EM / REEMforest: replace the tree prediction by one fixed effect per leaf inside the LMM, and allow a non-diagonal R_i 2208.04112 §2.3.2.
• Stochastic variants: also estimate \hat\omega_i and subtract it from the outcome in each round 2208.04112 §2.3.2.
• GLMM / BiMM: replace the LMM step by a GLMM, with a link function and, for BiMM, a Bayesian fit 2208.04112 §2.3.2 2208.04112 §2.3.3.

Not stated in the review, so set them yourself or take them from the original papers: the convergence tolerance, the BiMM stopping threshold, the number of trees, mtry and node size 2208.04112 §2.3.2 2208.04112 §2.3.3.

Contradictions

No known conflict between the two sources. The review reports no conflicting results for the hybrid methods, and the applied study does not test them. The evidence the review cites comes from simulations and one vaccine trial dataset 2208.04112 §2.3.2.

Within the applied study, the reported test degrees of freedom imply sample sizes that do not match the 38 completers. The paired OQ-45 test has t(36), the robot test t(13), the worksheet test t(10), and the NARS regression F(1, 9) 2402.17937 §3.2.1 §3.4.2. The paper does not explain this 2402.17937 §2.3.3. Treat the mixed-model estimates as reported, not as checked. Other sources may disagree, and none are included here.

Papers

• Random-forest review for longitudinal data: a narrative review of random-forest extensions for longitudinal data. It covers the mixed-effects hybrids on this page and runs no experiments of its own.
• LLM-Powered Socially Assistive Robot-Delivered Cognitive Behavioral Therapy Exercises: an Exploratory Study with University Students: an exploratory three-arm study that uses a linear mixed model for repeated anxiety scores. It is an applied use of the model, not a hybrid method.