---
kind: concept
slug: mixed-effects-model
title: Mixed-effects models
summary: >-
  Mixed-effects models add per-subject random effects to a regression, so repeated measurements from the same subject are modelled correctly. Tree and forest hybrids (MERF, RE-EM, GMERF, BiMM and relatives) replace the linear fixed part with a learned one. Trade-off: the random-effect assumptions must hold, and the fitting loops are heavier than a plain forest.
tags: [mixed-models, longitudinal-data, biostatistics]
papers: ["2208.04112", "2402.17937"]
related: [random-forest, longitudinal-data-analysis, human-evaluation]
status: draft
updated_at: 2026-10-19
---

## Overview

A linear mixed model (LMM) writes the outcomes of subject $i$ as a fixed part,
shared by all subjects, plus a random part that gives each subject its own
offset and error structure: $y_i = X_i\beta + Z_ib_i + \epsilon_i$, with
$b_i\sim N(0,D)$ and $\epsilon_i\sim N(0,R_i)$. It is fitted by maximum
likelihood or REML [arxiv:2208.04112 §2.3.2]. The random effects absorb the
correlation between repeated measurements of one subject, which a standard
random forest ignores [arxiv:2208.04112 §2.1] [arxiv:2208.04112 §2.3.2].

The hybrids covered here keep the random part and replace the linear fixed
part $X_i\beta$ by a function $f(X_i)$ learned by a tree or a forest
[arxiv:2208.04112 eq. 3]. The generalised version (GLMM) does the same for
non-Gaussian outcomes through a link function. For binary outcomes, a Bayesian
variant also exists [arxiv:2208.04112 §2.3.3].

Two rules hold across the hybrids. For a subject seen in training, prediction
adds the estimated random effect $\hat b_i$. For a new subject, only the fixed
part is used [arxiv:2208.04112 §2.3.2]. Do not add random effects you do not
have.

## Variants

The tree and forest variants below come from one review. None has been compared
with the others in a single benchmark, and the review notes that the software
packages have never been benchmarked against each other [arxiv:2208.04112 §5].
The last subsection is a plain linear mixed model, used as an analysis step in
an applied study rather than as a hybrid method.

### MERT and MERF (Gaussian outcome)

The mixed-effects regression tree (MERT) and forest (MERF) fit the model by an
EM-style loop. Each round removes the current random effects from the outcome,
grows a tree or forest on the adjusted outcome, and then re-estimates the
random effects and variances from a linear mixed model. The loop stops when
the generalised log-likelihood stops changing [arxiv:2208.04112 §2.3.2].
The review assumes $R_i=\sigma^2 I_{n_i}$ for these methods [arxiv:2208.04112
§2.3.2]. MERF draws observation-level bootstrap samples and uses out-of-bag
predictions to limit overfitting. This is valid only if the random effects
capture all within-subject correlation [arxiv:2208.04112 §2.3.2]. Its
starting values are zero random effects, $\sigma^2=1$ and $D=I_q$, described
as slightly modified from the original papers [arxiv:2208.04112 §2.3.2]. The
code is not publicly available, according to the review [arxiv:2208.04112 §4] [arxiv:2208.04112 §5].

### RE-EM tree and REEMforest (Gaussian outcome)

The RE-EM tree uses the same first step as MERT. In the second step, the tree
only defines a partition of observations into leaves. A linear mixed model
then uses one fixed effect per leaf, fitted with `nlme::lme`. Because of that
fitter, $R_i$ may be non-diagonal, for example autocorrelated [arxiv:2208.04112
§2.3.2]. REEMforest averages $K$ RE-EM trees [arxiv:2208.04112 §2.3.2]. Both are
available as R packages on CRAN [arxiv:2208.04112 tab. 3]. REEMforest is one of
only two methods the review reports as evaluated on high-dimensional data
[arxiv:2208.04112 §5].

### Stochastic variants (SMERT, SMERF, SREEMtree, SREEMforest)

These add a subject-specific centred Gaussian process $\omega_i$, which models
deviations that are correlated over time within a subject. The covariance is
$\mathrm{Cov}(\omega_i(s),\omega_i(t))=\gamma^2\Gamma(s,t)$, and $\omega_i$ is
independent of $b_i$ and $\epsilon_i$ [arxiv:2208.04112 §2.3.2]. The
subject-level residual used for the next tree step subtracts the estimated
$\omega_i$ as well as $Z_i\hat b_i$ [arxiv:2208.04112 §2.3.2].

### GLMM hybrids (non-Gaussian outcome)

- **GMERT** runs penalised quasi-likelihood. A weighted MERT pseudo-model
  replaces the weighted LMM pseudo-model in each round. It needs starting
  means $\hat\mu_i$. The GMERT simulation used 0.25 for binary zeros and 0.75
  for binary ones, with no justification given [arxiv:2208.04112 §2.3.2].
- **GMET and GMERF** are the GLMM version of RE-EM. The leaf indicators enter
  as fixed effects, and the tree target is the linear predictor minus the
  random part. The starting linear predictor comes from a GLM on all
  predictors. This is impossible when the number of predictors exceeds the
  number of observations. GMERF replaces the tree with a forest
  [arxiv:2208.04112 §2.3.2].

### BiMM tree and forest (binary outcome, Bayesian)

The model is $\mathrm{logit}(\mu_{it})=\mathrm{CART}(X_{it})\beta+Z_{it}b_{it}$,
where the CART terms are leaf-membership indicators. The method alternates
between growing a tree or forest and fitting a Bayesian GLMM, and stops when the
change in posterior log-likelihood falls below a threshold
[arxiv:2208.04112 §2.3.3]. The threshold and the rule for the working outcome
are not stated [arxiv:2208.04112 §2.3.3]. The review reports BiMM as one of the
two methods evaluated on high-dimensional data [arxiv:2208.04112 §5].

### Plain linear mixed model in an applied study (no learned fixed part)

The fixed part stays linear here. An exploratory human-robot study of
LLM-guided exercises fitted a linear mixed model to per-session anxiety scores
from three delivery modes. The fitting used lme4 with lmerTest, REML, Satterthwaite
degrees of freedom, and full-information maximum likelihood for missing data
[arxiv:2402.17937 §2.6.3] [tab. 4]. Pre-session anxiety was included as a
covariate, with condition-specific slopes [arxiv:2402.17937 §3.2.2] [tab. 4].
Pre-session anxiety differed between conditions on day 1, and the mixed model
is the only adjustment for that difference [arxiv:2402.17937 §3.1] [§3.2.2].

Estimates reported against the worksheet reference: robot b = -6.26 (SE 1.80,
p = .001) and chatbot b = -6.74 (SE 2.04, p = .002). The day effect was b =
-0.33 (SE 0.21, p = .111) with 38.7% power, so the paper treats it as
inconclusive rather than as no change [arxiv:2402.17937 §3.2.2] [tab. 4].
Sample size was 38 completers, but the study is exploratory and small
[arxiv:2402.17937 §2.3.3].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Continuous outcome, errors close to independent within a subject | MERF (or MERT for one tree) | Fixed part is learned, random effects are per subject; diagonal $R_i$ is assumed [arxiv:2208.04112 §2.3.2] |
| Continuous outcome, within-subject residuals correlated (e.g. autocorrelated) | REEMforest | `nlme::lme` allows a non-diagonal $R_i$ [arxiv:2208.04112 §2.3.2] |
| Continuous outcome, subject deviations correlated over time | SMERF or SREEMforest | Adds a Gaussian process per subject. The review reports that a misspecified process has limited impact [arxiv:2208.04112 §2.3.2] |
| Continuous outcome, many predictors | REEMforest | Evaluated on high-dimensional data, where it beat LMM and standard RF in the cited study [arxiv:2208.04112 §2.3.2] [arxiv:2208.04112 §5] |
| Non-Gaussian outcome, number of predictors below number of observations | GMERF or GMET | Both need a GLM start, which fails when predictors outnumber observations [arxiv:2208.04112 §2.3.2] |
| Non-Gaussian outcome, predictors outnumber observations | GMERT, or BiMM for binary outcomes | GMERT starts from fixed values, not a GLM [arxiv:2208.04112 §2.3.2]. The review has not evaluated GMERT on high-dimensional data [arxiv:2208.04112 §5] |
| Binary outcome, Bayesian inference wanted | BiMM tree or forest | Bayesian GLMM with CART fixed terms [arxiv:2208.04112 §2.3.3] |
| Repeated per-session scores, a condition effect to estimate, small sample, fixed linear predictors | Plain linear mixed model (lme4, REML) | No tree is needed; the model handles repeated measures per participant. Report power for any non-significant time term, since a null result is not evidence of no change [arxiv:2402.17937 §2.6.3] [§3.2.2] |

Implementation access also matters. The CRAN packages LongituRF and REEMtree
cover RE-EM trees, REEMforest and the stochastic variants. The MERT and MERF
programs are listed as an R program from Dr. Ahlem Hajjem. The GMERT and BiMM
code are in the paper supplements [arxiv:2208.04112 tab. 3].

## Implementation sketch

The shared core of the Gaussian hybrids is one loop. The variants differ in how
the fixed part is grown and which random terms are subtracted.

```pseudocode
# Shared loop for tree/forest + random-effects hybrids  [arxiv:2208.04112 §2.3.2]
def fit_hybrid(y, X, Z, subjects, grow_fixed_part):
    # Start random effects at zero, sigma^2 = 1, D = identity  [§2.3.2]
    b = {i: zeros(q) for i in subjects}
    sigma2, D = 1.0, identity(q)
    while not converged(y, b, sigma2, D):
        # Remove the current random effects from the outcome  [§2.3.2]
        y_star = {i: y[i] - Z[i] @ b[i] for i in subjects}
        # Learn the fixed part with a tree or forest  [§2.3.2]
        f_hat = grow_fixed_part(y_star, X)
        # Re-estimate random effects and variances given f_hat  [§2.3.2]
        b, sigma2, D = fit_lmm(y, f_hat(X), Z)
    return f_hat, b, sigma2, D
```

Where variants differ:

- **RE-EM / REEMforest:** replace the tree prediction by one fixed effect per
  leaf inside the LMM, and allow a non-diagonal $R_i$ [arxiv:2208.04112 §2.3.2].
- **Stochastic variants:** also estimate $\hat\omega_i$ and subtract it from
  the outcome in each round [arxiv:2208.04112 §2.3.2].
- **GLMM / BiMM:** replace the LMM step by a GLMM, with a link function and,
  for BiMM, a Bayesian fit [arxiv:2208.04112 §2.3.2] [arxiv:2208.04112 §2.3.3].

Not stated in the review, so set them yourself or take them from the original
papers: the convergence tolerance, the BiMM stopping threshold, the number of
trees, $mtry$ and node size [arxiv:2208.04112 §2.3.2] [arxiv:2208.04112 §2.3.3].

## Contradictions

No known conflict between the two sources. The review reports no conflicting
results for the hybrid methods, and the applied study does not test them. The
evidence the review cites comes from simulations and one vaccine trial dataset
[arxiv:2208.04112 §2.3.2].

Within the applied study, the reported test degrees of freedom imply sample
sizes that do not match the 38 completers. The paired OQ-45 test has t(36), the
robot test t(13), the worksheet test t(10), and the NARS regression F(1, 9)
[arxiv:2402.17937 §3.2.1] [§3.4.2]. The paper does not explain this
[arxiv:2402.17937 §2.3.3]. Treat the mixed-model estimates as reported, not as
checked. Other sources may disagree, and none are included here.

## Papers

- [Random-forest review for longitudinal data](/papers/2208.04112): a narrative review of random-forest extensions for longitudinal data. It covers the mixed-effects hybrids on this page and runs no experiments of its own.
- [LLM-Powered Socially Assistive Robot-Delivered Cognitive Behavioral Therapy Exercises: an Exploratory Study with University Students](/papers/2402.17937): an exploratory three-arm study that uses a linear mixed model for repeated anxiety scores. It is an applied use of the model, not a hybrid method.