Paper · 2026-10-19 · cs.LG
Scaling Parameter-Efficient Fine-Tuning for Brain Encoding
LoRA (rank 16) on omni-modal backbones of 4.7B to 104B parameters, read at decoder layer 32, plus a linear or cross-attention readout, predicts whole-brain fMRI from movies. One causal pass per 149 s window (about 350x cheaper extraction). Best single model: 0.327 ID and 0.228 OOD Pearson r on Algonauts 2025 (author-reported).
Parameter-efficient fine-tuningLow-rank adaptation (LoRA and variants)fMRI encoding modelMultimodal language modelbrain-encodingfmriparameter-efficient-fine-tuningloramultimodal
arXivPDFCodeMarkdown15 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A brain encoding model tries to predict how a person's brain responds when they watch or listen to something. The usual recipe takes a large AI model that was trained on other tasks, freezes it, and trains only a small output layer to map its internal features onto brain measurements. The authors ask whether the big model itself can be adjusted too. Brain data is scarce and expensive, so they change only a small set of extra weights and leave the rest of the model as it was. This keeps the general knowledge of the big model and lets the brain data shape how it represents the stimulus.
Running a large model over a long movie again and again for every moment of brain data would be far too slow. The authors feed each short clip through the model once and read off the internal states at every moment, instead of re-running it on overlapping windows. The result is a model that predicts brain activity across the whole cortex for movies it was not trained on, and that beats the previous best published single model on the same public benchmark, according to the authors.
TL;DR
- Parameter-efficient fine-tuning (LoRA on the attention projections of the first 32 decoder layers of Qwen3-Omni, plus a readout) predicts whole-brain fMRI responses to movies better than a frozen large backbone with a readout. §4 §5
- One causal forward pass per 149 s window gives per-timestep features and makes fine-tuning through the backbone affordable. §4
- The authors report a new single-model state of the art on Algonauts 2025 and a 261-model ensemble that ranks first on the post-challenge leaderboard at the time of writing. Both are author-reported. §5.2 §5.5
- No code for the ensemble, the batched multi-LoRA kernel, the alternative PEFT methods, or the transfer experiments is in the official repository. app. C app. H
Problem
Brain encoding models mostly use a frozen pretrained network with a trained readout. Training smaller models from scratch and fully fine-tuning a pretrained backbone on neural data have so far been reported to underperform this approach, and neural data is scarce relative to the size of current models §1. Low-rank adaptation won Algonauts 2023 with a small ViT but had not been tested on large multimodal models §1. Large backbones are also costly to run: prior work on the Algonauts 2025 benchmark re-encodes overlapping windows for each fMRI timestep, about 2,800 forward passes per clip, which makes fine-tuning through the backbone impractical §4 app. C.
The benchmark is CNeuroMod/Algonauts 2025: whole-brain fMRI of subjects watching seven seasons of Friends with synchronised video, audio and transcript, with held-out Season 7 (ID) and out-of-distribution movies (OOD) §5.
Method
Formulation. An encoder is a pretrained backbone f_\theta followed by a readout g_\phi trained to minimise a loss \mathcal{L} between the measured response Y and the prediction eq. 1. The frozen regime fixes \theta at its pretrained value \theta_0. PEFT restricts \theta to \theta_0 + \Delta(\psi) with \psi of dimension k \ll d. Full fine-tuning lets all d parameters move eq. 2 §3. The paper's main method sits in the PEFT regime, using LoRA as \Delta §3.
\hat{Y}=g_{\phi}\big(f_\theta(X)\big),\qquad \min_{\theta\in\Theta,\;\phi}\;\mathcal{L}\big(Y,\hat{Y}\big) eq. 1
\Theta_{\text{frozen}}=\{\theta_0\}\subset\Theta_{\text{PEFT}}=\{\theta_0+\Delta(\psi):\psi\in\mathbb{R}^k\}\subset\Theta_{\text{full}}=\mathbb{R}^{d},\qquad k\ll d,\;\;\Delta(0)=0 eq. 2
Single-pass extraction §4 app. B. The authors cut the clip into 149 s windows (100 TRs). Each window, preceded by a 60 s context lead, is encoded in one forward pass of the causally masked backbone, which yields a representation for every timestep. The paper reports this as about 350x less feature-extraction compute than the sliding-window protocol of the prior state of the art (about 700 down to 2 GPU-hours on an A800) §4 tab. 4. The released code interleaves audio, video and transcript in 20 s chunks, with each word placed at its end time inputs.py.
Adapter §4 app. S. LoRA with rank 16 and alpha 32 and no dropout is attached to the query, key, value and output projections of each decoder layer kept in the model. The decoder is truncated after layer 32 of 48, so layers above it are neither run nor adapted app. D. The adapter adds 8.91M trainable parameters. The audio and vision towers are frozen encoder.py encoder.py.
Linear readout §4 app. S tab. 19. Features are projected per modality from 2,048 to 512 dimensions, shared across frames. Each TR then averages the frames that fall into eight log-spaced lag blocks, which gives 8 x 3 x 512 = 12,288 features per TR. A per-subject linear map sends these to 1,000 parcels. The readout has 52.3M trainable parameters.
# Linear readout: project per modality, then lag-average [§4] [app. S]
def linear_readout(frames, subject):
# Project each modality's 2048-d frame state to 512-d, shared over time
projected = einsum(frames - proj_mean, proj) # (298, 3, 512)
# Average the frames inside each of the 8 lag blocks, per TR [tab. 19]
lagged = lag_operator @ projected.reshape(298, -1) # (100, 8 * 3 * 512)
# Centre the features, then apply this subject's linear map [app. S]
features = lagged - feature_mean # (100, 12288)
return features @ W[subject] + b[subject] # (100, 1000)Cross-attention readout §4 app. S. In place of the lag average, each TR attends over the language-model token sequence: about 3.2k tokens, consisting of audio at its native rate, every word, and one mean-pooled vision vector per second. There are 8 learned queries per modality, a key/query width of 128 and a per-block additive bias that is floored at -12 and starts out reproducing the lag average. This adds 0.2M parameters. The paper states that its gain over the linear readout comes mainly from access to the larger token set, not from attention over the same pooled features §5.3 app. K.
Training §4 app. S app. E. Adapter and readout are trained jointly on one window per step, with the mean-squared error over all four subjects' 100 x 1,000 targets. The main text does not name the training loss; Appendix L refers to minimising squared error, and the code uses MSE. The schedule runs for 1,429 steps, one epoch over the training windows (Friends seasons 1-5 and movie10). Learning rates are 6e-4 for the LoRA parameters, 6e-4 for the per-modality projection and 3e-4 for the readout. Training uses 30 warm-up steps, then cosine decay to zero, with global-norm gradient clipping at 1.0. Training starts from a closed-form ridge solution on frozen features. A second epoch overfits app. E tab. 8.
# Train LoRA and readout, one window per step [§4] [app. S]
def train_step(window, backbone, readout, optimizer):
# Encode 60 s of lead plus the 149 s window in one causal pass [§4]
span = media_span(window.start - 60.0, window.end)
# Tap the output of decoder layer 32 at each token [app. D]
tokens = backbone.forward_to_layer(span, layer=32)
# Pool the states onto the 2 Hz grid, per stream [app. B]
frames = grid_frames(tokens, span) # (298, 3, 2048)
# Predict all 4 subjects at once [§4]
preds = readout(frames) # (4, 100, 1000)
# Mean squared error against z-scored fMRI
# [code: src/brain_lora/training.py#L168-L170]
loss = mean_squared_error(preds, window.fmri)
# Clip the global norm to 1.0, then one AdamW step [app. S]
clip_global_norm(trainable_parameters, 1.0)
optimizer.step()
return lossClosed-form re-fit §4 app. D app. E. After training, each subject's linear map is re-solved by ridge regression on the adapted features. The penalty is chosen separately for each parcel from the grid {1e3, 1e4, 1e5, 1e6, 1e7}, using the Pearson r on a 15% holdout of the training windows. The solution is then recomputed on fit and holdout together. The authors report this step as the largest single training-side gain after the adapter §5.3 tab. 2.
# Closed-form re-fit of each subject's linear map [§4] [app. D]
def refit(train_windows):
# Hold out 15% of the training windows [app. D]
fit_set, holdout_set = split(train_windows, holdout_fraction=0.15)
# Accumulate X'X, X'Y and column sums per subject, in float64
moments_fit = accumulate(fit_set)
moments_holdout = accumulate(holdout_set)
for each parcel:
# Choose the penalty with the best holdout Pearson r [app. D]
best_alpha = argmax(alpha in ALPHAS, holdout_r(fit_set, alpha))
# Re-solve on fit and holdout together, with the chosen penalty
W = solve(XtX_all + best_alpha * I, XtY_all) # per parcel
return WEnsemble §5.5 app. G app. S. The authors train 261 adapted models that differ in hyperparameters, readout and backbone, and combine them per parcel with a minimum-variance combiner. The weights come from out-of-fold predictions on Season 6, split by episode. A 15-model subset is picked greedily and then refined by trying single swaps app. D.
Modality attribution §5.4 app. Q. For each window the authors compute exact Shapley values over video, audio and transcript. An absent modality's input embeddings are replaced with their training-window mean, which keeps the token count and positions. All eight subsets are evaluated attribution.py.
Multi-LoRA kernel §5.5 app. C. A fused batched gather-matmul trains many rank-16 adapters on one shared backbone. The authors report a 29.8x speedup over sequential training at N=32 and a step cost of 1.08x a single adapter. The speedup comes from hardware utilisation, not reduced FLOPs app. C. The kernel is not in the official repository.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbone | Qwen3-Omni, 31.7B total parameters (MoE, 3B active) | tab. 12 constants.py | Hugging Face Qwen/Qwen3-Omni-30B-A3B-Thinking, revision 2f443cfc4c54 tab. 20 |
| Window | 149 s = 100 TRs; TR 1.49 s | §4 app. S constants.py | 298 frames on a 2 Hz grid |
| Context lead | 60 s | §4 app. B tab. 3 | Accuracy flat beyond 60 s app. B |
| Hemodynamic shift | 4.47 s, i.e. 9 frames | app. S | TR j's block centre sits at frame 9 + 2.98 j |
| Feature layer | decoder layer 32 of 48; decoder truncated there | app. D app. S | Chosen on frozen-feature sweep; layers 31-34 within 0.0006 tab. 6 |
| LoRA rank / alpha | 16 / 32 | app. S encoder.py | Dropout 0 |
| LoRA targets | q, k, v, o projections of every kept decoder layer | app. S encoder.py | 8.91M parameters §4 |
| LoRA init | A Kaiming-uniform, B zero | app. S | Training starts at the frozen model |
| Projection | 2,048 to 512 per modality, shared across frames | app. S | 52.3M readout parameters in total §4 |
| Lag blocks | (delay, width) s: (0.5, 2), (2.5, 2), (4.5, 4), (8.5, 8), (16.5, 16), (32.5, 32), (64.5, 64), (128.5, 64) | tab. 19 | Tiles [0.5, 192.5) s; 158 of 800 rows are zero per window |
| Cross-attention | 8 queries per modality, key width 128, bias floor -12, +0.2M parameters | app. S | About 3.2k tokens per span |
| Optimiser | AdamW, default betas; weight decay 0.01 | app. S training.py | Weight decay is not applied to readout groups in code, see Pitfalls |
| Learning rates | LoRA 6e-4; projection 6e-4; readout 3e-4 | app. S tab. 7 training.py | Readout optimum at 3e-4 app. E |
| Batch | 1 window (400 target rows) | app. S | No gradient accumulation |
| Schedule | 1,429 steps; 30 warm-up; cosine to zero | app. S app. E training.py | One epoch; second epoch overfits tab. 8 |
| Gradient clip | global norm 1.0 | app. S training.py | |
| Precision | bf16 backbone; patch-embed conv in fp32 | app. S encoder.py | fp32 conv is about four orders of magnitude faster on cuDNN 9 app. S |
| Ridge penalty | per parcel, from {1e3, 1e4, 1e5, 1e6, 1e7} | app. S app. D refit.py | Chosen on 15% holdout of training windows |
| Training data | Friends seasons 1-5 and movie10; 1,429 windows | app. E | Season 6 never used for fitting or selecting penalties app. D |
| Validation | Friends season 6, 250 windows | app. D | Used for all selection |
| Seeds | 15 seeds, LoRA + linear: 0.32609 +/- 0.00024 | app. D | Differences below 0.00077 are not separated |
| Full fine-tuning LR | 1e-5 (best of sweep 3e-6 to 1e-4) | app. S | 20.25B backbone parameters trained |
| Ensemble | 261 models; per-parcel minimum-variance combiner | app. G app. S | Shrinkage grid {0.01, 0.03, 0.1, 0.3, 0.6, 0.9} app. S |
Results
Author-reported. Setup: CNeuroMod/Algonauts 2025. Models train on four subjects; Season 6 is the validation set; ID and OOD scores come from the challenge server §5. The metric is the mean over subjects of the per-parcel Pearson r over 1,000 parcels §5 metrics.py.
Table 1 tab. 1. Main results. BrainLoRA rows are the authors' runs. Rows marked with a dagger are the cited authors' own numbers quoted from their papers.
| Model | Trainable params | val | ID | OOD |
|---|---|---|---|---|
| Single model | ||||
| TRIBE (dagger) tab. 1 | -- | -- | 0.303 | 0.196 |
| MIRAGE (dagger) tab. 1 | 1.1B | 0.319 | 0.310 | 0.217 |
| Linear + frozen backbone | 52.3M | 0.291 | 0.287 | 0.187 |
| Linear + full fine-tuning | 20.3B | 0.323 | 0.320 | 0.217 |
| BrainLoRA (Linear) | 61.2M | 0.326 | 0.322 | 0.222 |
| Cross-attn + frozen backbone | 52.5M | 0.312 | 0.307 | 0.212 |
| Cross-attn + full fine-tuning | 20.3B | 0.330 | 0.326 | 0.225 |
| BrainLoRA (Cross-attn) | 61.4M | 0.331 | 0.327 | 0.228 |
| Ensemble | ||||
| TRIBE, 1000 models (dagger) | -- | -- | 0.320 | 0.215 |
| Vibe (dagger) | 20 x 329M | -- | 0.323 | 0.212 |
| MIRAGE, 15 models (dagger) | 15 x 1.1B | 0.335 | 0.323 | 0.227 |
| BrainLoRA, best 15 models | 15 x 61M | 0.347 | 0.341 | 0.247 |
| BrainLoRA, 261 models | 261 x 61M | 0.348 | 0.343 | 0.248 |
Table 2 tab. 2. Ablation, Friends s06 subject mean. Each row changes one choice from the main configuration (LoRA r16 with linear readout, 60 s lead, closed-form re-fit).
| Configuration | s06 |
|---|---|
| Frozen backbone | 0.2907 |
| + LoRA r16 (ours) | 0.3263 |
| without closed-form re-fit | 0.3202 |
| Rank 32 | 0.3268 |
| Gated-MLP readout | 0.3269 |
| Cross-attention readout | 0.3310 |
| No audio | 0.3026 |
| Qwen3.8-27B backbone (as printed) | 0.3191 |
| Gemma-3-27B backbone | 0.3029 |
Key findings, all author-reported:
- Adapter gain. LoRA improves the frozen backbone by +0.0356 on s06 and on every parcel tab. 2 app. P. The gain holds on all nine backbones, from 4.7B to 104B parameters, by +0.032 to +0.066 with a linear readout §5.1 tab. 12. Other PEFT families (IA3, bottleneck adapters) also improve on the frozen backbone, reaching 62% to 79% of the LoRA gain app. H tab. 11. At a matched size, the Houlsby bottleneck adapter trails LoRA by 0.0073 app. H.
- Full fine-tuning of 20.25B parameters scores 0.323 on validation with a linear readout and 0.330 with cross-attention. That is no better than LoRA, although it trains 330 times as many parameters §5.2.
- Transfer. The adapter keeps 97% of its gain on a held-out subject §5.2 app. M tab. 14. Applied zero-shot to the NNDb dataset, which has a different site, scanner and TR, it adds +11.1 points of the noise ceiling, on par with the +10.7 gain on Algonauts validation §5.2 tab. 15.
- Compute. Extraction takes 1.99 GPU-hours against 701 for the sliding window. End-to-end, including training and the re-fit, the single-pass configuration takes 8.5 GPU-hours against 702 (83x less) on one A800 tab. 4 app. C.
- Modality attribution. Early visual cortex is driven mainly by video (72.5%). Auditory cortex is explained mostly by audio (72.9%). In lateral temporal cortex, audio accounts for 60.3% and text for only 12.5% §5.4 fig. 4. Adaptation raises the average audio contribution by 39% and lowers text by 17%. The share of cross-modal information read out rises from 8.9% to 15.3% §5.4 app. Q.
- Ensemble. The 261-model ensemble reaches 0.3483 validation, 0.3428 ID and 0.2483 OOD, and is first on the public post-challenge leaderboard at the time of writing §5.5 tab. 1. Ensemble accuracy follows m(n) = m_inf - A n^(-b) with b of about 0.94 (R^2 = 0.998) §5.5 fig. 5.
Paper vs code
Repository: https://github.com/zfying/brain-lora, commit
9cfa54fdb4021a991d3a5afdf9286c9eadbaff12, Apache-2.0. The paper links it on
its first page, and the README cites the paper README.md.
The repository contains the training, closed-form re-fit, evaluation,
attribution and comparison code, plus the adapter checkpoints on the Hugging Face
Hub README.md. Not in the repository: the ensemble combiner and
roster, the multi-LoRA kernel, the full fine-tuning runs, the Houlsby, Pfeiffer
and IA3 adapters, the anchored-ridge calibration, the NNDb pipeline and the
shifted-target control.
Matches (checked): backbone revision and model identity constants.py; TR, HRF shift and window geometry constants.py; LoRA rank 16, alpha 2r, dropout 0, targets on q/k/v/o of 32 layers encoder.py encoder.py; readout dimensions and the lag operator readouts.py; cross-attention size and bias floor training.py; 1,429 steps, 30 warm-up, learning rates, clipping training.py; ridge grid refit.py; re-solve on fit plus holdout refit.py.
Differences:
- Readout initialisation lead. Appendix S says the initialisation is ridge on frozen features "at the same context lead" as the main run app. S. The released pipeline fits the initialisation on frozen features at a 30 s lead reproduce.sh and the feature extractor defaults to 30 s extract_features.py. The main runs use 60 s. The effect on the result is not measured.
- Initialisation is PCA plus ridge. The paper describes a ridge solution on frozen features app. S. The code first fits a per-modality PCA basis (2,048 to 512) on a random sample of training frames, then ridge fit_readout_init.py. The paper does not mention the PCA step.
- Weight decay scope. The paper says "AdamW ... weight decay 0.01" without scope app. S. In code, the LoRA group takes the default 0.01 training.py. The readout and projection groups explicitly set weight decay to 0 readouts.py.
- Holdout selection. The paper says the penalty holdout is a "deterministic strided 15%" of the training windows app. D. The code draws a seeded random permutation and takes its first 15% refit.py refit.py refit.py.
- Context lead at clip start. The lead is clipped at zero, so the first window of a clip gets less than 60 s of context dataset.py. The paper does not state this.
- Audio and video chunking. The code interleaves audio, video and text in 20 s chunks inputs.py. The paper does not describe the chunk structure.
- Loss. The main text does not name the training loss; Preliminaries allow "squared error or negative correlation" §3, and Appendix L refers to squared error app. L. The code uses mean-squared error over all subjects, parcels and TRs training.py.
Pitfalls
- Lead at clip boundaries. The first window of each clip has less than the 60 s lead, because the code clips the lead at time 0 dataset.py. Plan for this in any clip-based pipeline; the paper's ablations do not isolate it.
- Weight decay scope. The code applies 0.01 to the LoRA parameters only and 0 to the readout and projection. Applying 0.01 to every parameter, as the paper's wording suggests, departs from the released code readouts.py.
- Not exactly causal numerically. Single-pass features differ from features computed on shorter prefixes. Audio cosine is about 0.96 at the 140 s prefix, and the first chunk deviates most. The authors attribute this to bf16 numerics in the mixture-of-experts backbone, not to future-stimulus leakage §4 app. F tab. 9.
- Patch-embed convolution in fp32. The vision patch embedding must run in float32; bf16 under cuDNN 9 is reported to be roughly four orders of magnitude slower app. S encoder.py.
- Re-fit is required. Without it the score drops by 0.0061 tab. 2. The readout learning rate interacts with the re-fit: after re-fit the rate effect is +0.00025 app. E.
- Noise threshold. Rows within 0.00077 of one another are not separated by seed variance §5 app. D. Rank 32 is +0.0006 over rank 16 and is not significant app. E tab. 5.
- Second epoch overfits. The peak is at the end of the first epoch, step 1,400; two epochs lower the score by 0.0047 to 0.0147 app. E tab. 8.
- Ensemble claims depend on unreleased code. The uniform mean of the 261 models scores 0.34007; the fitted stack 0.34834 app. D. The combiner and roster are not in the repository, so these numbers cannot be reproduced from code.
- Data and storage. The media cache at 336x224 is about 130 GB, and the Algonauts data requires its own download README.md.
- Compute numbers are for one A800. Extraction (1.99 GPU-h) and training (4.84 GPU-h) are measured on the authors' hardware tab. 4. The 350x figure depends on that hardware §4.
- Ablation names. Two backbone names appear as printed in the paper (Qwen3.8-27B, Gemma-3-27B). Both lack audio input and are not comparable to the nine omni-modal backbones app. I tab. 2.
- Shifted-target control. Training on misaligned fMRI reduces the score by 0.0733 relative to its matched control, so the aligned gain is not a statistical artefact of the targets app. L tab. 13. This control is not in the repository.
Lineage
- Builds on: MIRAGE, the prior state of the art on this benchmark. It uses a frozen Qwen3-Omni backbone, a 1.1B transformer readout and sliding-window extraction §4 §5.2 tab. 1. This paper keeps MIRAGE's window geometry and fixed hemodynamic shift app. S README.md.
- Builds on: LoRA as the adapter §3 §4. The cross-attention readout extends the lag-average readout and starts from the same ridge solution app. S.
- Related work: MEM used LoRA on a vision backbone and won Algonauts 2023 §2. The paper states that its scale, a 104B backbone, goes beyond prior brain-tuned models, which stay near 10B app. A.
- Compared against: TRIBE, Vibe and MIRAGE ensembles tab. 1, and the PEFT alternatives IA3 and bottleneck adapters app. H.
Source and license
Adapted from “Scaling Parameter-Efficient Fine-Tuning for Brain Encoding” by Zhuofan Josh Ying, Guanglong Sun, Nolan Ho, Kieran Liu, Rui Yang, Andrew Chen, Nikolaus Kriegeskorte, Pony Ma, Hossein Adeli (arXiv:2610.26802), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/zfying/brain-lora at commit 9cfa54fdb402 (Apache-2.0). Any quoted code is shown under that license.