Paper · 2026-10-05 · cs.LG
RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning
RoSA freezes the early layers of an LLM, keeps the last layer trainable, and rotates one block of 8 consecutive layers through the rest each epoch, with NanoAdam sparse optimizer states inside the active block. On Llama-2-7B it reaches 79.61% average on eight commonsense tasks against 79.99% for full AdamW, at 34.9 GiB peak VRAM against 78.5 GiB (author-reported).
Parameter-efficient fine-tuningDynamic sparse trainingTransfer learningfine-tuningmemory-efficiencylayer-schedulingsparse-optimizationllm
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Fine-tuning a large language model normally needs a lot of memory, because the optimizer keeps extra numbers for every trainable weight. Most cheap methods solve this by fixing a small set of trainable parameters for the whole run. RoSA takes a different route. It freezes the layers nearest the input, keeps the last layer trainable, and lets a block of neighbouring layers in the middle and upper part of the model take turns being trainable. At any moment only one such block is learning, so the memory needed right now stays small, while over the whole run more of the network gets updated. Inside the active block, only a sparse subset of weights keeps optimizer state.
Frozen layers also do not need the backward pass to reach them, so the later stages of training run a little faster. The price is time. The schedule needs more epochs, so a RoSA run can take longer than full fine-tuning when full fine-tuning fits in memory.
TL;DR
RoSA fine-tunes an LLM while keeping only one block of consecutive layers trainable at a time. Early layers stay frozen, the final layer is always trainable, and the block moves to the next group of layers each epoch, with NanoAdam keeping optimizer states for a sparse subset of the active weights. On Llama-2-7B it reaches 79.61% average accuracy against 79.99% for full AdamW, at 34.9 GiB peak VRAM against 78.5 GiB (author-reported). It is slower in wall-clock time: the two-cycle run took 6.5 h against 4.37 h for full fine-tuning tab. 3.
Problem
Full fine-tuning with AdamW keeps optimizer states for every trainable parameter, which is the main memory cost for a 7B model §1. The paper reports 78.5 GiB peak for this setting on Llama-2-7B §3 tab. 3. Parameter-efficient methods such as LoRA, DoRA and QLoRA cut this by restricting training to a fixed low-rank or adapter subspace for the whole run §1. The authors argue that this restricts adaptation capacity, and they ask whether the trainable part can move over time instead while a large input-side prefix stays frozen §1.
The paper also argues that the sequential structure of a Transformer can be used directly: frozen early layers need no optimizer state, no parameter gradients, and no backward pass through them §2.
Method
RoSA combines two mechanisms §2:
- Rotation. The layers fall into three parts: a frozen prefix (the input embeddings and early layers), an always-active anchor (the last Transformer layer and the output head), and the middle-to-upper layers that remain. The middle layers are cut into K runs of neighbouring layers, and in each epoch one run is trainable alongside the anchor §2 alg. 1.
- Sparse optimization inside the active block. NanoAdam keeps optimizer states only for a subset of the active parameters, chosen by weight magnitude and refreshed periodically §2.
The rotation rule is per epoch, not per step. Epoch e activates group k = 1 + ((e-1) \bmod K), and the active set is the anchor plus that group alg. 1. A group that goes inactive loses its optimizer states. When it is activated again, its moments are rebuilt from zero (a cold start), so optimizer memory does not grow with each visit §2 alg. 1. The paper reports that this cold start does not cause loss spikes at group transitions, including the return to the first group app. E fig. 5.
The memory argument has two factors. Dense Adam-style fine-tuning needs optimizer memory proportional to the total parameter count \Psi times the bytes per parameter K_{\mathrm{opt}}. RoSA is stated to scale with (l/L)\,\Psi\,\rho\,K_{\mathrm{opt}}, where l is the active block size, L the number of layers and \rho the sparsity density inside the active block §2 tab. 1. This is the optimizer-state part only; the frozen prefix also removes parameter gradients and activation storage for weight gradients in those layers §2 app. C.
The sparse subset is the bottom-\rho fraction of the active weights ordered by absolute value, so the small-magnitude weights are the ones that get optimizer states §2 alg. 1. The mask is recomputed every \Delta t steps because magnitudes change during training §2 alg. 1.
The paper states that the frozen prefix is not used for caching in its experiments. The forward pass is always computed in full. Caching the output of the last frozen layer is described as a future option, and the paper notes that it would add storage and bandwidth costs §2 § Limitations.
# RoSA training loop [alg. 1]
# Input: params, data, epochs E, group size b, density rho, refresh dt
# Groups G_1..G_K are consecutive, |G_k| <= b; anchor A always active
# [alg. 1] [§2]
def rosa_train(params, data, num_epochs, groups, anchor, rho, dt):
for epoch in range(1, num_epochs + 1):
# Epoch e activates group k = 1 + ((e - 1) mod K) [alg. 1]
k = 1 + ((epoch - 1) % len(groups))
# Active set S_e is the anchor plus group k [alg. 1]
active = anchor + groups[k - 1]
# Freeze everything else and drop its optimizer states [alg. 1]
freeze_outside(params, active)
# Fresh moments for each active layer, including revisited ones [alg. 1]
init_optimizer_states(active)
for step, (x, y) in enumerate(data):
# Gradients only flow through the active parameters [alg. 1]
loss = model_loss(x, y)
grads = backward(loss, active)
# Recompute the bottom-rho-by-|w| mask every dt steps [alg. 1] [§2]
if step % dt == 0:
mask = select_bottom_rho_by_magnitude(active, rho)
# Update only masked entries; store moments only for them [alg. 1]
adamw_update(active, grads, mask)The main Llama-2-7B schedule is two cycles over three groups, which is six epochs in total. The paper does not state the group boundaries explicitly for rotation. The static-subset rows (layers 8–15, 16–23 and 24–30) suggest the partition used tab. 8 app. A, but this is an inference from the table.
Hyperparameters
Values are for the default Llama-2-7B RoSA run unless noted. Where the paper does not state a value, the table says so.
| Name | Value | Where | Notes |
|---|---|---|---|
| Model | Llama-2-7B | §3 | Qwen-2.5-3B and Mistral-v0.1-7B also used app. A |
| Frozen prefix | input embeddings and Layers 0–7 | §2 app. A | same prefix for all three models app. A |
| Always-active anchor | final Transformer layer and output head | §2 app. A | |
| Rotatable layers | Layers 8–30 | app. A | the final Transformer layer is the anchor; total depth is not stated in the paper |
| Active group size b | 8 layers | app. A tab. 6 | Mistral-v0.1-7B uses 8, Qwen-2.5-3B uses 9 app. A |
| Schedule | cyclic, 6 epochs (two cycles) | app. A | single cycle of 3 epochs also reported §3 tab. 3 |
| Optimizer inside active block | NanoAdam | app. A | |
| Learning rate | 1\times10^{-4} | app. A | |
| Betas | \beta_1 = 0.9, \beta_2 = 0.95 | app. A | |
| Sparsity density \rho | 0.2 | app. A tab. 11 | 0.1 used for matched-sparsity comparison and ablation §3 app. B |
| Mask refresh \Delta t | 600 steps | app. A alg. 1 | 100 steps in the high-frequency ablation app. B |
| Moment storage | bfloat16 moments with 16-bit indices | app. C | reduces optimizer memory; changing precision changes the reported numbers |
| Micro-batch size | 1 | app. A | |
| Total batch size | 128 | app. A | |
| Gradient checkpointing | not used | app. A | |
| Cutoff length | 256 | app. A tab. 7 | the GaLore baseline used 64 tab. 7 |
| Hardware | 4 × A100 40 GB | app. A | baselines above 40 GB run on 80 GB A100s app. A |
| Learning-rate schedule, warmup, weight decay | not stated | ||
| LoRA and DoRA rank, alpha, target modules | not stated | applies to the RoSA-LoRA and RoSA-DoRA hybrids too | |
| LISA | 8 active layers; learning rate 1\times10^{-4} chosen from five values | app. A | |
| GaLore | 8-bit variant, rank 64, learning rate 1\times10^{-4}, cutoff 64 | app. A tab. 7 | run on 80 GB GPUs |
| Baseline batch and epochs | global batch 128, micro-batch 1, 3 epochs | app. A | for LISA, GaLore and static subsets |
Results
Author-reported. Setup: Llama-2-7B unless stated. The main metric averages eight commonsense reasoning benchmarks covering yes/no questions, physical and social reasoning, pronoun resolution, grade-school science and open-book questions §3 app. A. GSM8K and the other models appear in separate tables §3.
| Setting | Metric | Baseline | This paper | Source |
|---|---|---|---|---|
| Llama-2-7B, 8 tasks | average accuracy | FFT AdamW 79.99; DoRA 79.70; LoRA 77.61 | RoSA + NanoAdam 79.61 | tab. 2 |
| Llama-2-7B | peak VRAM | FFT AdamW 78.5 GiB | RoSA 34.9 GiB | tab. 3 |
| Llama-2-7B | training time | FFT AdamW 4.37 h, 3 epochs | RoSA 1 cycle 3.5 h (79.10%); 2 cycles 6.5 h (79.61%) | tab. 3 |
| Llama-2-7B, matched \rho=0.1 | peak VRAM | full-model NanoAdam 46.3 GiB (81.71%) | rotation 33.9 GiB | §3 app. C tab. 3 |
| Llama-2-7B, group size | average accuracy | size 4: 77.92% (12 epochs, 22.46 h); size 12: 79.33% (4 epochs, 7.22 h) | size 8: 79.61% (6 epochs, 6.50 h) | tab. 6 |
| Llama-2-7B | average accuracy, vs LISA and GaLore | LISA 70.05% at 30.71 GiB; GaLore 31.28% at 41.44 GiB (cutoff 64) | RoSA group 8, 1 cycle 79.10% at 34.9 GiB; group 6 78.50% at 30.93 GiB | tab. 7 |
| Llama-2-7B, static subsets | average accuracy | best static block (8–15) 78.32% at 26.2 GiB; 8–23: 78.07% at 31.5 GiB | RoSA 1 cycle 79.10% at 34.9 GiB | tab. 8 |
| Llama-2-7B, core-only | average accuracy | full schedule 79.61% | rotation only over 8–23: 79.56% | tab. 10 |
| Llama-2-7B, sparsity | average accuracy | \rho=0.2, \Delta t=600: 79.6%; \rho=0.1, \Delta t=600: 76.7%; \rho=0.1, \Delta t=100: 78.5% | tab. 11 app. B | |
| Llama-2-7B, seeds | average accuracy | original run 79.61% | second seed 79.49% | tab. 13 app. E |
| Qwen-2.5-3B, 8 tasks | accuracy and peak VRAM | NanoAdam 83.47% at 22.48 GB | RoSA + NanoAdam 82.91% at 14.37 GB | tab. 5 |
| Mistral-v0.1-7B, 8 tasks | accuracy and peak VRAM | NanoAdam 84.25% at 45.30 GB | RoSA + NanoAdam 81.86% at 33.28 GB | tab. 5 |
| Qwen-2.5-3B, DoRA | accuracy and peak VRAM | DoRA 81.58% at 16.42 GB | RoSA-DoRA 81.79% at 12.87 GB; 3 seeds 81.83 ± 0.32% | tab. 9 app. E |
| GSM8K, consumer GPU budget | accuracy and peak VRAM | LoRA + AdamW 36.69% at 16.20 GB; BAdam 24.48% at 16.7 GB | RoSA-LoRA 35.25% at 14.6 GB | tab. 4 |
The paper also reports that the active-block memory footprint is about 7.35% of the parameters at a single step, and that across a full run RoSA touches about 18.8% of the parameters, or roughly 1.26 billion on Llama-2-7B §3 app. C. These are authors' counts of the sparse update set, not measured weight changes.
Figures, described in words: the schedule diagram shows the frozen prefix, the anchor and the rotating group fig. 1; the VRAM plot shows full fine-tuning and full-model sparse optimization above the 40 GB budget fig. 2; the gradient plot shows early layers quiet and middle-to-upper layers active, which motivates the freezing fig. 3; similar per-layer profiles appear for Mistral-v0.1-7B and Qwen-2.5-3B fig. 4 app. D; the training loss steps down at each epoch boundary, with no spike at the return to the first group fig. 5 app. E.
Paper vs code
No official code.
The arXiv abstract page has no code link, and the paper text has no URL. A GitHub repository search for "RoSA rotational sparse adaptation", "RoSA rotational" and "rotational sparse adaptation" returned nothing relevant. The GitHub profile of the corresponding author Rebekka Burkholz lists only unrelated repositories. No GitHub handle was found for the other two authors. No code was checked against the paper.
Pitfalls
- Rotation is per epoch, and group boundaries are not stated. The algorithm changes the group at each epoch alg. 1. The text gives group size 8 over Layers 8–30 app. A. It does not write out the three groups, so an implementer has to choose them. The static-subset rows use 8–15, 16–23 and 24–30 tab. 8.
- Mask selection is the small-magnitude end. The text and the algorithm both say the subset is the bottom fraction by absolute value §2 alg. 1. This is the opposite of keeping large-magnitude weights. A reader who implements a top-k rule will get different results.
- Mask-refresh counter scope is not stated. The algorithm refreshes when t \bmod \Delta t = 0 alg. 1, but t is not defined as a global, per-epoch or per-group counter. It is also not stated whether the mask is rebuilt at group boundaries.
- Optimizer-state precision changes memory. The reported memory uses bfloat16 moments and 16-bit indices app. C. Storing them in fp32 would increase the optimizer part of the memory budget.
- Cold starts are the default. Moments are discarded when a group leaves and re-created when it returns §2 alg. 1. This is part of the method, not a bug. The paper reports no loss spikes at transitions app. E fig. 5.
- Units are mixed. The paper reports memory in GiB, but its 40 GB budget is in GB §3 fig. 2. Our conversion of the 34.9 GiB headline is about 37.5 GB, so it still fits under 40 GB, but the margin is smaller than the GiB number suggests.
- Accuracy gaps are not discussed. Full-model NanoAdam at \rho=0.1 reaches 81.71% against 79.61% for RoSA tab. 3. On Mistral-v0.1-7B, NanoAdam reaches 84.25% against 81.86% for RoSA tab. 5. The paper calls RoSA close to full fine-tuning, which holds against full AdamW. The NanoAdam baselines score higher than RoSA in these tables.
- The speedup is only in the backward pass. The forward pass is always computed in full §2 § Limitations. The "up to 27%" faster per-iteration figure applies to later groups §2 §3. Total wall-clock time is longer than full fine-tuning for the two-cycle run tab. 3.
- Epoch counts in the group-size study follow a pattern the paper does not state. The 4-, 8- and 12-layer groups use 12, 6 and 4 epochs tab. 6 app. A. If the rotatable Layers 8–30 are split into ceil(23 / b) groups, each run is two cycles. This is our inference from the numbers.
- Some table averages do not reproduce exactly. The per-task RoSA values in tab. 2 average to 79.625, against 79.61 in the table. DoRA averages to 79.69, against 79.70 in the table. This is probably rounding in the per-task values, and the difference is at most 0.01 points.
- The core-only ablation table lists six of the eight tasks tab. 10. Its caption says the average covers all eight, so that average cannot be checked from the table alone.
- Hybrid adapter settings are not stated. The LoRA and DoRA rank, alpha and target modules used with RoSA are not given tab. 4 tab. 9. Adapter optimizer states are described as already small, so memory savings come mostly from removing gradients and states in inactive layers §3.
- Caching the frozen prefix is future work. The paper estimates that the boundary activations for Llama-2-7B at cutoff length 256 take up to 2 MiB per sequence, and it notes that offloading them to CPU is bandwidth-limited § Limitations. Caching is not implemented in any reported run.
- The frozen prefix length was not ablated. Layers 0–7 are used for all three models. The gradient probes suggest a longer prefix may work for Qwen and Mistral, but this was not tested § Limitations app. D.
- Only one extra seed was run for Llama-2-7B app. E tab. 13. Seed variance is reported only for the Qwen DoRA setting, over three seeds.
Lineage
- Builds on: FreezeOut (progressive freezing for a shorter backward pass) and greedy layer-wise training, both cited in related work §1.
- Compared against: BAdam (block-coordinate Adam), LISA (random layer sampling), GaLore (low-rank gradient projection), LoRA, DoRA, and static layer subsets §1 §3 app. A.
- Uses: NanoAdam as the sparse optimizer in the active block §2.
- Superseded by: not known from this wiki.
Source and license
Adapted from “RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning” by Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz (arXiv:2610.06243), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.