wikillm

Paper · 2026-10-05 · cs.LG

RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning

RoSA freezes the early layers of an LLM, keeps the last layer trainable, and rotates one block of 8 consecutive layers through the rest each epoch, with NanoAdam sparse optimizer states inside the active block. On Llama-2-7B it reaches 79.61% average on eight commonsense tasks against 79.99% for full AdamW, at 34.9 GiB peak VRAM against 78.5 GiB (author-reported).

Parameter-efficient fine-tuningDynamic sparse trainingTransfer learningfine-tuningmemory-efficiencylayer-schedulingsparse-optimizationllm

arXivPDFMarkdown11 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Fine-tuning a large language model normally needs a lot of memory, because the optimizer keeps extra numbers for every trainable weight. Most cheap methods solve this by fixing a small set of trainable parameters for the whole run. RoSA takes a different route. It freezes the layers nearest the input, keeps the last layer trainable, and lets a block of neighbouring layers in the middle and upper part of the model take turns being trainable. At any moment only one such block is learning, so the memory needed right now stays small, while over the whole run more of the network gets updated. Inside the active block, only a sparse subset of weights keeps optimizer state.

Frozen layers also do not need the backward pass to reach them, so the later stages of training run a little faster. The price is time. The schedule needs more epochs, so a RoSA run can take longer than full fine-tuning when full fine-tuning fits in memory.

TL;DR

RoSA fine-tunes an LLM while keeping only one block of consecutive layers trainable at a time. Early layers stay frozen, the final layer is always trainable, and the block moves to the next group of layers each epoch, with NanoAdam keeping optimizer states for a sparse subset of the active weights. On Llama-2-7B it reaches 79.61% average accuracy against 79.99% for full AdamW, at 34.9 GiB peak VRAM against 78.5 GiB (author-reported). It is slower in wall-clock time: the two-cycle run took 6.5 h against 4.37 h for full fine-tuning tab. 3.

Problem

Full fine-tuning with AdamW keeps optimizer states for every trainable parameter, which is the main memory cost for a 7B model §1. The paper reports 78.5 GiB peak for this setting on Llama-2-7B §3 tab. 3. Parameter-efficient methods such as LoRA, DoRA and QLoRA cut this by restricting training to a fixed low-rank or adapter subspace for the whole run §1. The authors argue that this restricts adaptation capacity, and they ask whether the trainable part can move over time instead while a large input-side prefix stays frozen §1.

The paper also argues that the sequential structure of a Transformer can be used directly: frozen early layers need no optimizer state, no parameter gradients, and no backward pass through them §2.

Method

RoSA combines two mechanisms §2:

  1. Rotation. The layers fall into three parts: a frozen prefix (the input embeddings and early layers), an always-active anchor (the last Transformer layer and the output head), and the middle-to-upper layers that remain. The middle layers are cut into K runs of neighbouring layers, and in each epoch one run is trainable alongside the anchor §2 alg. 1.
  2. Sparse optimization inside the active block. NanoAdam keeps optimizer states only for a subset of the active parameters, chosen by weight magnitude and refreshed periodically §2.

The rotation rule is per epoch, not per step. Epoch e activates group k = 1 + ((e-1) \bmod K), and the active set is the anchor plus that group alg. 1. A group that goes inactive loses its optimizer states. When it is activated again, its moments are rebuilt from zero (a cold start), so optimizer memory does not grow with each visit §2 alg. 1. The paper reports that this cold start does not cause loss spikes at group transitions, including the return to the first group app. E fig. 5.

The memory argument has two factors. Dense Adam-style fine-tuning needs optimizer memory proportional to the total parameter count \Psi times the bytes per parameter K_{\mathrm{opt}}. RoSA is stated to scale with (l/L)\,\Psi\,\rho\,K_{\mathrm{opt}}, where l is the active block size, L the number of layers and \rho the sparsity density inside the active block §2 tab. 1. This is the optimizer-state part only; the frozen prefix also removes parameter gradients and activation storage for weight gradients in those layers §2 app. C.

The sparse subset is the bottom-\rho fraction of the active weights ordered by absolute value, so the small-magnitude weights are the ones that get optimizer states §2 alg. 1. The mask is recomputed every \Delta t steps because magnitudes change during training §2 alg. 1.

The paper states that the frozen prefix is not used for caching in its experiments. The forward pass is always computed in full. Caching the output of the last frozen layer is described as a future option, and the paper notes that it would add storage and bandwidth costs §2 § Limitations.

Pseudocode
# RoSA training loop  [alg. 1]
# Input: params, data, epochs E, group size b, density rho, refresh dt
# Groups G_1..G_K are consecutive, |G_k| <= b; anchor A always active
# [alg. 1] [§2]

def rosa_train(params, data, num_epochs, groups, anchor, rho, dt):
    for epoch in range(1, num_epochs + 1):
        # Epoch e activates group k = 1 + ((e - 1) mod K)  [alg. 1]
        k = 1 + ((epoch - 1) % len(groups))
        # Active set S_e is the anchor plus group k  [alg. 1]
        active = anchor + groups[k - 1]
        # Freeze everything else and drop its optimizer states  [alg. 1]
        freeze_outside(params, active)
        # Fresh moments for each active layer, including revisited ones  [alg. 1]
        init_optimizer_states(active)
        for step, (x, y) in enumerate(data):
            # Gradients only flow through the active parameters  [alg. 1]
            loss = model_loss(x, y)
            grads = backward(loss, active)
            # Recompute the bottom-rho-by-|w| mask every dt steps  [alg. 1] [§2]
            if step % dt == 0:
                mask = select_bottom_rho_by_magnitude(active, rho)
            # Update only masked entries; store moments only for them  [alg. 1]
            adamw_update(active, grads, mask)

The main Llama-2-7B schedule is two cycles over three groups, which is six epochs in total. The paper does not state the group boundaries explicitly for rotation. The static-subset rows (layers 8–15, 16–23 and 24–30) suggest the partition used tab. 8 app. A, but this is an inference from the table.

Hyperparameters

Values are for the default Llama-2-7B RoSA run unless noted. Where the paper does not state a value, the table says so.

NameValueWhereNotes
ModelLlama-2-7B§3Qwen-2.5-3B and Mistral-v0.1-7B also used app. A
Frozen prefixinput embeddings and Layers 0–7§2 app. Asame prefix for all three models app. A
Always-active anchorfinal Transformer layer and output head§2 app. A
Rotatable layersLayers 8–30app. Athe final Transformer layer is the anchor; total depth is not stated in the paper
Active group size b8 layersapp. A tab. 6Mistral-v0.1-7B uses 8, Qwen-2.5-3B uses 9 app. A
Schedulecyclic, 6 epochs (two cycles)app. Asingle cycle of 3 epochs also reported §3 tab. 3
Optimizer inside active blockNanoAdamapp. A
Learning rate1\times10^{-4}app. A
Betas\beta_1 = 0.9, \beta_2 = 0.95app. A
Sparsity density \rho0.2app. A tab. 110.1 used for matched-sparsity comparison and ablation §3 app. B
Mask refresh \Delta t600 stepsapp. A alg. 1100 steps in the high-frequency ablation app. B
Moment storagebfloat16 moments with 16-bit indicesapp. Creduces optimizer memory; changing precision changes the reported numbers
Micro-batch size1app. A
Total batch size128app. A
Gradient checkpointingnot usedapp. A
Cutoff length256app. A tab. 7the GaLore baseline used 64 tab. 7
Hardware4 × A100 40 GBapp. Abaselines above 40 GB run on 80 GB A100s app. A
Learning-rate schedule, warmup, weight decaynot stated
LoRA and DoRA rank, alpha, target modulesnot statedapplies to the RoSA-LoRA and RoSA-DoRA hybrids too
LISA8 active layers; learning rate 1\times10^{-4} chosen from five valuesapp. A
GaLore8-bit variant, rank 64, learning rate 1\times10^{-4}, cutoff 64app. A tab. 7run on 80 GB GPUs
Baseline batch and epochsglobal batch 128, micro-batch 1, 3 epochsapp. Afor LISA, GaLore and static subsets

Results

Author-reported. Setup: Llama-2-7B unless stated. The main metric averages eight commonsense reasoning benchmarks covering yes/no questions, physical and social reasoning, pronoun resolution, grade-school science and open-book questions §3 app. A. GSM8K and the other models appear in separate tables §3.

SettingMetricBaselineThis paperSource
Llama-2-7B, 8 tasksaverage accuracyFFT AdamW 79.99; DoRA 79.70; LoRA 77.61RoSA + NanoAdam 79.61tab. 2
Llama-2-7Bpeak VRAMFFT AdamW 78.5 GiBRoSA 34.9 GiBtab. 3
Llama-2-7Btraining timeFFT AdamW 4.37 h, 3 epochsRoSA 1 cycle 3.5 h (79.10%); 2 cycles 6.5 h (79.61%)tab. 3
Llama-2-7B, matched \rho=0.1peak VRAMfull-model NanoAdam 46.3 GiB (81.71%)rotation 33.9 GiB§3 app. C tab. 3
Llama-2-7B, group sizeaverage accuracysize 4: 77.92% (12 epochs, 22.46 h); size 12: 79.33% (4 epochs, 7.22 h)size 8: 79.61% (6 epochs, 6.50 h)tab. 6
Llama-2-7Baverage accuracy, vs LISA and GaLoreLISA 70.05% at 30.71 GiB; GaLore 31.28% at 41.44 GiB (cutoff 64)RoSA group 8, 1 cycle 79.10% at 34.9 GiB; group 6 78.50% at 30.93 GiBtab. 7
Llama-2-7B, static subsetsaverage accuracybest static block (8–15) 78.32% at 26.2 GiB; 8–23: 78.07% at 31.5 GiBRoSA 1 cycle 79.10% at 34.9 GiBtab. 8
Llama-2-7B, core-onlyaverage accuracyfull schedule 79.61%rotation only over 8–23: 79.56%tab. 10
Llama-2-7B, sparsityaverage accuracy\rho=0.2, \Delta t=600: 79.6%; \rho=0.1, \Delta t=600: 76.7%; \rho=0.1, \Delta t=100: 78.5%tab. 11 app. B
Llama-2-7B, seedsaverage accuracyoriginal run 79.61%second seed 79.49%tab. 13 app. E
Qwen-2.5-3B, 8 tasksaccuracy and peak VRAMNanoAdam 83.47% at 22.48 GBRoSA + NanoAdam 82.91% at 14.37 GBtab. 5
Mistral-v0.1-7B, 8 tasksaccuracy and peak VRAMNanoAdam 84.25% at 45.30 GBRoSA + NanoAdam 81.86% at 33.28 GBtab. 5
Qwen-2.5-3B, DoRAaccuracy and peak VRAMDoRA 81.58% at 16.42 GBRoSA-DoRA 81.79% at 12.87 GB; 3 seeds 81.83 ± 0.32%tab. 9 app. E
GSM8K, consumer GPU budgetaccuracy and peak VRAMLoRA + AdamW 36.69% at 16.20 GB; BAdam 24.48% at 16.7 GBRoSA-LoRA 35.25% at 14.6 GBtab. 4

The paper also reports that the active-block memory footprint is about 7.35% of the parameters at a single step, and that across a full run RoSA touches about 18.8% of the parameters, or roughly 1.26 billion on Llama-2-7B §3 app. C. These are authors' counts of the sparse update set, not measured weight changes.

Figures, described in words: the schedule diagram shows the frozen prefix, the anchor and the rotating group fig. 1; the VRAM plot shows full fine-tuning and full-model sparse optimization above the 40 GB budget fig. 2; the gradient plot shows early layers quiet and middle-to-upper layers active, which motivates the freezing fig. 3; similar per-layer profiles appear for Mistral-v0.1-7B and Qwen-2.5-3B fig. 4 app. D; the training loss steps down at each epoch boundary, with no spike at the return to the first group fig. 5 app. E.

Paper vs code

No official code.

The arXiv abstract page has no code link, and the paper text has no URL. A GitHub repository search for "RoSA rotational sparse adaptation", "RoSA rotational" and "rotational sparse adaptation" returned nothing relevant. The GitHub profile of the corresponding author Rebekka Burkholz lists only unrelated repositories. No GitHub handle was found for the other two authors. No code was checked against the paper.

Pitfalls

  • Rotation is per epoch, and group boundaries are not stated. The algorithm changes the group at each epoch alg. 1. The text gives group size 8 over Layers 8–30 app. A. It does not write out the three groups, so an implementer has to choose them. The static-subset rows use 8–15, 16–23 and 24–30 tab. 8.
  • Mask selection is the small-magnitude end. The text and the algorithm both say the subset is the bottom fraction by absolute value §2 alg. 1. This is the opposite of keeping large-magnitude weights. A reader who implements a top-k rule will get different results.
  • Mask-refresh counter scope is not stated. The algorithm refreshes when t \bmod \Delta t = 0 alg. 1, but t is not defined as a global, per-epoch or per-group counter. It is also not stated whether the mask is rebuilt at group boundaries.
  • Optimizer-state precision changes memory. The reported memory uses bfloat16 moments and 16-bit indices app. C. Storing them in fp32 would increase the optimizer part of the memory budget.
  • Cold starts are the default. Moments are discarded when a group leaves and re-created when it returns §2 alg. 1. This is part of the method, not a bug. The paper reports no loss spikes at transitions app. E fig. 5.
  • Units are mixed. The paper reports memory in GiB, but its 40 GB budget is in GB §3 fig. 2. Our conversion of the 34.9 GiB headline is about 37.5 GB, so it still fits under 40 GB, but the margin is smaller than the GiB number suggests.
  • Accuracy gaps are not discussed. Full-model NanoAdam at \rho=0.1 reaches 81.71% against 79.61% for RoSA tab. 3. On Mistral-v0.1-7B, NanoAdam reaches 84.25% against 81.86% for RoSA tab. 5. The paper calls RoSA close to full fine-tuning, which holds against full AdamW. The NanoAdam baselines score higher than RoSA in these tables.
  • The speedup is only in the backward pass. The forward pass is always computed in full §2 § Limitations. The "up to 27%" faster per-iteration figure applies to later groups §2 §3. Total wall-clock time is longer than full fine-tuning for the two-cycle run tab. 3.
  • Epoch counts in the group-size study follow a pattern the paper does not state. The 4-, 8- and 12-layer groups use 12, 6 and 4 epochs tab. 6 app. A. If the rotatable Layers 8–30 are split into ceil(23 / b) groups, each run is two cycles. This is our inference from the numbers.
  • Some table averages do not reproduce exactly. The per-task RoSA values in tab. 2 average to 79.625, against 79.61 in the table. DoRA averages to 79.69, against 79.70 in the table. This is probably rounding in the per-task values, and the difference is at most 0.01 points.
  • The core-only ablation table lists six of the eight tasks tab. 10. Its caption says the average covers all eight, so that average cannot be checked from the table alone.
  • Hybrid adapter settings are not stated. The LoRA and DoRA rank, alpha and target modules used with RoSA are not given tab. 4 tab. 9. Adapter optimizer states are described as already small, so memory savings come mostly from removing gradients and states in inactive layers §3.
  • Caching the frozen prefix is future work. The paper estimates that the boundary activations for Llama-2-7B at cutoff length 256 take up to 2 MiB per sequence, and it notes that offloading them to CPU is bandwidth-limited § Limitations. Caching is not implemented in any reported run.
  • The frozen prefix length was not ablated. Layers 0–7 are used for all three models. The gradient probes suggest a longer prefix may work for Qwen and Mistral, but this was not tested § Limitations app. D.
  • Only one extra seed was run for Llama-2-7B app. E tab. 13. Seed variance is reported only for the Qwen DoRA setting, over three seeds.

Lineage

  • Builds on: FreezeOut (progressive freezing for a shorter backward pass) and greedy layer-wise training, both cited in related work §1.
  • Compared against: BAdam (block-coordinate Adam), LISA (random layer sampling), GaLore (low-rank gradient projection), LoRA, DoRA, and static layer subsets §1 §3 app. A.
  • Uses: NanoAdam as the sparse optimizer in the active block §2.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “RoSA: Rotational Sparse Adaptation for Memory-Efficient Fine-Tuning” by Muhammad Azeem Lodhi, Chao Zhou, Rebekka Burkholz (arXiv:2610.06243), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.