wikillm

Paper · 2026-10-26 · cs.LG, cs.CL

Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching

Adaptive-LoRA adapts only the layers that a frozen-base calibration ranks highest: each layer's attention and MLP outputs are zeroed in turn on a small input set, and the top ~30% of layers get rank-16 LoRA. Author-reported on Qwen2.5-1.5B over 11 reasoning tasks: 59.1% commonsense average vs 56.9% for all-layer LoRA, using 8 of 28 layers.

Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningActivation patchingMechanistic interpretabilityloraparameter-efficient-fine-tuninglayer-selectionmechanistic-interpretabilityactivation-patching

arXivPDFMarkdown12 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

LoRA is the usual way to fine-tune a large language model cheaply. It leaves the original weights alone and trains small add-on matrices, and by default it adds them to every layer of the model. The authors ask whether every layer needs them. They look inside the model to see how much each part contributes when LoRA is used, and they find that the contributions are uneven. The feed-forward part of a layer matters more than attention, and the layers that matter most depend on the task.

Their method, Adaptive-LoRA, uses that finding to choose layers before any training starts. It runs the untrained base model on a small set of example inputs. For each layer, it switches off the attention part and then the feed-forward part, and it records how much the model's top predicted word changes. Layers that change the answer a lot are kept, and LoRA is trained only on those. Roughly a third of the layers are used. The authors report that this beats the all-layer approach on their reasoning tests while training far fewer parameters.

TL;DR

Adaptive-LoRA picks the LoRA layers from a calibration pass on the frozen base model. Each layer's attention and MLP outputs are zeroed in separate forward passes, and the summed absolute change in the top token's probability is that layer's score §4.3 eq. 4 eq. 5. The top k layers (about 30%) get rank-16 LoRA on Q/K/V or Q/V, and the rest stay frozen §4.4 §4.5. On Qwen2.5-1.5B, the authors report higher average accuracy than all-layer LoRA at every rank tested, using 8 of 28 layers tab. 1.

Problem

Standard LoRA applies the same adapter rank to every transformer layer, although layers may contribute unequally §1. Layer-wise pruning work suggests layer importance varies §1. Heuristics such as tuning only the last layers or certain middle layers are static and ignore task differences §2. Choosing the best k of L layers by exhaustive search means checking C(L, k) subsets, which is impractical §4.1. The paper asks whether interventions on a frozen model can identify task-specific layers before training §1.

Method

Analysis (section 3, motivation). On Qwen2.5-0.5B (24 layers) with rank-8, alpha-16 LoRA on Q/K/V, the authors trained adapters on four instruction datasets §3.1. They measure a component's contribution as the absolute change in the top-token probability when that component is ablated eq. 1, averaged over inputs eq. 2. Their reported findings are: MLP blocks account for 58.4% of the measured effect against 41.6% for attention §3.2; within attention, the value projection carries about 46% §3.3; layer importance is non-uniform, with early layers (L0 to L2) shared across tasks §3.4 §3.5; and the top-7 layer sets for different task pairs have an average Jaccard similarity of 39.5% eq. 3 §3.5. The MLP share is measured as mediation through the block output: the LoRA adapters themselves sit only on Q, K and V §3.2 app. B.

Phase 1, calibration §4.3 alg. 1. Calibration uses only the frozen base model and needs no labels. For each input x, take the top token t* and its probability. Then, for each layer l, run two independent forward passes, one with the attention output zeroed and one with the MLP output zeroed eq. 4. The layer score is the mean over inputs of the sum of the two absolute probability drops eq. 5. No gradients and no adapters are used §4.3 app. E.

Pseudocode
# Phase 1: score each layer on the frozen base model  [alg. 1] [eq. 5]
def calibration_scores(model, calib_inputs, num_layers):
    # One running score per layer, starting at zero  [alg. 1]
    scores = zeros(num_layers)                      # (L,)
    for x in calib_inputs:
        # Top token of the base model; no gold label is used  [§4.3]
        probs = model.next_token_probs(x)
        top = argmax(probs)
        p_full = probs[top]
        for l in range(num_layers):
            # Zero the attention output of layer l  [eq. 4]
            p_no_attn = model.next_token_probs(x, zero=("attn", l))[top]
            # Zero the MLP output of layer l, separately  [eq. 4]
            p_no_mlp = model.next_token_probs(x, zero=("mlp", l))[top]
            # Add both absolute drops to the layer's score  [eq. 5]
            scores[l] += abs(p_full - p_no_attn) + abs(p_full - p_no_mlp)
    # Average over the calibration inputs  [eq. 5]
    return scores / len(calib_inputs)

Phase 2, selection §4.4 alg. 1. Keep the k highest-scoring layers. In accuracy mode, all layers are candidates. In efficiency mode, layers below a depth floor d* are removed first, so that the frozen prefix can be cached §4.4 app. E. The main reasoning experiments use k of about 0.3L (8 of 28 for Qwen2.5-1.5B) §5.1. The authors also report that selecting the top half of layers is a practical default and improves on full LoRA on eight models §4.4 app. F.

Pseudocode
# Phase 2: keep the k best layers  [alg. 1] [§4.4]
def select_layers(scores, k, min_depth=None):
    # Efficiency mode only: drop layers above the depth floor d*  [§4.4]
    candidates = list(range(len(scores)))
    if min_depth is not None:
        candidates = [l for l in candidates if l >= min_depth]
    # Top-k by score; tie-breaking is not stated  [alg. 1]
    return top_k(candidates, scores, k)

Phase 3, selective training §4.5 alg. 1. For each selected layer and each target projection with weight W of shape (d_out, d_in), attach a rank-r LoRA pair so that the update is ΔW = B·A. Unselected layers and all base weights stay frozen. The paper states that the adapter shapes accommodate unequal input and output dimensions §4.5. Which projections to adapt is decided per experiment §4.5.

Pseudocode
# Phase 3: LoRA on the selected layers only  [alg. 1] [§4.5]
def build_adapted_model(model, selected, rank, targets):
    for l in selected:
        for proj in targets:                     # e.g. Q, K, V  [§4.5]
            # Shapes (r, d_in) and (d_out, r); init scheme not stated  [§4.5]
            A, B = new_lora_factors(proj, rank)
            attach_lora(model, l, proj, A, B)    # delta W = B @ A
    freeze_base_weights(model)                   # base stays frozen  [§4.5]
    return model

Hyperparameters

NameValueWhereNotes
LoRA rank r4, 8, 16, 32tab. 1 §5.1Full-layer baselines and the adaptive method at each rank
LoRA alpha32 at r16§4.5Main method text. Mechanistic analysis uses r8, alpha 16 §3.1
Adapted layers k8 of 28 (about 30%)§5.1 tab. 1Main Qwen2.5-1.5B runs; k about 0.3L in general §4.4
Calibration inputs N250tab. 1 §5.1 app. CMain 1.5B table and its text. Default in method text is 30 §4.3 app. C
Calibration inputs N (default)30§4.3 app. EUsed in the cross-model table tab. 2 and in calibration cost studies
Learning rate1e-4§4.5 app. CAdamW
OptimizerAdamW, weight decay 0.01, betas (0.9, 0.999), eps 1e-8app. CWeight decay scope not stated
Warm-upnoneapp. CLR schedule after warm-up not stated
Epochs3§4.5 §5.1All configurations app. C
Effective batch size64 per step; 256 with DDP across 4 GPUs§4.5 app. CThe two values conflict; see pitfalls
Max sequence length512 tokens§5.1 app. CLonger samples truncated app. C
PrecisionFP16 mixed precision (AMP); adapters kept in FP32 and cast to FP16app. CBase weights frozen in FP16
Training data, commonsensecommonsense_170k, 10,000 samples§5.1Subset selection not stated
Training data, mathmath_10k, 5,000 samples§5.1Subset selection not stated
In-context examplesnoneapp. CCommonsense: multiple-choice answer extraction
Decodinggreedy, temperature 0app. CMath: chain-of-thought with regex answer parsing
Seeds42 (main tables); 42, 123, 7 (supplement)tab. 1 app. BSingle seed in the main 1.5B comparison
Calibration hardwaresingle A100app. CTraining: 4 x A100 40GB, 512 GB RAM app. C
Dropout, LoRA scaling convention, grad clippingnot statedn/a
Depth floor d* for main runsnot stated§4.4Only the efficiency study sets it app. E

Results

Author-reported. Setup: Qwen2.5-1.5B (28 layers), 8 commonsense tasks (1,000 eval samples each; OpenBookQA 500) and 3 math tasks, chain-of-thought for math, seed 42 §5.1 tab. 1. Baselines are LoRA, LoRA+ and DoRA on all 28 layers.

SettingMetricLoRA / LoRA+ / DoRA (28 of 28 layers)Adaptive-LoRA (8 of 28)Source
r4commonsense avg48.5 / 49.5 / 47.149.8tab. 1
r8commonsense avg53.1 / 53.9 / 51.755.1tab. 1
r16commonsense avg56.9 / 57.9 / 55.559.1tab. 1
r32commonsense avg60.4 / 61.1 / 59.162.2tab. 1
r16math avg (GSM8K, SVAMP, AQUA)41.1 / 42.9 / 39.543.3tab. 1
r32math avg (GSM8K, SVAMP, AQUA)45.5 / 47.3 / 43.747.8tab. 1

Generalization at r16, 10,000 training samples, seed 42, top 30% of layers, compared with full-layer LoRA on the same 8 commonsense tasks tab. 2 §5.3:

ModelFull LoRA avgAdaptive-LoRA avgDifferenceLayers adapted
Qwen2.5-0.5B37.138.6+1.57 of 24
Qwen2.5-1.5B56.959.1+2.28 of 28
Qwen2.5-3B66.968.8+1.911 of 36
LLaMA-3-3B61.262.4+1.28 of 28
Qwen3-8B82.582.50.011 of 36
Qwen3-32B87.387.1-0.220 of 64

Layer-strategy ablation, perplexity on Alpaca-ZH (lower is better), r16, 3 epochs, 3,000 samples §5.4 app. B. At 12 of 36 layers: Adaptive 12.32, Random 16.42, Sequential 14.06, Reverse 15.00. All strategies reach 5.75 at 36 layers. This is the only comparison of selection strategies in the paper, and it is perplexity, not task accuracy.

Other reported figures: calibration takes 29 to 62 s on one A100 across the models in the cost study, or 4.1 to 6.3% of full-layer training time app. E §5.4. Training speedups of 1.08 to 1.13x (no prefix reuse) and up to 1.93x (steady epoch, 3B and 8B models) come from an H200 timing study of efficiency-mode configurations on other models app. B app. E.

Paper vs code

No official code.

No code link appears in the paper, its abstract or the source. The NeurIPS checklist (not a numbered section) says code will be released under MIT but gives no link. Searches (GitHub repository search via the API) for "adaptive-lora", "Adaptive-LoRA layer selection", "lora path patching", "layer sparsity lora", "mechanistic lora zero-ablation" and the title "Mechanistic Insights LoRA" returned no repository by the authors. Author-name searches for Yizhou Huang (Brunel), Fei Zuo (ECNU) and Kezhi Wang (Brunel) returned no users. The "adaptive-lora" search returned 484 matches, but the list was truncated and none of the visible items came from the authors. No project page was linked.

Pitfalls

  • Calibration size differs by table. The main 1.5B table uses 250 calibration inputs tab. 1 §5.1. The method's default is 30 §4.3 app. C, and the cross-model table also uses 30 tab. 2. Read N from each table caption. The paper does not say how the calibration examples are drawn from the training set, nor which prompt format and token position give p(t*|x) app. E.
  • Target projections for Adaptive-LoRA are not stated for the main table. The method text says the set is experiment-specific and that Finding 2 motivates a Q/V configuration §4.5. The baselines use Q, K and V §5.1. The 3.5x comparison is described at identical rank and target projections §5.2, which suggests Q/K/V for both. Q/V configurations appear in the appendix app. C app. B. Treat Q/K/V as the reading implied by the text, not as a stated setting.
  • Effective batch size conflicts. The method text gives 64 §4.5. The appendix gives 8 per device with 8 accumulation steps, which is 64 per device, and 256 across 4 GPUs with DDP app. C. The paper does not say which applied to the main results.
  • Calibration time range differs. Sections 4.5 and 6 give 20 to 62 s §4.5 §6. The cost table gives 29 to 62 s app. E, and the N=30 run on the 1.5B model takes about 40 s app. E. No table supports the 20 s lower bound.
  • Supplementary numbers conflict with the main table. The seed-42 Qwen2.5-3B gain at r16 is +1.9, with 66.9 for full LoRA and 68.8 for the method tab. 2. The supplement lists "Qwen-3B / r16 / 10K" at +1.5 for seed 42 app. B, and its absolute panel lists full LoRA at 58.13 for "Qwen-3B". The model naming also varies: Qwen-3B, Qwen2.5-3B and LLaMA-7B, LLaMA-13B and Qwen-7B appear in appendix tables but not in Table 2. LLaMA-3-3B tab. 2, LLaMA-3.2 (3B) app. F and Llama-3.2-3B app. E appear as separate names. Do not mix these values or model names.
  • The headline comparison is one seed on one model. The 1.3 to 2.2 point gains over LoRA on Qwen2.5-1.5B come from seed 42 only tab. 1 app. C. The multi-seed runs cover other models and do not include Qwen2.5-1.5B app. B.
  • No accuracy comparison against other 8-layer choices. The paper compares Adaptive-LoRA with full LoRA on accuracy. Random, Sequential and Reverse selection are compared only on perplexity §5.4 app. B. The one accuracy reference to random selection is a single row (70.2%) at Qwen-3B with 5,000 samples, and its k is not stated app. E. Adaptive is not best at every budget: at 28 layers Sequential is lower (6.55 vs 6.97), and at 32 layers Sequential (6.03) and Random (6.30) are lower than Adaptive (6.32) app. B.
  • The 3.5x figure is a count, not a measurement. It is L/k at equal rank: 28/8 on the LoRA factors, computed from model dimensions. It excludes DoRA's magnitude vectors and any other method-specific parameters app. B tab. 2. The same ratio for Qwen2.5-3B is 3.27x tab. 2. A parameter-matched baseline, such as full-layer LoRA at a lower rank, is not reported. The closest match in rank-times-layers is full LoRA at r4 (28 x 4 = 112) against Adaptive at r16 (8 x 16 = 128); by this count, the results are 48.5 and 59.1 tab. 1. This is a count from the card's arithmetic, not a figure the paper reports.
  • Training-time savings are not measured for the main results. The efficiency speedups come from a single H200 with a deeper frozen prefix and other models, and the efficiency-mode accuracy trade-off is a perplexity study on Qwen2.5-0.5B with two seeds app. B app. E. The depth floor d* used for main runs is not stated.
  • Zero-ablation is not the same as the trained-LoRA analysis. Section 3 measures contributions of a trained LoRA model eq. 1 and calls its intervention activation patching §3.1. Section 4.3 zeroes the base model's sublayer outputs without any adapter eq. 4. The appendix states these are different interventions app. B. The paper does not state the hook point: whether "output" means the sublayer output before the residual add or the block output after it §4.3.
  • Calibration is a single top-token probability. The score uses only the base model's top-token probability and sums two absolute drops, so a layer whose removal raises that probability scores the same as one whose removal lowers it. Tie-breaking in Top-k is not stated alg. 1.
  • Several training details are missing. The paper does not state the LoRA initialisation of A and B, dropout, the scaling convention alpha/r, the learning rate schedule after warm-up, gradient clipping, or whether each task is trained separately. The mechanistic-analysis training settings (learning rate, epochs) are also not stated §3.1 app. C.
  • MLP dominance does not mean MLP adapters help. The 58.4% figure is measured through MLP blocks, which have no adapters in that analysis §3.2 app. B. Adaptive-LoRA adapts only Q/K/V, so the MLP finding is motivation, not a design choice that the method tests.
  • Some appendix claims have no table or figure to check. The training-dynamics numbers (loss of 0.82 vs 0.91 and 0.89 at step 500; gradient norms 1.4 to 1.8x) are given in the text only app. B. The cross-architecture layer-position and MLP-ratio statements are qualitative app. F. The Pearson correlations for base-vs-adapter agreement are stated in the text but not tabulated §4.3 app. B.
  • Minor arithmetic differences. From the layer-ablation table, the reduction over Random at 12 layers is (16.42 - 12.32) / 16.42 = 25.0%, as stated, and the reduction relative to Reverse is (15.00 - 12.32) / 15.00 = 17.9%, where the text says 17.8% app. B. In the depth study, the deepest-eight reference's Code-Alpaca PPL increase is 2.343 / 2.294 - 1 = 2.1%, where the text says 2.2% app. E.
  • The 500-item reference-token check does not name its model. Replacing t* with the gold token gives a selection Jaccard of 0.81 and accuracies of 58.7% and 58.5% app. E. The model and task of that check are not stated.

Lineage

  • Builds on: LoRA, arXiv:2106.09685. Adaptive-LoRA keeps the adapter form and changes which layers receive it.
  • Builds on: path patching, Interpretability in the Wild (IOI), and activation patching, arXiv:2202.05262. The paper applies these to LoRA analysis and uses a base-model zero-ablation for selection.
  • Compared against: LoRA, arXiv:2106.09685; LoRA+, arXiv:2402.12354; DoRA, arXiv:2402.09353 tab. 1. Single-task comparison with InfLoRA, GainLoRA and TreeLoRA app. B.
  • Related but not compared: AdaLoRA, arXiv:2303.10512 (rank allocation) §2.
  • Superseded by: not known from this card's sources.

Source and license

Adapted from “Mechanistic Insights into LoRA: Layer Sparsity for Adaptive Fine-Tuning via Path Patching” by Fei Zuo, Yizhou Huang, Kezhi Wang (arXiv:2610.34756), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.