wikillm

Paper · 2026-11-03 · cs.LG

Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning

M2LE merges each LoRA update into the inference weights with element-wise strengths instead of one shared scalar. HIM solves a Fisher-weighted joint old/new loss for the merged update; TSP trains the update on label-disjoint halves of each batch so it tolerates later merges. Author-reported: 81.34 last accuracy on 10-task ImageNet-R, versus 79.95 for LoRA-P&M.

Continual learningLow-rank adaptation (LoRA and variants)Model mergingCatastrophic forgettingcontinual-learningloramodel-merging

arXivPDFCodeMarkdown16 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A model that learns tasks one after another tends to forget the earlier ones. A common cheap approach trains a small add-on (a LoRA update) for each new task and folds it into the main model, so there is one model to serve. The usual folding step scales the whole add-on by a single number. That treats every weight change the same, even though some of them matter a lot for old tasks and others hardly matter.

This paper changes the folding step. For each weight, it chooses how strongly to apply the new change, based on how sensitive the old tasks are to that weight. It also changes how the add-on is trained: part of each training batch is used to nudge the add-on and check the effect on the other part, so the add-on ends up in a region that survives being merged with later tasks. The authors report that this beats earlier adapter and LoRA methods on several image benchmarks, without keeping old data.

TL;DR

  • Replaces the shared merge scalar of LoRA-based continual learning with a per-element weight derived from a joint old/new loss objective §4.3.
  • Adds a training step that perturbs the current LoRA on one half of a batch and scores it on the other half, to make the update robust to later merges §4.4.
  • Author-reported gains on ImageNet-R, ImageNet-A, CIFAR100, CUB and DomainNet over LoRA-P&M and other rehearsal-free methods tab. 1 tab. 2.
  • Released code covers only the 10-task ImageNet-R configuration; several settings in the paper are not reproducible from the repository imgr10.json.

Problem

Rehearsal-free continual learning must merge each new task's update into one inference model without access to old data §1 §4.1. Existing LoRA merging methods apply one shared factor to the whole update. The authors argue that different elements of the update prefer different strengths §3 fig. 1. The second issue is that a merged update in a sharp region of the loss landscape can lose old-task accuracy when later merges move the weights §1 §4.4.

Method

The inference weights after task t are \mathbf{W}_t = \mathbf{W}_{t-1} + \Delta\mathbf{W}^*_t. The backbone is frozen and the task-t update is a low-rank product \Delta\mathbf{W}_t = \mathbf{B}_t\mathbf{A}_t §4.2 eq. 1. Two components shape the merged update: TSP shapes \Delta\mathbf{W}_t during training, and HIM computes \Delta\mathbf{W}^*_t after training §4.2.

HIM, Hessian-guided adaptive merging §4.3. The method defines two losses. \mathcal{J}_{\mathrm{old}} is the summed change in old-task losses caused by a candidate merged update, and \mathcal{J}_{\mathrm{new}} is the change in the new-task loss relative to the trained update eq. 2 eq. 3. The target is the update that minimises \mathcal{J}_{\mathrm{old}} + \lambda\mathcal{J}_{\mathrm{new}} eq. 4. Old data are unavailable, so the authors expand each loss to second order around the converged points and drop the first-order terms eq. 5 eq. 6 eq. 7. Setting the gradient to zero gives a linear system in the sum of old-task Hessians and the current-task Hessian eq. 8. Appendix B derives it app. B.

Replacing each Hessian by its diagonal empirical Fisher gives a closed element-wise rule. Every entry of \Delta\mathbf{W}_t is scaled by \lambda F_t / (F_{1:t-1} + \lambda F_t + \varepsilon) eq. 9. The Fisher of task t is estimated on that task's training data at \mathbf{W}_{t-1}+\Delta\mathbf{W}_t, after training §4.3. The running sum F_{1:t}=F_{1:t-1}+F_t is stored and reused for the next task, so old data are not needed §4.3 app. C.4 tab. 10.

Pseudocode
# Hessian-guided adaptive merge after task t  [§4.3] [eq. 9]
# delta_w:      trained LoRA update of task t, (d_out, d_in)
# fisher_now:   diagonal empirical Fisher of task t, same shape
# fisher_past:  running sum of Fisher over tasks 1..t-1 (zeros at t=1)
# lam:          weight of the current-task term (lambda)
# damping:      small constant (epsilon), value not stated in paper

def him_merge(inference_weight, delta_w, fisher_now, fisher_past, lam, damping):
    # Per-element merge strength  [eq. 9]
    strength = lam * fisher_now / (fisher_past + lam * fisher_now + damping)
    # Merged update, element-wise rescaling of delta_w  [§4.3] [eq. 9]
    merged_update = strength * delta_w
    # Merge into the inference weights  [§4.2]
    inference_weight = inference_weight + merged_update
    # Accumulate Fisher for the next task  [§4.3]
    fisher_past = fisher_past + fisher_now
    return inference_weight, fisher_past

TSP, task simulation perturbation §4.4. Each mini-batch is split at random into two subsets \mathcal{S}_1 and \mathcal{S}_2 with disjoint label sets §4.4 alg. 1. The procedure runs in two directions. For direction \mathcal{S}_1 \to \mathcal{S}_2, it takes the gradient of the source subset eq. 10, builds a probe along the descent direction scaled by \rho eq. 11, and scores the target subset at both the probed and the unprobed LoRA eq. 12. The training objective mixes the two target losses with weight \beta on the probed term eq. 13. One gradient step follows eq. 14, and the reverse direction \mathcal{S}_2 \to \mathcal{S}_1 repeats the step from the updated parameters eq. 15. Training on each task runs until convergence alg. 1. Appendix A gives the full loop app. A alg. 1.

Pseudocode
# Task simulation perturbation, one mini-batch  [§4.4] [eq. 10-15]
# lora: current-task LoRA factors (A_t, B_t)
# rho:  probe scale;  beta: weight of the probed loss

def tsp_step(batch_inputs, batch_labels, lora, rho, beta, lr):
    # Split the batch into two label-disjoint subsets  [§4.4]
    subset_one, subset_two = split_by_label(batch_inputs, batch_labels)
    for source, target in [(subset_one, subset_two),
                           (subset_two, subset_one)]:
        # Gradient of the source subset  [eq. 10]
        grad = gradient(loss(source, lora), lora)
        # Probe along the descent direction  [eq. 11]
        # (code normalises per tensor, see Paper vs code)
        probe = -rho * grad / (squared_norm(grad) + damping)
        # Target loss at the probed and the unprobed weights  [eq. 12]
        probed_loss = loss(target, lora + probe)
        clean_loss = loss(target, lora)
        # Mixed objective  [eq. 13]
        objective = beta * probed_loss + (1 - beta) * clean_loss
        # One SGD step on the mixed objective  [eq. 14, 15]
        lora = lora - lr * gradient(objective, lora)
    return lora

Cross-task view. The authors compare TSP with SAM. SAM perturbs the same task it trains on, while TSP perturbs with one subset and evaluates on the other §5.3 tab. 5 fig. 2.

Hyperparameters

Values in the table come from the paper unless marked "code". The paper does not state several of them; the code is the only source for those values.

NameValueWhereNotes
BackboneViT-B/16, ImageNet-21K pretrained§5 network.pyBackbone frozen; only LoRA and current-task classifier train
LoRA rank r32 (main); 4, 8, 16, 32, 64 studiedapp. C.2 tab. 8 imgr10.jsonRank 32 chosen as trade-off app. C.2
LoRA placementKey and value projections of every blockattention.py attention.pyPaper says "each ViT layer" §4.2; query is untouched in code
OptimiserSGD§5Momentum 0.9 and weight decay 0 are code only m2le.py imgr10.json
ScheduleCosine annealing§5 m2le.pyRestarted per task in code m2le.py
Epochs per task20§5 imgr10.json imgr10.json
Batch size48§5 imgr10.json
Learning rate0.02 (ImageNet-R), 0.01 (other datasets)§5 imgr10.jsonCode uses 0.02 for all tasks on ImageNet-R imgr10.json
\lambda (HIM)5, all datasets§5 eq. 9 imgr10.jsonNamed alpha in code
\rho (TSP probe)0.5 (ImageNet-R; also CIFAR100, CUB, DomainNet); 0.3 (ImageNet-A)§5 imgr10.jsonCode multiplies \rho by a fresh uniform draw each step m2le.py
\beta (TSP mix)0.8 (ImageNet-R; also others); 0.6 (ImageNet-A)§5 imgr10.json
Damping \varepsilon (HIM)not stated§4.3 eq. 9Code default 10^{-9} m2le.py
Probe constant (TSP)not statedeq. 11Code adds 10^{-12} to each tensor norm m2le.py
Classifier lossnot stated in paper—Code: cosine logits with CosFace margin, s=20, m=0.1 network.py m2le.py imgr10.json
LoRA initialisationnot stated—Code: \mathbf{A} Kaiming-uniform, \mathbf{B}=0 attention.py
Tasks and classes10 tasks, 20 classes each (ImageNet-R); also 5, 20 tasks, DomainNet 10 tasks§5 tab. 2Code: init_cls and increment 20 imgr10.json
Seeds3 seedstab. 1 imgr10.jsonSeeds 1993, 1996, 1997 in code
HardwareOne NVIDIA RTX 3090§5
Fisher dataCurrent task training data, after training§4.3 alg. 1Code: all batches, model in eval mode m2le.py

Results

Author-reported. Setup: ViT-B/16 (ImageNet-21K), rehearsal-free unless marked "FR" (feature replay), 3 seeds, 10 incremental tasks unless stated §5. Each cell is last accuracy \mathcal{A}_{\mathrm{Last}} and average accuracy \mathcal{A}_{\mathrm{Avg}} with standard deviation. Best and second-best markers from the source are dropped.

Table 1: 10 incremental tasks tab. 1

MethodVenueFRImageNet-R LastImageNet-R AvgImageNet-A LastImageNet-A AvgCIFAR100 LastCIFAR100 AvgCUB LastCUB Avg
L2PCVPR'22no72.34±0.1777.36±0.6444.04±0.9351.24±2.2684.06±0.8888.26±1.3467.02±1.9070.62±1.60
CODAPromptCVPR'23no73.31±0.5078.47±0.5352.08±0.1263.92±0.1283.21±3.3987.71±3.1777.23±1.1281.90±0.85
InfLoRACVPR'24no74.75±0.6480.67±0.5549.20±1.1260.92±0.6186.75±0.3591.72±0.1570.82±0.2381.39±0.14
PLANICCV'25no75.25±0.4280.41±0.56----87.54±0.3192.21±0.35----
SEMACVPR'25no78.0083.5653.3264.5386.9891.37----
SD-LoRAICLR'25no77.34±0.3582.04±0.2455.96±0.7364.95±1.6388.01±0.3192.54±0.1877.48±0.2085.59±0.44
BiLoRACVPR'25no77.95±0.1481.52±0.26----87.46±0.7692.50±0.62----
LoRA-P&MNIPS'25no79.95±0.1885.29±0.9356.57±0.7865.35±1.8188.45±0.3592.89±1.1378.29±0.5083.39±0.61
EWC-LoRAICLR'26no72.86±0.7978.95±0.8659.89±0.2668.33±0.6787.91±0.5792.27±0.39----
M2LE (this paper)--no81.34±0.1686.53±0.5162.45±1.3471.02±1.8289.97±0.0493.31±0.9581.40±0.2189.23±0.48
SLCAICCV'23yes79.35±0.2883.29±0.4661.05±0.6368.88±2.3191.26±0.3794.29±0.9284.68±0.0990.77±0.79
SSIATCVPR'24yes79.38±0.5983.63±0.4362.43±1.6370.83±1.6391.35±0.2694.35±0.6088.75±0.3893.00±0.90
MOSAAAI'25yes77.68±0.4182.06±0.5354.75±1.0963.32±2.3891.53±0.3594.21±0.9189.97±0.3293.43±0.60
MACILICML'25yes81.88±0.0785.95±0.2764.14±0.5871.45±1.3591.94±0.1794.43±0.7990.52±0.1393.93±0.47
SAEFCVPR'26yes80.0284.5464.9173.3990.6094.53----
M2LE + CA--yes82.68±0.3087.05±0.5165.96±0.9074.02±0.8591.86±0.2194.62±0.9090.35±0.1693.92±0.38

Against LoRA-P&M on last accuracy, the 10-task gains are 1.39 (ImageNet-R), 5.88 (ImageNet-A), 1.52 (CIFAR100) and 3.11 (CUB) points, computed from the table values §5.1 tab. 1. The largest margin is on ImageNet-A.

Table 2: other task settings and DomainNet tab. 2

MethodVenueFR5S-ImageNet-R LastAvg20S-ImageNet-R LastAvg20S-ImageNet-A LastAvg10S-DomainNet LastAvg
L2PCVPR'22no70.83±0.5878.34±0.4769.64±0.4275.28±0.5740.48±1.7849.62±1.4681.17±0.8387.43±0.95
CODAPromptCVPR'23no74.91±0.3379.25±0.5369.96±0.5075.34±0.8544.62±1.9254.86±0.5080.04±0.7986.27±0.82
InfLoRACVPR'24no77.52±0.3782.01±0.1271.01±0.4577.28±0.4546.81±2.3057.13±0.9581.45±0.6887.55±0.57
SEMACVPR'25no79.7884.7574.5381.75--------
SD-LoRAICLR'25no79.15±0.2083.01±0.4275.26±0.3780.22±0.7249.12±1.7859.63±1.5781.01±0.4286.85±0.40
BiLoRACVPR'25no78.32±0.3783.31±0.5272.41±0.6579.28±0.76----76.56±0.4183.20±0.35
LoRA-P&MNIPS'25no81.47±0.5685.96±0.5276.37±0.0982.77±0.7152.27±1.8462.15±1.3384.35±0.4889.43±0.27
EWC-LoRAICLR'26no76.36±0.2181.43±0.1370.18±1.0677.06±0.54--------
M2LE--no82.67±0.1887.31±0.3578.62±0.4384.62±0.2653.87±1.4563.69±1.8784.78±0.1389.74±0.44
SLCAICCV'23yes81.01±0.1184.18±0.2874.63±1.5579.92±1.2936.69±21.3156.35±7.0984.35±0.7288.56±0.21
SSIATCVPR'24yes80.52±0.0784.25±0.3175.67±0.1482.30±0.3659.16±1.0368.45±1.9285.11±0.5689.80±0.34
MOSAAAI'25yes78.76±0.1782.37±0.2675.16±0.5980.53±0.70--------
MACILICML'25yes83.37±0.2687.00±0.3579.43±0.3484.34±0.3259.36±1.2768.03±2.0086.62±0.3590.88±0.20
M2LE + CA--yes82.74±0.2987.25±0.2881.12±0.1486.12±0.3364.25±0.5672.54±0.9287.61±0.1391.51±0.18

Against LoRA-P&M in the same table, the paper reports last-accuracy gains of 0.43 to 2.25 points across these settings §5.1 tab. 2.

Ablation (ImageNet-R and ImageNet-A, 10 and 20 tasks) tab. 3

IdxHIMTSP10S-IN-R LastAvg10S-IN-A LastAvg20S-IN-R LastAvg20S-IN-A LastAvg
1----73.05±0.3580.40±0.6052.56±2.0863.36±1.3266.14±0.2073.72±0.2546.06±2.1459.43±1.77
2yes--80.36±0.4085.78±0.4461.35±1.0670.23±2.0478.04±0.3584.31±0.5452.80±1.4662.64±2.14
3--yes74.20±0.9180.67±0.4756.57±2.3166.26±1.4669.17±0.1077.16±0.3949.50±0.7260.45±1.52
4yesyes81.34±0.1686.53±0.5162.45±1.3471.02±1.8278.62±0.4384.62±0.2653.87±1.4563.69±1.87

Merge rule comparison, 10 tasks tab. 4

Strategy10S-IN-R LastAvg10S-IN-A LastAvg
Seq-FT (no merge)73.05±0.3580.40±0.6052.56±2.0863.36±1.32
EMA-style79.14±0.1085.08±0.4159.64±1.2269.52±1.82
LoRA-P&M78.89±0.4484.99±0.4559.93±1.5769.87±2.05
HIM-last (only task t-1 in old term)76.23±0.7683.81±0.6158.17±2.3868.84±1.93
HIM (this paper)80.36±0.4085.78±0.4461.35±1.0670.23±2.04

Perturbation strategy, 10 tasks tab. 5

Strategy10S-IN-R LastAvg10S-IN-A LastAvg
w/o TSP80.36±0.4085.78±0.4461.35±1.0670.23±2.04
SAM79.55±0.1784.92±0.4460.46±1.5169.00±1.98
TSP (this paper)81.34±0.1686.53±0.5162.45±1.3471.02±1.82

Task-by-task forgetting tab. 6 (full table in source; final-stage values only here)

DatasetSettingFinal accuracy (task 9)Final forgetting
ImageNet-Rw/o TSP80.36±0.407.76±0.38
ImageNet-Rw/ TSP81.34±0.165.77±0.64
ImageNet-Aw/o TSP61.35±1.0614.01±0.90
ImageNet-Aw/ TSP62.45±1.3410.04±1.29

Overhead on 10S-ImageNet-A tab. 7

MethodPeak device memory, MiBSeconds per batchAdded storage, MBLast acc.
Seq-FT45600.120.052.56±2.08
TSP only28810.340.056.57±2.31
HIM only45600.1254.061.35±1.06
HIM with SAM45680.2654.060.46±1.51
HIM with TSP28810.3454.062.45±1.34

In this setting the paper reports 3.8 seconds per task spent on Fisher estimation §5.3. It does not explain why memory is lower with TSP.

LoRA rank, 20S-ImageNet-R tab. 8

MethodVenueExtra memoryTrainable paramsTrain GFLOPsExtra paramsInference GFLOPsLastAvg
CODACVPR-2315.4MB3.84M (4.46%)35.173.84M (4.46%)35.1769.96±0.5075.34±0.85
SLCAICCV-23450.6MB86M (100%)17.600.00M17.6074.63±1.5579.92±1.29
SSIATCVPR-24450.6MB1.19M (1.38%)17.811.19M (1.38%)17.8175.67±0.2482.30±0.36
InfLoRACVPR-2411.3MB0.37M (0.43%)17.670.00M17.6771.01±0.4577.28±0.45
VPT-NSPNIPS-2428.4MB0.09M (0.10%)17.720.09M (0.10%)17.7275.69±0.6181.87±0.59
SD-LoRAICLR-2514.1MB0.37M (0.43%)20.510.00M17.7275.26±0.3780.22±0.72
VPT-CPGAAAI-2528.4MB1.66M (1.93%)17.721.66M (1.93%)17.7275.33±0.3780.22±0.72
M2LE r=4--54.0MB0.15M (0.17%)17.590.00M17.6076.92±0.2083.28±0.25
M2LE r=8--54.0MB0.29M (0.34%)17.620.00M17.6077.32±0.5583.68±0.51
M2LE r=16--54.0MB0.59M (0.69%)17.680.00M17.6078.01±0.1384.07±0.43
M2LE r=32--54.0MB1.18M (1.37%)17.800.00M17.6078.62±0.4384.62±0.26
M2LE r=64--54.0MB2.36M (2.74%)18.030.00M17.6078.20±0.3084.66±0.70

The VPT-CPG average is printed identically to SD-LoRA's in the source table tab. 8; this card does not check it further.

HIM variants that keep only recent tasks in the old term tab. 9 (Table 9 in source)

Variant10S-IN-R LastAvg10S-IN-A LastAvg
HIM-last (k=1)76.23±0.7683.81±0.6158.17±2.3868.84±1.93
k=378.88±0.6285.37±0.5060.81±1.1069.97±2.28
k=580.04±0.3585.73±0.4760.92±0.9770.14±2.48
k=780.21±0.4585.77±0.4561.16±1.1570.17±2.42
HIM full80.36±0.4085.78±0.4461.35±1.0670.23±2.04

Accumulated versus recomputed Fisher, 10S-ImageNet-R, one seed tab. 10

MetricT0T1T2T3T4T5T6T7T8T9
Similarity S_t--1.0000.9880.9860.9860.9870.9880.9870.9870.986
HIM Last acc.96.6692.8189.4187.3984.6983.8483.1882.6281.6880.67
Recomputed Last acc.96.6692.8189.3687.1184.7683.7083.1582.7081.6880.67

CLIP backbone (LAION-400M, no replay) tab. 11 tab. 12

The source marks best and second-best values; stds are omitted here. Columns are Last / Avg per setting.

MethodIN-R B0 Inc10IN-R B50 Inc10CIFAR100 B0 Inc10CIFAR100 B50 Inc10CUB B0 Inc20CUB B100 Inc20
CoOp37.52 / 60.7339.77 / 54.2024.24 / 47.0024.12 / 41.238.57 / 27.6110.14 / 24.03
ZS-CLIP77.17 / 83.3777.17 / 79.5771.38 / 81.8171.38 / 76.4963.06 / 74.3863.06 / 67.96
L2P66.52 / 75.9766.77 / 72.8273.03 / 82.7473.61 / 81.1457.93 / 70.8766.12 / 75.64
DualPrompt66.65 / 76.2167.58 / 73.2272.44 / 81.6372.57 / 80.1257.46 / 69.8964.84 / 71.40
CODA-Prompt68.95 / 77.6968.05 / 73.7173.43 / 82.4371.58 / 78.6962.98 / 73.1262.21 / 73.95
SimpleCIL74.48 / 81.0674.48 / 76.8476.63 / 84.1576.63 / 80.2077.52 / 83.8177.52 / 79.75
RAPF79.62 / 86.2879.21 / 84.0379.04 / 86.1978.25 / 82.3562.77 / 79.0962.93 / 72.82
PROOF (reproduced)77.25 / 82.6977.03 / 80.5676.29 / 84.8875.93 / 81.8572.60 / 79.9471.80 / 76.43
M2LE82.05 / 87.1682.65 / 85.2480.52 / 87.8883.12 / 87.8080.03 / 86.9280.41 / 84.25
MethodAircraft B0 Inc10Aircraft B50 Inc10Cars B0 Inc10Cars B50 Inc10UCF B0 Inc20UCF B100 Inc20
CoOp7.14 / 14.547.77 / 13.0521.65 / 36.4620.87 / 37.4033.46 / 47.8524.74 / 42.02
ZS-CLIP17.22 / 26.6617.22 / 21.7076.37 / 82.6076.37 / 78.3267.64 / 75.5067.64 / 71.44
L2P28.29 / 47.1932.13 / 44.0761.82 / 76.6365.64 / 76.3776.43 / 86.3476.62 / 83.95
DualPrompt25.83 / 44.3033.57 / 46.0762.94 / 76.2667.55 / 76.8875.82 / 85.2176.35 / 84.31
CODA-Prompt27.69 / 45.9832.28 / 45.1466.47 / 80.2164.19 / 75.0680.14 / 87.7675.03 / 83.04
SimpleCIL48.09 / 59.2448.09 / 53.0586.85 / 92.0486.85 / 88.9685.68 / 90.4485.68 / 88.12
RAPF23.61 / 50.3825.44 / 40.4762.85 / 82.8963.19 / 75.8780.33 / 92.2881.55 / 90.31
PROOF (reproduced)39.03 / 47.3345.36 / 54.3584.06 / 86.8782.66 / 88.6881.47 / 87.0481.85 / 86.15
M2LE60.43 / 70.6363.85 / 69.5090.57 / 94.3990.91 / 92.4090.53 / 94.3891.28 / 94.81

The appendix states that M2LE is best in every CLIP setting app. C.6. In the tables above, M2LE has the highest last accuracy in all twelve columns. The closest baselines in last accuracy are RAPF or SimpleCIL, depending on the column.

Paper vs code

Official repository: github.com/bbdeeer/M2LE_NeurIPS2026, linked from the paper's abstract, cloned at commit 304a4039cb1beba563e8abf3bb9a06e235279ff1. The repository has no LICENSE file, so the code is not licensed for reuse. We quote no code and cite only paths and lines.

The repository contains one configuration (ImageNet-R, 10 tasks) and says that more configurations will follow README.md. Hyperparameters for ImageNet-A, CIFAR100, CUB, DomainNet and the CLIP experiments are therefore not checkable.

Differences found on reading methods/m2le.py and models/attention.py:

  • TSP probe normalisation. Paper: probe is -\rho\,g/(\|g\|_2^2+\varepsilon) over the gradient of the source subset eq. 11. Code: normalises each parameter tensor's gradient by its plain norm, not the squared norm, and multiplies \rho by a fresh uniform draw each step, so the probe norm per tensor is at most \rho m2le.py m2le.py.
  • Probe target. Paper writes the probe as added to the product \mathbf{B}_t\mathbf{A}_t eq. 12. Code perturbs the two factors \mathbf{A}_t and \mathbf{B}_t separately, and computes the gradient with respect to them m2le.py m2le.py.
  • Fisher estimator. Paper defines F_t as the expectation over samples of squared per-sample gradients §4.3. Code takes the gradient of the batch-mean loss, squares it, and scales it by batch size divided by the number of batches m2le.py m2le.py m2le.py. We read these as equal only if per-sample gradients in a batch are uncorrelated; this is our reading, not the authors'.
  • First task. Paper's merge rule eq. 9 at t=1 gives zero for elements with zero Fisher. Code skips the rule at task 0 and keeps the whole update m2le.py.
  • Classifier and loss. Paper does not state the loss. Code uses cosine logits with a CosFace margin (s=20, m=0.1) for both training and Fisher network.py m2le.py imgr10.json.
  • LoRA placement. Paper describes one LoRA per ViT layer §4.2. Code attaches LoRA to the key and value projections of each block, with separate adapters for each task attention.py attention.py. The 54.0 MB Fisher storage in the paper tab. 7 matches 24 matrices of 768\times768 in float32, that is key and value in 12 blocks, but the paper does not say so.
  • Optimiser details. Paper says SGD. Code uses momentum 0.9 and restarts the cosine schedule for every task m2le.py m2le.py m2le.py.
  • Consistent with paper: \lambda=5 (alpha), \rho=0.5, \beta=0.8, batch 48, 20 epochs, learning rate 0.02, rank 32, three seeds imgr10.json §5.
  • Paper vs code, merge state. Code keeps the merged weight as a separate buffer and zeroes the task-t term at inference attention.py. This matches \mathbf{W}_t=\mathbf{W}_{t-1}+\Delta\mathbf{W}^*_t §4.2.

Pitfalls

  • The probe is the most sensitive step. The paper divides \rho by the squared norm of the gradient and does not say whether that norm is global or per tensor. The code divides by an unsquared per-tensor norm and multiplies \rho by a random [0,1) factor each step eq. 11 m2le.py m2le.py. These give different perturbation sizes, so state which one you use.
  • Cost per batch. The code runs one optimiser step per direction, so two per batch. Each direction does three forward passes: source, clean target and probed target m2le.py. Training time rises from 0.12 s to 0.34 s per batch in the paper's setting tab. 7 §5.3.
  • Fallback path. If a batch contains a single label, the code falls back to plain training, with no perturbation m2le.py.
  • The damping constants are unknown. The paper gives no value for \varepsilon in eq. 9 or for the probe constant in eq. 11. The code uses 10^{-9} and 10^{-12}. Because the code's Fisher is scaled by batch size over batch count, the damping's relative size may change with batch size and dataset (our reading).
  • Zero-Fisher elements. For t>1, an entry whose current-task Fisher is zero gets a merge weight of zero, so that element of the update is dropped from the merge eq. 9 m2le.py. Check whether this matters for your data.
  • Learning rate reuse. The code restarts the cosine schedule at the same 0.02 for every task. The paper says "initial learning rate" without saying whether later tasks restart §5.
  • Only one configuration is released. Hyperparameters for the other datasets are reported in the paper only, and cannot be run from the repository. The ImageNet-A values \rho=0.3 and \beta=0.6 are in the paper §5.
  • Storage and cost. HIM stores one accumulated Fisher per adapted matrix. The paper reports 54.0 MB, independent of task count and of rank tab. 7 app. C.2, and 3.8 seconds per task spent on Fisher estimation §5.3.
  • Historical Fisher reuse. The accumulated Fisher is close to a recomputed Fisher (similarity at least 0.986 after task 1), so the paper argues old data are not needed app. C.4 tab. 10. This is one seed, on ImageNet-R only.
  • Hyperparameter sensitivity is stated in text only. The appendix says \lambda is stable for 1\le\lambda\le10, \rho for 0.1\le\rho\le1.0 and \beta for 0.2\le\beta\le1.0 app. C.5. The sweep plots are fig. 6 and fig. 7; their values are not transcribed here.
  • Limitations stated by the authors. Benefit depends on the backbone being suited to low-rank updates; the paper flags safety-sensitive uses such as medical imaging app. D app. E.

Lineage

  • Builds on: LoRA-based merging with a shared coefficient, namely LoRA-P&M §2 tab. 4, and the Fisher-based EWC penalty §2. Sharpness-aware minimisation (SAM) is the comparison point for TSP §4.4 tab. 5.
  • Compared against: L2P, CODA-Prompt, InfLoRA, PLAN, SEMA, SD-LoRA, BiLoRA, EWC-LoRA, SLCA, SSIAT, MOS, MACIL, SAEF, and on CLIP, RAPF, PROOF, SimpleCIL tab. 1 tab. 2 app. C.6.
  • Superseded by: not known.

Source and license

Adapted from “Modulating Merging Strengths via Joint Loss Estimation for LoRA-based Continual Learning” by Kun Gu, De Cheng, Zhipeng Xu, Lingfeng He, Di Xu, Nannan Wang (arXiv:2611.04931), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/bbdeeer/M2LE_NeurIPS2026 at commit 304a4039cb1b (none (no LICENSE file in repository)). Any quoted code is shown under that license.