SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning

In plain words

Teaching a multimodal chatbot new skills one after another often breaks the skills it already had. This paper studies a design where the model has many small add-on modules, called experts, and a router that decides which experts handle each input. The authors see two kinds of drift. The router starts sending old kinds of inputs to different experts. And the experts themselves get overwritten by new tasks, so they stop doing their old job even when the router is fixed.

The method, SAME, addresses both. It lets the router change only in the directions of the input space that matter for the current task. It scales expert updates so they change little along directions that earlier tasks relied on. And during each task it freezes the experts that the current task barely uses but earlier tasks depended on. It does not store old training data. The authors also build a longer test sequence, TriGap, with ten tasks in a row, and report better average accuracy than the baselines on it.

TL;DR

• A mixture of LoRA experts in the feed-forward layers of the language model of LLaVA-v1.5-7B, trained one task at a time without replay. It adds spectral-aware router updates, curvature-aware scaling of expert updates and per-task freezing of experts §4 §5.1.
• Author-reported average accuracy: 66.82 on CoIN, 67.12 on UCIT and 46.53 on the new TriGap benchmark. Each is the highest in its table and above the MoELoRA baseline tab. 2 tab. 3 tab. 1.
• The released code differs from the paper's equations and hyperparameters in several places. Routing is computed once per batch, each expert has rank r // E, the experts' products are mixed in a different way, and task-level resets are never called. Read Paper vs code before implementing.
• Licences: paper CC BY 4.0. Code MIT (LAMDA-CL/Prism, commit 9aae984).

Problem

• Multimodal LLMs must learn new instruction tasks in sequence and keep old ones. Without special care they forget §1.
• Mixture-of-experts with LoRA experts is a common remedy, and the authors cite it as still degrading on earlier tasks §1 §2.
• The authors diagnose two sources. Router drift: inputs from Task 1 are routed to different experts as later tasks arrive fig. 1 §1. Expert drift: when the router is re-fitted to Task 1 on frozen expert snapshots, accuracy still falls with later tasks fig. 1 §1.
• Setting: rehearsal-free continual instruction tuning, so no stored old samples [abstract] §1.
• Assumptions: the LoRA experts sit only in the feed-forward (FFN) layers of the language model. The vision encoder and the projector are frozen §3 §5.1.
• Benchmarks: CoIN has eight tasks, UCIT has six, and TriGap (introduced here) has ten §5.1.

Method

The base model stays frozen. Each targeted FFN linear layer of the language model gets a set of LoRA experts and a router §3. The paper's Equation 3 gives the layer output as the base output plus the router-weighted sum of each expert's low-rank update. SAME adds three mechanisms on top of that.

1. Router covariance. Keep a running, uncentered covariance C of the router's input over all tasks. Update it with a count-weighted average, one task at a time §4.1 eq. 4. Storing the full d x d matrix is too costly, so keep only the leading directions whose cumulative energy reaches the threshold δ §4.1 eq. 5 eq. 6.
2. Spectral-aware router update. Split the router gradient into two parts. The first lies in the leading subspace V_par and is rescaled by a sliding-window factor §4.1 eq. 7 eq. 8 eq. 9. The second lies in the complementary subspace V_perp and is not rescaled §4.1 eq. 10. The router update is their sum eq. 12. The authors argue that inputs of earlier tasks lie close to the null space of V_perp, so their routing is preserved §4.1 eq. 11 app. B.
3. Curvature-aware expert scaling. For each expert, precondition its Euclidean gradient by an approximate inverse of the previous task's covariance C^(t-1). The inverse uses a damped low-rank form: the retained subspace gets 1/(sigma + mu), and the complement gets 1/mu §4.2 eq. 13 eq. 14 eq. 15 eq. 16. The derivation with lambda_t = 1 / (2 eta_t) is in app. C.
4. Adaptive expert activation. During task t, track two per-expert signals: utilisation U(i), the mean routing weight on this task eq. 17, and historical importance F(i), the routing-weighted input energy eq. 18. Normalise both to [0, 1] within the layer and set Score(i) = U~(i) - F~pre(i) eq. 19. Freeze the experts with score below tau_score for the current task only. They are active again at the next task and at inference §4.3 alg. 1 alg. 2.

Algorithm 1 alg. 1 runs per mini-batch of each task, in this order:

1. Route the batch and update C^t with Equation 4. Compute V_par and V_perp (Equations 5 and 6).
2. Update U(i) and F_cur(i) (Equations 17 and 18), compute the score (Equation 19), and freeze experts with score below tau_score.
3. Compute the router gradient. Form the update in V_par (Equations 7 and 9) and in V_perp (Equation 10). Combine them (Equation 12) and apply.
4. Precondition the expert gradients with the approximate inverse of C^(t-1) (Equations 15 and 16). Update only the unfrozen experts.

At the end of each task, F_pre(i) takes the value of F_cur(i) for every expert. At the start of training, C^0 = 0, alpha_0 = 0 and F_pre(i) = 0. At the start of each task, n = 0 and U(i) = 0, and F_cur(i) starts from F_pre(i) alg. 1.

Algorithm 2 alg. 2 is inference. It routes the input with the learned router, selects the top-k experts, and sums their outputs with the routing weights. The paper does not state the value of k. No freezing is applied at inference alg. 2 §4.3.

The equations below are reproduced in the paper's notation. The paper writes A_i as in x r; this card uses the usual r x in layout.

Layer output, one token, the paper's Eq. 3 eq. 3:

\mathbf{h} = \mathbf{W}_0 \mathbf{x} + \sum_{i=1}^{n} \omega_i \mathbf{B}_i \mathbf{A}_i \mathbf{x}, \qquad \omega_i = \mathrm{Softmax}(\mathbf{W}_G \mathbf{x})_i eq. 3

Covariance update, per task eq. 4:

\mathbf{C}^t = \frac{\alpha_{t-1} \mathbf{C}^{t-1} + n_t \hat{\mathbf{C}}^t}{\alpha_t}, \qquad \alpha_t = \alpha_{t-1} + n_t eq. 4

Projected router update, leading subspace and scaling eq. 7 eq. 9:

\Delta \mathbf{W}_{\parallel}^t = \Delta \mathbf{W}_G^t \mathbf{V}_{\parallel} \, g(\mathbf{\Sigma}) \, \mathbf{V}_{\parallel}^\top eq. 9

Scaling function eq. 8:

g(\mathbf{\Sigma}) = \mathrm{diag}(\alpha_1 \sigma_1, \dots, \alpha_r \sigma_r), \qquad \alpha_i = 1 / \hat{\sigma}_i eq. 8

Combined router update eq. 12:

\Delta \mathbf{W}_G^t = \Delta \mathbf{W}_{\parallel}^t + \Delta \mathbf{W}_{\perp}^t eq. 12

Expert update, Riemannian form eq. 15:

\Delta \mathbf{W}_i = -\eta \, \nabla_{\mathbf{W}_i} \mathcal{L} \, (\mathbf{C}^{t-1})^{-1} eq. 15

Damped low-rank inverse eq. 16:

(\mathbf{C}^{t-1})^{-1} \approx \mathbf{V}_k (\mathbf{\Sigma}_k + \mu \mathbf{I})^{-1} \mathbf{V}_k^\top + \frac{1}{\mu} (\mathbf{I} - \mathbf{V}_k \mathbf{V}_k^\top) eq. 16

Utilisation, running average eq. 17:

\mathcal{U}(i) \leftarrow \frac{n \, \mathcal{U}(i) + \sum_{\mathbf{x} \in \mathcal{B}} \omega_i(\mathbf{x})}{n + |\mathcal{B}|}, \qquad n \leftarrow n + |\mathcal{B}| eq. 17

Historical importance proxy, running average eq. 18:

\mathcal{F}^{\mathrm{cur}}(i) \leftarrow \frac{n \, \mathcal{F}^{\mathrm{cur}}(i) + \sum_{\mathbf{x} \in \mathcal{B}} \omega_i(\mathbf{x}) \|\mathbf{x}\|^2}{n + |\mathcal{B}|}, \qquad n \leftarrow n + |\mathcal{B}| eq. 18

Freezing score eq. 19:

\mathrm{Score}(i) = \tilde{\mathcal{U}}(i) - \tilde{\mathcal{F}}^{\mathrm{pre}}(i) eq. 19

The remaining main-text equations follow. Equation 1 is the autoregressive likelihood, and Equation 2 is the MCIT objective over all seen tasks §3.

p_\theta(\mathbf{y} \mid \mathbf{z}) = \prod_{j=1}^{L} p_\theta(y_j \mid \mathbf{z}, \mathbf{y}_{<j}) eq. 1

\theta_t^{*} = \arg\min_{\theta} \mathbb{E}_{(\mathbf{v},\mathbf{q},\mathbf{y}) \sim \mathcal{D}_{\le t}} \Big[ -\sum_{j=1}^{L} \log p_\theta(y_j \mid \mathbf{z}, \mathbf{y}_{<j}) \Big] eq. 2

Eigen-decomposition and the subspace split eq. 5 eq. 6:

\mathbf{C}^t = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^\top, \qquad \mathbf{\Sigma} = \mathrm{diag}(\sigma_1 \ge \dots \ge \sigma_d) eq. 5

\mathbf{V}_{\parallel} = \mathbf{V}[:, :k], \qquad \mathbf{V}_{\perp} = \mathbf{V}[:, k:d] eq. 6

Leading-subspace update before scaling eq. 7:

\Delta \mathbf{W}_{\parallel}^t = \Delta \mathbf{W}_G^t \mathbf{V}_{\parallel} \mathbf{V}_{\parallel}^\top eq. 7

Null-space update and its effect on old inputs eq. 10 eq. 11:

\Delta \mathbf{W}_{\perp}^t = \Delta \mathbf{W}_G^t \mathbf{V}_{\perp} \mathbf{V}_{\perp}^\top eq. 10

\Delta \mathbf{W}_{\perp}^t \mathbf{X}^{\mathrm{old}} = \Delta \mathbf{W}_G^t \mathbf{V}_{\perp} \mathbf{V}_{\perp}^\top \mathbf{X}^{\mathrm{old}} \approx \mathbf{0} eq. 11

Degradation on historical inputs, and the soft-margin objective with tolerance epsilon and weight lambda eq. 13 eq. 14:

\Delta_{\mathrm{degrad}} \triangleq \mathbb{E}_{\mathbf{x} \sim \mathcal{D}_{<t}} \big[ \|\Delta \mathbf{W}_i \mathbf{x}\|^2 \big] = \mathrm{tr}\big( \Delta \mathbf{W}_i \mathbf{C}^{t-1} \Delta \mathbf{W}_i^\top \big) eq. 13

\min_{\Delta \mathbf{W}_i} \; \mathcal{L}(\Delta \mathbf{W}_i) + \lambda \max(0, \Delta_{\mathrm{degrad}} - \epsilon) eq. 14

Pseudocode of the three mechanisms. The learning rate is applied by the optimizer, not in these functions. Each function follows the paper, and the code deviations are listed under Paper vs code.

# Router covariance of one task, count-weighted  [eq. 4]
# Input:  C (d, d) running covariance, alpha (count so far)
#         x (n, d) router inputs of this task's batches
# Output: updated C and alpha

def update_covariance(C, alpha, x):
    # Sample second moment, uncentered  [§4.1] [app. B]
    C_hat = x.T @ x / n
    # Count-weighted running average  [eq. 4]
    C = (alpha * C + n * C_hat) / (alpha + n)
    alpha = alpha + n
    return C, alpha

# Leading directions by cumulative energy  [§4.1] [eq. 5] [eq. 6]
# Input:  C (d, d) covariance, delta (energy threshold)
# Output: U (d, k) leading directions, S (k,) their singular values

def leading_directions(C, delta):
    # C = U diag(S) V^T; the paper's V_par is U[:, :k]  [eq. 5]
    U, S, _ = svd(C)
    # Energy is built from sigma_i squared  [§4.1]
    energy = S ** 2
    ratio = cumsum(energy) / sum(energy)
    # Smallest k at which the cumulative ratio passes delta  [§4.1]
    k = first index where ratio > delta
    return U[:, :k], S[:k], k

# Spectral-aware router update  [§4.1] [eq. 7] [eq. 8] [eq. 9] [eq. 10]
# Input:  grad (E, d) router gradient; U (d, k); S (k,); k
# Output: router update, same shape as grad

def router_update(grad, U, S, k):
    # Important directions for the new task  [eq. 6]
    V_par = U[:, :k]
    # Remaining directions, which carry old-task knowledge  [eq. 6]
    V_perp = U[:, k:]
    # Sliding-window mean of the leading singular values, window k  [§4.1]
    smooth = window_mean(S[:k], window=k)
    # Scale for direction i: alpha_i * sigma_i, alpha_i = 1 / smooth_i  [eq. 8]
    scale = diag(S[:k] / smooth)
    # Rescaled update in the leading subspace  [eq. 9]
    update_par = grad @ V_par @ scale @ V_par.T
    # Unscaled update in the complementary subspace  [eq. 10]
    update_perp = grad @ V_perp @ V_perp.T
    # Sum of the two parts  [eq. 12]
    return update_par + update_perp

# Curvature-aware scaling of one expert gradient  [§4.2] [eq. 15] [eq. 16]
# Input:  grad (d_out, d_in) Euclidean gradient of one LoRA expert
#         V_k (d_in, k) leading directions of C^(t-1), S_k (k,) their values
#         mu (damping)
# Output: preconditioned gradient (d_out, d_in)

def curvature_scaled_grad(grad, V_k, S_k, mu):
    # Damped inverse inside the retained subspace  [eq. 16]
    inside = V_k @ diag(1 / (S_k + mu)) @ V_k.T
    # Scale 1/mu on the orthogonal complement  [eq. 16]
    outside = (eye(d_in) - V_k @ V_k.T) / mu
    # Right-multiply the gradient by the inverse  [eq. 15]
    return grad @ (inside + outside)

# Adaptive expert activation, one MoE layer  [§4.3] [eq. 17] [eq. 18] [eq. 19]
# Input:  gates (B, E) routing weights of a batch
#         energy (B,) squared input norms of the same tokens
#         state: utilization (E,), importance (E,), count n, tau
# Output: freeze mask (E,), 1 = trainable

def update_freeze_mask(gates, energy, state, tau):
    # Batch size of this step, used in both running averages  [eq. 17]
    batch_size = gates.shape[0]
    # Utilisation: count-weighted running mean  [eq. 17]
    util_sum = state.n * state.utilization + sum(gates)
    state.utilization = util_sum / (state.n + batch_size)
    # Historical importance proxy: routing-weighted energy  [eq. 18]
    weighted = sum(gates * energy)
    imp_sum = state.n * state.importance + weighted
    state.importance = imp_sum / (state.n + batch_size)
    state.n = state.n + batch_size
    # Min-max normalise both signals within the layer  [§4.3]
    score = min_max(state.utilization) - min_max(state.importance)  # [eq. 19]
    # Keep trainable where the score reaches tau  [§4.3]
    return score >= tau

Notes on the pseudocode. The paper defines sigma_i both as singular values of C eq. 5 and as eigenvalues app. B. The first function follows the energy definition of §4.1. The delta, mu and tau values, and the window size, are given under Hyperparameters.

Hyperparameters

Results

Author-reported. Setup: LLaVA-v1.5-7B, CLIP-L/14-336, one epoch per task, 8 GPUs, following the Prism framework §5.1. Metric: A-bar = (1/T) sum_s A(s,T) §5.1. The paper reports no seeds, repeat runs or error bars. Table numbers follow the source order. Bold and underline markers are dropped.

TriGap, ten tasks, baselines reproduced under the same setup tab. 1:

CoIN, eight tasks, baseline numbers taken from the original publications tab. 2:

UCIT, six tasks tab. 3:

Ablation on CoIN, columns are cumulative tab. 4. The "Baseline" column is MoELoRA, "w/ Router" adds spectral-aware routing, "w/ Expert" adds curvature-aware scaling, and "w/ Activation" is the full method:

The VizWiz entry in the ablation (54.53) differs from the CoIN table (54.13). The CoIN table's average of 66.82 is consistent with 54.13, so the ablation entry looks like a typo.

Further results:

• Hyperparameter sensitivity on UCIT app. F. δ in {0.80, 0.85, 0.90, 0.95} gives 66.5, 66.4, 67.4, 66.9, with 0.90 selected. μ in {0.09, 0.9, 9} gives 62.3, 67.4, 61.4, with 0.9 selected. tau_score in {0.075, 0.1, 0.125} gives 65.7, 67.4, 66.3, with 0.1 selected.
• Design choices on UCIT app. H: sliding-window scaling 67.1 vs global mean 63.3. Adaptive freezing 67.1 vs fixed 50% freezing 60.4. Anisotropic scaling 67.12 vs GPM-style hard projection 61.72 and constant subspace scaling 61.83.
• Task order app. G: under two other orders, SAME averages 67.0 and 66.8, and MoELoRA averages 47.2 and 54.3.
• Forgetting on CoIN app. J: average forgetting over the listed tasks is −4.23 for SAME, −6.38 for HiDe-LLaVA and −19.04 for MoELoRA. The table lists seven of the eight CoIN tasks (see Pitfalls).
• Cross-benchmark OOD, trained on UCIT and tested on MMMU app. K: SAME averages 36.35, MoELoRA 34.83, zero-shot 33.75.
• Efficiency app. L: covariance storage falls from 32 MB to about 0.5 MB per layer, with top-64 directions. The full state fits a 139 MB checkpoint at a 0.68% accuracy cost. Inference throughput rises from 1.03 to 1.26 it/s.
• Training time and memory on the sequence §5.3 fig. 5: the authors report an average saving of 32.1 minutes and 2.3 K MiB per GPU per task, with larger savings on Tasks 4 and 8.

Paper vs code

The official repository is https://github.com/LAMDA-CL/Prism, linked in the abstract. The code commit is 9aae98417b548f685b34b7ee02d0e75d9d678837, cloned to raw/code/2602_01990. The LICENSE file is MIT. SAME is registered as --method same. The implementation is in PEFT/tuners/custom/same.py and method/custom/same/integration.py, and the defaults are in config/methods/same.py. The paper is ICML 2026 accepted-paper style. The repository README reports acceptance [README.md].

Each entry below cites both the paper and the code.

Routing and experts

• Routing granularity. The paper gives a per-input gate omega_i(x) eq. 3 eq. 17. The code averages the token-level softmax over all tokens in the forward pass, so every token in a batch receives one shared routing vector same.py. Evaluation does the same, with logits divided by the expert count and a top-2 cut same.py.
• Expert product. The paper sums the products omega_i B_i A_i x eq. 3. The code computes sum_i omega_i B_i (sum_j omega_j A_j x). This contains cross terms omega_i omega_j B_i A_j x same.py same.py.
• Expert rank. The paper gives each expert rank r eq. 3. The code gives each expert rank r // expert_num, because SAMELinearA and SAMELinearB size their experts that way same.py same.py.
• Rank value. The paper says LoRA rank 8 §5.1. The code's per-benchmark defaults are 64 (CoIN), 48 (UCIT) and 80 (TriGap) same.py. The paper gives no rank for the benchmark runs.
• Number of experts. The paper does not state it. The code sets it to the benchmark's task count integration.py config_loader.py.
• Expert computation. Frozen experts get a zero routing weight. Every expert still runs its forward pass, and the code does not change requires_grad for frozen experts same.py. The paper says frozen experts stop forward and backward propagation §4.3. The efficiency gains in §5.3 and app. L cannot come from skipped experts in this code path.
• Current-task expert. The code never freezes expert number min(cur_task, E-1) same.py same.py. The paper does not mention this.

Covariance and router update

• Centering. The paper says the covariance is uncentered §4.1 app. B. The code subtracts the batch mean before computing the covariance same.py.
• Accumulation. The paper defines a count-weighted running average over all seen data eq. 4. The code replaces the stored factors with the current batch's estimate, and refreshes them only when the count is zero or every 20 steps same.py same.py. The count cov_alpha grows, but it is never used to weight the old estimate. Equation 4 is therefore not implemented.
• Estimator. The code uses a single randomized range-finder pass, with no power iteration same.py. The paper cites randomized decomposition only in the storage analysis app. L.
• Singular values. The code's S is the square root of the mean squared projection same.py. That is the square root of an eigenvalue. The paper defines sigma_i as singular values in eq. 5 and as eigenvalues in app. B. The damping mu in the code is added to the square-rooted value.
• Sliding window. The paper's window size is k, the number of retained directions §4.1. The code uses a fixed window of 3 same.py same.py.
• Scale factor. The paper's scale is sigma_i / smooth_i eq. 8 eq. 9, so low-energy directions are updated less. The code computes 1 / smooth, divides by its maximum and clips to [0.8, 1.0] same.py. Low-energy directions therefore get the larger scale, the reverse of the text in §4.1. The code also omits the sigma_i factor.
• Warm-up and clamp. The code skips router rescaling for the first 10 training steps of each layer, and clamps the scale by ±0.2 same.py same.py. Neither appears in the paper.
• Null-space part. The code uses grad - grad V V^T same.py. That matches eq. 10 only if the stored directions span everything outside the retained ones. The stored rank is capped at 64, so the complement is implicit.

Curvature-aware scaling

• Scope. The paper preconditions expert updates eq. 15. The code registers the curvature hook on the loraA experts only same.py. The loraB experts are not preconditioned.
• Task 1. The code returns the gradient unchanged when cur_task == 0 same.py. The paper uses C^0 = 0 alg. 1, and its inverse is undefined at that point. It does not say what to do on Task 1.
• Hidden constants. The derivation in app. C relates lambda_t and epsilon_t to the learning rate. Neither epsilon, lambda nor eta appears in the code's hook. Only the inverse of eq. 16 is implemented same.py.
• Optimiser. The paper does not state the optimiser. The code default is AdamW config_loader.py, and the SAME config does not override it same.py. The paper's update is derived for plain gradient steps. The paper does not analyse what Adam's per-coordinate normalisation does to a preconditioned gradient.

Freezing

• Per-task reset. Algorithm 1 resets n, U(i), and F_cur at each task alg. 1. The code defines reset_for_new_task, which resets the covariance counter and utilisation, but nothing calls it same.py. Utilisation is an exponential average with decay 0.95 that starts at 1/E and is never reset same.py same.py.
• Importance. The paper's F_cur is a count-weighted average, initialised from F_pre at each task eq. 18 alg. 1. The code uses the same 0.95 decay, starts at 1/E, and never resets it same.py same.py. Its carry-over across tasks is therefore implicit.
• Global freeze flag. The code sets all_frozen once every expert except the current one is masked, and never clears it. Later freezing is skipped same.py same.py. The paper does not describe this rule.
• Factor dropped. The paper's AGOP derivation gives a factor d_out app. D. The code uses only ||x||^2 same.py. A constant factor cancels in min-max normalisation, so this should not change the scores.
• Snapshot. At task end the code copies the current covariance into the previous-task buffers integration.py same.py. This matches the role of C^(t-1), but it inherits the accumulation difference above.

Inference and training configuration

• Top-k inference. Algorithm 2 selects top-k experts without giving k alg. 2. The code uses k = 2 same.py.
• Batch sizes. The paper uses batch size 6 for all methods §5.1. The code's per-task values are 12 for CoIN, 8 for UCIT task 0 and 6 for later UCIT tasks, and 12 for TriGap task 0 and 8 for tasks 1 to 9 same.py. The TriGap benchmark file lists 12 for its first tasks TriGap.py TriGap.py. The runner turns these into gradient accumulation run.py. The paper does not describe that mapping.
• Energy threshold comment. A comment in the code says "energy threshold 0.95", while the value used is 0.9 same.py same.py. The comment is stale.
• Threshold rule. The code's k is the first index where the ratio exceeds δ same.py. That matches §4.1. The appendix writes "≥" app. B. The difference is minor.

Pitfalls

• Routing is batch-level in the code. An implementer following eq. 3 would route per token. The released code routes once per forward pass, so outputs depend on which other examples share a batch. This holds at evaluation too, where the default batch size is 1 same.py: a single sequence still shares one routing vector across its tokens. Decide which one to implement and say so.
• The expert product is not Equation 3. The code mixes the A and B experts separately, which gives cross terms. Use eq. 3 only if you intend to match the paper, not the code.
• Rank is split across experts in the code. r // expert_num per expert, so r = 8 with eight experts would give rank 1 per expert. The paper does not say whether this split is intended same.py.
• Freezing does not save compute in the code. Frozen experts are masked to zero routing weight, but all experts still run same.py. The paper's speed and memory numbers §5.3 app. L cannot be reproduced from this path by reading.
• Per-task state is never reset. reset_for_new_task has no call site. all_frozen is never cleared. Utilisation and importance carry over from earlier tasks. Check whether this is the intended behaviour before trusting any per-task claim.
• Equation 4 is not what the code computes. The covariance is replaced at intervals from the current batch, so the router update relies on a short-memory estimate. The paper's claim that old inputs lie near the null space app. B depends on the long-memory form.
• Task-1 behaviour is not specified. C^0 = 0 makes the curvature inverse undefined on Task 1. The code skips the hook on Task 1 same.py. Make this an explicit choice.
• The scale direction is reversed relative to the text. See Paper vs code. Keep the code's clamp of [0.8, 1.0] in mind if you copy it.
• Hyperparameters were chosen on UCIT training data. The paper selects δ, μ and tau_score on a 20% sample of UCIT's training set and then fixes them for every benchmark app. F. The selection data overlaps the evaluated benchmark's distribution.
• The paper reports no seeds or variance. The numbers are single-run figures, as far as the source shows. Treat small gaps, such as 0.2 on task order app. G, with caution.
• Text and table disagree on TriGap. The text says the gain is "particularly clear" on FloodNetVQA §5.2, but MoE-LoRA scores 90.41 there and SAME scores 81.09 tab. 1. SAME also trails MoE-LoRA on CLEVR (54.50 vs 67.00) and CL-MoE on ArxivQA (90.46 vs 93.00) tab. 1.
• TriGap size is given two ways. The text says "over 250k training instances" §5.1. The table in app. E totals 196,313. The text says per-task sizes are "10k--40k" §5.1, but the table includes FloodNetVQA at 5,898 and the appendix says "5.8k to 40k" app. E.
• Baseline provenance differs. The CoIN baselines are "directly adopted from their original publications" tab. 2. The TriGap baselines are "reproduced under the same experimental setup" tab. 1. The source of the UCIT baselines is not stated. Compare CoIN numbers across tables with care.
• The OOD text does not match its table. app. K says SAME is +3.64 over zero-shot on Tech & Engineering. The table gives 33.96 vs 28.16, which is +5.80. The +3.64 is the gap to MoELoRA (33.96 vs 30.32).
• The forgetting table omits a CoIN task. app. J lists seven of the eight CoIN tasks, without OCR-VQA. The MoELoRA average (−19.04) does not equal the mean of the listed columns (−20.41). SAME and HiDe-LLaVA match their listed means.
• Hyperparameters that are unstated in the paper. LoRA alpha, LoRA dropout, expert count, the optimiser, the Task-1 curvature rule, and the number of retained directions are not stated in the paper. The code's values are in Paper vs code.
• Version. The raw metadata records arXiv v3 (updated 2026-08-26) [raw/arxiv metadata]. The fetched LaTeX source is the version this card describes.

Lineage

• Builds on: mixture-of-experts layers [jacobs1991adaptive] and LoRA [hu2022lora] as cited in §1. The MoE-LoRA baseline is MoELoRA from the CoIN benchmark [chen2024coin] §5.1. The gradient-projection comparison is GPM-style hard projection app. H. The AGOP trace used for importance is from §4.3 app. D. The code is built on the Prism framework [tang2026prism] §5.1.
• Compared against: MoELoRA, Continual LLaVA (arXiv:2411.02564), ModalPrompt, SEFE, ProgLoRA, LLaVA-CMoE (arXiv:2503.21227), CL-MoE, HiDe-LLaVA, Replay-LoRA, FT-LoRA and zero-shot inference §5.1 tab. 1 tab. 2 tab. 3.
• Benchmarks: CoIN [chen2024coin], UCIT [guo2025hide] and TriGap (introduced here, §5.1). The paper says TriGap uses datasets absent from LLaVA's training mix §5.1.
• Superseded by: not stated.