Paper · 2026-10-04 · cs.AI, cs.LG
JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)
JIVEAdapter adds to each frozen weight a shared low-rank Joint term plus a per-task Individual term kept orthogonal to it, with rank allocated per pool. On GLUE with DeBERTaV3-base at matched per-task effective rank 8, the author-reported multi-task average is 89.07 (HydraLoRA 88.80, MTL-LoRA 88.42).
Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningMulti-task learningContinual learningparameter-efficient-fine-tuningmulti-task-learninglow-rank-adaptationinterpretability
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
When one model must handle several tasks, a common trick is to add a small set of trainable weights per task. The usual options both waste effort. A separate adapter per task has to relearn what all the tasks share. One adapter shared by everyone blurs each task's own patterns together.
JIVEAdapter splits each adapter's change into two parts: one shared part that every task uses, and one small part per task. It pushes each task's part to point in directions perpendicular to the shared part, so the two stay easy to tell apart and can be inspected on their own. The number of directions each part may use is chosen separately, so the shared part is not crowded out by the many task parts.
Training runs in two stages. First the shared part is learned from a group of tasks, all at once or one task at a time. Then it is frozen, and a new task is served either by borrowing the task part of the most similar earlier task and rescaling it, or by training a small new task part. The base model is never changed, and no mixture-of-experts router is added.
TL;DR
- Each frozen weight receives a shared low-rank Joint update plus one low-rank Individual update per task. The Individual is projected orthogonal to the Joint in the forward pass, and a regulariser keeps the factors near-orthonormal §3.1 eq. 2 eq. 6 eq. 5.
- Rank is allocated separately to the Joint pool and to the per-task Individual pools, by an AdaLoRA-style importance score or a permutation test §3.2.1 app. B.1.
- On GLUE with DeBERTaV3-base at matched per-task effective rank 8, the author-reported multi-task average is 89.07, against 88.80 for HydraLoRA and 88.42 for MTL-LoRA tab. 2. On SuperGLUE the same method averages 82.06, just below HydraLoRA's 82.24 tab. 2.
- Results come from a single fixed seed with no variance reported app. C.3 tab. 5. No official code is available § Reproducibility Statement.
Problem
Single-task low-rank adapters such as LoRA and AdaLoRA learn one update per task. Used per task, they relearn shared structure from scratch and transfer nothing. One adapter used for every task washes out what is specific to each task §1.
Earlier multi-task adapters add a shared piece and per-task pieces, but they combine them multiplicatively. They do not force the shared and task-specific pieces to cover separate variation, and they either hold each task's rank fixed or spread one rank budget over the whole model §1 §2. The paper also targets a growing task pool: when earlier training data is no longer available, a shared structure must be kept without retraining it §4.1.2.
Method
Formulation
The frozen weight is W. For task t the update is ΔW_t = J + I_t, where J is shared by all T tasks and I_t is the Individual of task t eq. 1 §3.1. The adapted output for a sample of task t is
y = x Wᵀ + drop(x) Jᵀ + drop(x) I_tᵀ eq. 2 §3.1
Each structure is a low-rank SVD-style triplet. The Joint and Individual are scaled as
J = α_J/(r_J + ε) · U_J (E_J ⊙ V_J) eq. 3 §3.1
I_t = α_I/(r_I + ε) · U_{I,t} (E_{I,t} ⊙ V_{I,t}) eq. 4 §3.1
The scaling by the rank r is stated to keep the effective magnitude of an update roughly constant as r changes, so that rank and α can be tuned separately §3.1. ε guards against r = 0 §3.1. The Individual factors carry a leading task axis of size T; the Joint factors are single tensors §3.1.
Orthogonality
The training loss is the task loss plus λ·R, with R summed over all layers §3.2.1:
R = Σ_layers ( ‖V_J V_Jᵀ − I‖F + ‖U_Jᵀ U_J − I‖F + Σ_t ( ‖V{I,t} V{I,t}ᵀ − I‖F + ‖U{I,t}ᵀ U_{I,t} − I‖_F ) ) eq. 5 §3.2.1
The Individual is kept orthogonal to the Joint in the forward pass, by projecting its output factor onto the complement of the Joint's column space §3.2.1:
Ũ_{I,t} = (I − U_J U_Jᵀ) U_{I,t}, and I_t uses Ũ_{I,t} in place of U_{I,t} in eq. 4 eq. 6 §3.2.1
The paper states that this projection is exact when U_Jᵀ U_J = I, which the regulariser only encourages eq. 6 §3.2.1.
Forward pass with routing
Alg. 1 gives the forward pass of one layer. Samples in a mixed batch are routed to their own Individual; the Joint applies to all samples. Grouping by distinct task ids in the batch costs one small product per active task rather than per sample §3.2.1.
# JIVEAdapter layer forward [alg. 1]
# Input: inputs (B, ..., d_in), task_ids (B,) or None, frozen_weight
# Output: outputs (B, ..., d_out)
def jive_forward(inputs, task_ids, frozen_weight):
# Frozen base path [alg. 1]
outputs = inputs @ frozen_weight.T
# Shared Joint, applied to every sample [eq. 3] [alg. 1]
joint = joint_scale * U_joint @ (E_joint * V_joint)
outputs = outputs + dropout(inputs) @ joint.T
# No task ids or one task: one Individual, no routing, no projection [alg. 1]
if task_ids is None or num_tasks == 1:
individual = (
individual_scale * U_ind[0] @ (E_ind[0] * V_ind[0])
)
return outputs + dropout(inputs) @ individual.T
# Mixed batch: route each sample to the Individual of its task [alg. 1]
individual_out = zeros_like(outputs)
for task in unique(task_ids):
mask = task_ids == task
# Project the Individual off the Joint column space [eq. 6] [alg. 1]
projected_U = (identity - U_joint @ U_joint.T) @ U_ind[task]
individual = (
individual_scale * projected_U @ (E_ind[task] * V_ind[task])
)
individual_out[mask] = dropout(inputs[mask]) @ individual.T
return outputs + individual_outMixed batches keep the shared Joint stable, because it receives one averaged gradient per step. Alternating per-task batches makes the Joint oscillate toward the task seen last. Accumulating gradients over per-task micro-batches before one optimizer step is offered as a compromise §3.2.1.
Merging
Merging folds ΔW into W so that a layer runs one matrix product. It is enabled only for a single task (T = 1). With several tasks it is disabled, because folding one Individual into W would corrupt the other tasks §3.2.1.
Rank allocation
Two strategies set the ranks, and the Joint pool and the Individual pool are budgeted separately §3.2.1.
- Importance-based (AdaLoRA-style): each singular value gets a score s_i = s(E)i + mean_in s(V){i,:} + mean_out s(U)_{:,i}, with s(p) = Ī ⊙ Ū, where Ī and Ū are exponential moving averages of |p ⊙ ∇p| and of its deviation from Ī eq. 9 eq. 10 app. B.1.1. The pool budget follows a cubic decay (Alg. 4). Every mask interval, the lowest-scoring values across the pool are zeroed to meet the budget (Alg. 5) app. B.1.1.
- Permutation-test (budget-free): run once before training on per-task gradient blocks of the frozen weight, with the classic JIVE alternating fit inside. The Joint rank is the median over modules of each module's Joint rank. The Individual rank is the median over modules of the per-module mean of the per-task Individual ranks. Nothing is pruned during training app. B.1.2 alg. 6.
# Triplet budget schedule for one pool [alg. 4]
# Input: full size N, target tau, step g, warm-ups t_i and t_f, total steps
# Output: active-rank budget b in [tau, N]
def triplet_budget(full_size, target, step, initial_warmup, final_warmup,
total_steps):
# Keep full rank during the initial warm-up [alg. 4]
if step <= initial_warmup:
return full_size
# Hold at the target during the final warm-up [alg. 4]
if step >= total_steps - final_warmup:
return target
# Cubic decay from full size to target in between [alg. 4]
progress = (step - initial_warmup) / (
total_steps - final_warmup - initial_warmup
)
return floor(target + (full_size - target) * (1 - progress) ** 3)Incremental folding of a new task
Given n tasks with a Joint J and Individuals {I_t}, and one incoming task, the Joint is re-estimated over all n+1 tasks and warm-started from the current J. Each Individual is then recovered as the rank-r_I residual orthogonal to the updated Joint §3.2.1 alg. 2.
# Incremental JIVE: fold a new task into the Joint [alg. 2]
# Input: joint, individuals per task, incoming task data, ranks r_J and r_I,
# iterations K, share threshold tau, task weights gamma
# Output: updated joint, incoming individual, refreshed individuals
def incremental_jive(joint, individuals, incoming_data):
# Per-task update matrices built from the current parts [alg. 2]
task_updates = [joint + ind for ind in individuals]
# Incoming Individual, fitted on its data with the Joint in place [alg. 2]
incoming = fit_individual(joint, incoming_data, rank_individual)
task_updates.append(joint + incoming)
# Normalised second-moment matrix per task [alg. 2]
covariances = [u @ u.T / squared_frobenius(u) for u in task_updates]
for iteration in range(num_iterations):
# Shared subspace: top-r_J eigenvectors of the weighted sum [alg. 2]
combined = weighted_sum(covariances, task_weights)
shared_basis = top_eigenvectors(combined, rank_joint)
# Keep only directions every task shares, at or above tau [alg. 2]
shared_basis = keep_shared(shared_basis, covariances, tau)
# Weighted-mean update, projected onto the shared subspace [alg. 2]
mean_update = weighted_mean(task_updates, task_weights)
joint = projector(shared_basis) @ mean_update
# Each Individual: residual off the new Joint, truncated to r_I [alg. 2]
individuals = [
truncate(off_projector(shared_basis) @ u, rank_individual)
for u in task_updates
]
# Stop when the shared subspace stops moving [alg. 2]
if subspace_distance(shared_basis, previous_basis) < epsilon:
break
previous_basis = shared_basis
# Final truncated SVD of the Joint at rank r_J [alg. 2]
joint = truncated_svd(joint, rank_joint)
return joint, individuals[-1], individualsThe growth step that would let the Joint rank increase is written in the source but commented out, so the growth cap κ in the algorithm's inputs is not used in the active version alg. 2.
Structure reuse for a new task
The Joint J is frozen together with W, and a new task T* is served in one of two modes §3.2.2 alg. 3.
- Similarity (2a): build a signature G_{T*} from the top-r_I left singular vectors of the task's gradient averaged over B minibatches at W + J eq. 7 §3.2.2. Pick the held-in task whose signature has the largest overlap, (1/r_I)·‖G_{T*}ᵀ G_t‖²_F eq. 8 §3.2.2. Reuse that task's Individual with its two bases frozen, train only the r_I diagonal scales E_{T*} and a fresh head alg. 3 §3.2.2. Routing uses the Individual alone, not W + J + I, since the shared parts would dominate the score §3.2.2.
- Adapt (2b): freeze W + J and train a fresh Individual of rank r_I alg. 3 §3.2.2.
The shared-energy ratio η = ‖U_J U_Jᵀ Δ̂W‖_F / ‖Δ̂W‖F is a diagnostic. Δ̂W is a reference single-task update when one is available, and otherwise the signature G{T*} §3.2.2 alg. 3.
# Reuse a frozen Joint for a new task T* [alg. 3]
# Input: frozen weight, frozen joint, held-in individuals and signatures,
# target data, rank r_I, mode in {similarity, adapt}
# Output: individual for T*, shared-energy ratio eta
def reuse_joint(frozen_weight, joint, held_in, target_data, mode):
if mode == "similarity":
# Signature: top-r_I left singular vectors of the averaged gradient [eq. 7]
signature = top_left_singular(
mean_gradient(frozen_weight + joint, target_data), rank_individual
)
# Nearest held-in task by subspace overlap [eq. 8] [alg. 3]
nearest = argmax_over_tasks(overlap(signature, held_in.signatures))
# Reuse its Individual bases; fit only per-direction scales [alg. 3]
individual = held_in.individuals[nearest]
scales = fit_scales(frozen_weight + joint, individual, target_data)
else:
# Adapt: train a fresh Individual, Joint and weights frozen [alg. 3]
individual = fit_individual(
frozen_weight + joint, target_data, rank_individual
)
# Diagnostic: share of the reference update inside the Joint [alg. 3]
reference = reference_update_or_signature(target_data)
eta = norm(U_joint @ U_joint.T @ reference) / norm(reference)
return individual, etaPermutation-test steps
For every task and for the shared module, a weight-shaped block X_t is built from the frozen matrix's gradient, averaged over a few batches of that task's loss app. B.1.2. The estimator then runs the following steps, repeated for up to M outer iterations alg. 6.
- Stack the blocks with their current individual parts removed, and permute each block's columns independently B times. Record the largest singular value of each permuted stack alg. 6.
- Set the Joint threshold at the (1 − α) quantile of those maxima. Keep as the Joint rank the count of leading singular values above it, never decreasing alg. 6.
- For each task, take the (1 − α) quantile of the top singular values of its row-permuted residual block, and count the singular values above it as that task's Individual rank alg. 6.
- Refit the Joint and Individuals with the classic JIVE alternating decomposition alg. 6 app. B.1.2 and stop when all ranks are unchanged alg. 6.
Hyperparameters
Shared settings for all runs are given in tab. 5. Values not listed there are stated in the text as cited.
| Name | Value | Where | Notes |
|---|---|---|---|
| Backbone | DeBERTaV3-base | §4 | Adapters on query, key, value, intermediate, layer.output and attention.output projections §4 app. C.3 |
| Adapted matrices | 72 | app. C.3 | Stated as "72 matrices" for the budget formula |
| Learning rate | 5e-4 | tab. 5 | Same for every method |
| Weight decay | 0 | tab. 5 | |
| Batch size | 8 | tab. 5 | |
| Max sequence length | 256 | tab. 5 | Parameter counts in tab. 2 are at sequence length 128 |
| Training steps | 20,000 | tab. 5 app. C.3 | |
| Warm-up steps | 1,000 | tab. 5 | |
| Evaluation interval | 2,000 steps | tab. 5 | |
| Seed | 6 | tab. 5 | One fixed seed for all methods app. C |
| Orthogonality coefficient λ | 0.1 | tab. 5 eq. 5 | Assumed to be the λ of the loss, from the table label |
| Classifier dropout | 0.1 | tab. 5 | Applies to the task heads; the adapter dropout rate is not stated |
| Adapter scaling α (both pools) | 16 | app. C.3 | Stated for JIVE and the controls |
| Initial rank per structure | 12 | tab. 5 app. C.3 | Joint and each Individual start at 12 |
| Target rank, multi-task, per structure | 4 | tab. 5 app. C.3 | Joint 4 + one Individual 4 = effective rank 8 |
| Allocator initial warm-up | 4,000 steps | tab. 5 | |
| Allocator final warm-up | 8,000 steps | tab. 5 | |
| Allocator mask interval | 100 steps | tab. 5 | |
| Optimizer and scheduler | not stated | tab. 5 | Only learning rate, weight decay and warm-up are given |
| Factor initialisation | Zero-initialised factors | §5 | Distribution of U and V is not stated |
| ε in the scaling | not stated | eq. 3 eq. 4 | |
| EMA coefficients β1, β2 | not stated | app. B.1.1 | |
| Incremental τ, K, ε, γ_t | not stated | alg. 2 | |
| Permutation test B, α, M, batches | not stated | alg. 6 | |
| Reuse: signature batches B | not stated | eq. 7 | |
| MTL-LoRA | rank 8, α 16, dropout 0.1, 3 B matrices, B scale 1.0 | tab. 5 | Structured baseline |
| HydraLoRA | rank 8, α 16, dropout 0.1, 3 experts | tab. 5 | Structured baseline |
| Hardware | 1 × NVIDIA A100 80 GB | app. C.1 | |
| Software | PyTorch 2.2, CUDA 12.1, patched Transformers | app. C.1 |
Results
Author-reported. Setup: DeBERTaV3-base, GLUE and SuperGLUE, following the AdaLoRA setting, with each method given the same backbone and parameter budget at matched per-task effective rank §4 app. C.3. The paper reports a single seed and no variance.
Single-task (T = 1) tab. 1
Column metrics: accuracy for MNLI, QNLI, SST-2 and RTE; Matthews correlation for CoLA; accuracy and F1 for QQP and MRPC; Pearson and Spearman for STS-B; F1a and EM for MultiRC; EM and F1 for ReCoRD; accuracy and F1 for CB tab. 1.
GLUE, from tab. 1:
| Method | MNLI | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | Mean | Params (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA | 89.52 | 92.21/89.67 | 93.79 | 94.72 | 70.44 | 81.59 | 89.22/92.14 | 91.73/91.69 | 87.90 | 1.33 |
| AdaLoRA (r̄ = 8, authors' reproduction) | 90.43 | 91.97/89.44 | 94.33 | 95.41 | 71.91 | 87.36 | 90.20/92.91 | 91.58/91.49 | 89.14 | 1.33 |
| JIVEAdapter | 90.55 | 91.89/89.34 | 94.62 | 96.22 | 71.02 | 88.81 | 90.44/93.10 | 91.58/91.97 | 89.44 | 1.33 |
SuperGLUE, from tab. 1:
| Method | BoolQ | CB | RTE | WiC | WSC | COPA | MultiRC | ReCoRD | Mean | Params (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA | 83.43 | 85.71/76.13 | 85.92 | 71.32 | 77.88 | 87.00 | 75.24/38.30 | 83.64/84.39 | 78.41 | 1.33 |
| AdaLoRA (r̄ = 8, authors' reproduction) | 82.94 | 87.50/78.08 | 85.92 | 70.06 | 78.85 | 94.00 | 60.19/14.80 | 85.38/85.93 | 77.21 | 1.33 |
| JIVEAdapter | 82.81 | 89.29/80.37 | 85.92 | 71.79 | 75.96 | 91.00 | 71.46/30.22 | 85.74/86.33 | 78.65 | 1.33 |
The single-task table is labelled "best per task" in its caption tab. 1; the selection rule is not stated.
Multi-task, all tasks trained jointly tab. 2
GLUE, from tab. 2, with the metrics of tab. 1:
| Method | MNLI | QQP | QNLI | SST-2 | CoLA | RTE | MRPC | STS-B | Mean | Params (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| HydraLoRA | 89.53 | 90.29/87.33 | 94.14 | 95.41 | 69.92 | 90.25 | 89.95/92.64 | 90.98/91.08 | 88.80 | 2.90 |
| MTL-LoRA | 89.93 | 90.36/87.15 | 94.49 | 95.53 | 65.61 | 89.17 | 90.93/93.36 | 91.71/91.68 | 88.42 | 2.66 |
| JIVEAdapter | 90.10 | 90.34/87.32 | 94.58 | 95.99 | 70.23 | 89.17 | 90.20/93.06 | 92.03/91.98 | 89.07 | 1.33 |
SuperGLUE, from tab. 2, with the metrics of tab. 1:
| Method | BoolQ | CB | RTE | WiC | WSC | COPA | MultiRC | ReCoRD | Mean | Params (M) |
|---|---|---|---|---|---|---|---|---|---|---|
| HydraLoRA | 85.54 | 100.00/100.00 | 88.45 | 70.85 | 79.81 | 91.00 | 76.94/39.24 | 83.87/84.50 | 82.24 | 2.90 |
| MTL-LoRA | 85.66 | 92.86/89.24 | 87.00 | 71.63 | 80.77 | 88.00 | 76.30/38.61 | 86.27/86.84 | 81.02 | 2.66 |
| JIVEAdapter | 84.86 | 92.86/89.32 | 85.92 | 69.28 | 87.50 | 91.00 | 77.28/42.39 | 86.76/87.28 | 82.06 | 1.33 |
The text notes that HydraLoRA takes the overall SuperGLUE average, driven by its perfect CB score, while JIVEAdapter reaches a comparable average without MoE routing §4.1.2. The #P column is per task head at sequence length 128 tab. 2.
Incremental folding tab. 6
Averages are as printed. Other per-task values are in tab. 6.
| Method | GLUE avg | SuperGLUE avg |
|---|---|---|
| JIVEAdapter, all at once | 89.07 | 82.06 |
| JIVEAdapter, incremental, multi-task base | 81.13 | 80.72 |
| JIVEAdapter, incremental, single-task base | 79.54 | 73.69 |
| Sequential adapter fine-tuning | 61.11 | 68.36 |
| Sequential full fine-tuning | 69.55 | 65.60 |
The two sequential baselines serve as continual-learning lower bounds, since neither keeps a frozen shared part §4.1.2 app. D.1. The GLUE base for the single-task variant is MNLI, and for the multi-task variant MNLI, QQP, SST-2 and QNLI. The SuperGLUE base composition is not stated app. D.1.
Structure reuse on held-out tasks fig. 2
Leave-one-out over the SuperGLUE pool, with the Joint trained on the other six tasks, and over the MNLI genre pool §4.1.3. Averages from the text:
- SuperGLUE pool: similarity reuse 70.62, adapt 81.82. Adapt wins on every held-out SuperGLUE task §4.1.3.
- MNLI genres: similarity reuse 89.33, adapt 89.36 §4.1.3.
Per-task bars and routing targets are shown only in fig. 2 and fig. 5. In the SuperGLUE pool, COPA routes to WiC and CB routes to BoolQ; RTE routes to MultiRC §4.1.3 app. D.4.2.
Parameter counts in tab. 4: similarity reuse 0.00 M, adapt reuse 0.66 M. The 0.00 M excludes the r_I scales and the fresh head that similarity reuse trains.
Rank estimation tab. 7
| Strategy | GLUE avg | SuperGLUE avg |
|---|---|---|
| Importance-based (effective rank 8) | 89.07 | 81.21 |
| Permutation-test (effective rank 8) | 88.54 | 82.06 |
The permutation test wins on SuperGLUE and the importance allocator on GLUE. The paper calls the permutation test the more convenient default where it stays competitive app. D.2 tab. 7.
Sharing ratio and effective rank
- Split at fixed effective rank 8 (SuperGLUE, multi-task), with π = r_J / (r_J + r_I): π = 0.25 gives 80.92, π = 0.50 gives 82.26, and π = 0.75 gives 82.14 fig. 6 app. D.4.2.
- Effective per-task rank sweep on GLUE: rank 4 gives 89.16, rank 8 gives 88.95, rank 16 gives 88.82 fig. 3 app. D.3.
Paper vs code
No official code.
Search, in the order the compile steps give:
- Paper text: no code link in the abstract, introduction or arXiv metadata in our records. The Reproducibility Statement says the full repository, with training and control scripts, is deferred to acceptance, and that only an anonymized JIVELayer implementation and its tests are disclosed as supplementary material § Reproducibility Statement. No project page is linked.
- GitHub repository search for "JIVEAdapter" and for "JIVE adapter multi-task LoRA": no results.
- GitHub repository search for the first author's name: no results. A user search for the same name returned three accounts with no link to the paper; none was used.
Because no official repository exists to read, no hyperparameter or algorithm step was checked against code, and no code citations are given.
Pitfalls
- Projection only applies when routing. In Alg. 1 the Individual is projected only in the multi-task branch. With one task (or no task ids) the Individual is used unprojected, and orthogonality then rests on the regulariser alone alg. 1 eq. 5.
- Approximate orthogonality. Eq. 6 is exact only when U_Jᵀ U_J = I, which the regulariser encourages with λ = 0.1 eq. 5 tab. 5. The rendered text gives no residual value for the orthogonality error. Measure ‖U_Jᵀ U_{I,t}‖ on your own runs.
- Write-back of the projection is not specified. Alg. 1 writes the projected factor as an assignment, U_{I,t} ← (I − U_J U_Jᵀ) U_{I,t}. Eq. 6 defines Ũ as a derived quantity. If it is written back into the trainable parameter, the optimiser sees a different parameter. The paper does not say which is meant; computing it as a derived value is the reading that matches eq. 6.
- Rank in the scaling is ambiguous. α/(r + ε) uses r, but the text does not say whether r is the initial rank or the active rank after pruning. ε is not stated eq. 3 eq. 4 §3.1.
- Adapter dropout not stated. The per-layer dropout on the update path is not given. The 0.1 in tab. 5 is for the classifier.
- DeBERTa position projections must be routed. Per-sample routing must also reach the relative-position query and key projections of disentangled attention. Leaving them shared, the paper reports, makes multi-task training fail to converge app. C.3.
- Mixed batches versus micro-batches. Mixed batches keep the Joint stable. The paper states that accumulating per-task micro-batches before one step recovers the same gradient §3.2.1 app. C.2. That equality holds only if the loss is normalised the same way in both cases, which the text does not state.
- Allocator constants. β1, β2, the score's mean terms and the zeroing rule are only partly stated. The EMA coefficients are not given app. B.1.1 alg. 5.
- Permutation test is under-specified. The number of permutations B, the significance level α, the outer iteration cap M, and the number of gradient batches are not stated in the rendered text. The classic JIVE alternating fit is cited, not described app. B.1.2 alg. 6.
- Incremental algorithm constants. τ, K, the convergence threshold, γ_t and the growth cap κ are not stated. κ is listed as an input, but the growth step is commented out of the active algorithm alg. 2.
- Incremental base for SuperGLUE. The GLUE base clusters are given; the SuperGLUE base is not app. D.1. The LaTeX source also has commented-out notes naming 15k and 8k step budgets for the incremental runs. These do not appear in the rendered paper, so they are not used here.
- Reuse is not data-free. Similarity reuse needs labelled minibatches of the new task to build its signature eq. 7, and it trains the scales and a fresh head on the new task's data alg. 3. The paper's phrase "little or no data" §4.1.3 therefore holds only for the number of trained parameters, not for labelled data.
- Shared-energy ratio is not comparable across references. When no single-task reference update exists, η is computed against the rank-r_I signature basis rather than an update alg. 3 §3.2.2. The two values are not on the same scale.
- Single seed, no variance. All results use one seed (value 6 in tab. 5). Gaps of a few tenths of a point, such as 89.07 against 88.80 on GLUE, are reported without error bars, so the paper alone cannot separate them from seed noise.
- Rank-8 point differs between figures. The rank-8 mean in the sweep is 88.95 fig. 3, while the headline matched-rank GLUE result is 89.07 tab. 2. The text does not say whether they are the same run.
- Baseline provenance. AdaLoRA is the authors' reproduction §4.1.1 tab. 1. The source of the LoRA numbers is not stated, and the single-task rows use "best per task" without a stated selection rule.
- Parameter counts differ at matched rank. Matching is on per-task effective rank, not parameter count. HydraLoRA and MTL-LoRA have 2.90 M and 2.66 M per task head, against 1.33 M for JIVEAdapter tab. 2 §4.
- Averaging convention in the incremental table. Single values in tab. 6 are the means of the two paired metrics in tab. 2. For example, QQP 88.83 is the mean of 90.34 and 87.32; MRPC 91.63 is the mean of 90.20 and 93.06. Compare the two tables with this in mind.
- SuperGLUE HydraLoRA lead. HydraLoRA's SuperGLUE average (82.24) is above JIVEAdapter's (82.06) in tab. 2. The text acknowledges this §4.1.2.
- Interpretability evidence is qualitative. The claim that each Individual is near-orthogonal to the Joint rests on a t-SNE and a companion matrix fig. 4, and no numerical overlap value is given in the rendered text.
- Merging is single-task only. Merging is disabled for T > 1, so multi-task inference keeps the frozen matrix plus one product for the Joint and one for each active task §3.4.
- Commented-out material is not part of the paper. The LaTeX source contains hidden text, including an ablation algorithm, extra hypotheses, an α-robustness study and an E5 analysis. These are not in the rendered paper and are not used on this card.
Lineage
- Builds on: JIVE, the joint and individual variation decomposition of Lock et al., refined through principal angles by Feng et al. §2; the AdaLoRA importance allocator app. B.1.1; the gradient-subspace idea of GPM and the task-gradient signature idea of Task2Vec §3.2.2.
- Compared against: LoRA, AdaLoRA (authors' reproduction), HydraLoRA and MTL-LoRA §4.1. MoE-based MoDE is named as not directly comparable §4. Sequential adapter and full fine-tuning are the incremental references §4.1.2.
- Superseded by: not stated.
Source and license
Adapted from “JIVEAdapter: A Multi-Task Additive Low-Rank Adapter via Joint and Individual Variation Explained (JIVE)” by Sara Abdali, Pashmina Cameron (arXiv:2610.07036), used under CC BY-SA 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. This page is licensed under CC BY-SA 4.0.