Paper · 2026-10-31 · cs.LG
Scute: Exact Batched Serving for PEFT Adapters Beyond LoRA
Scute writes LoRA, DoRA, (IA)^3, VeRA and OFT adapters in one six-slot normal form. Slots that merge into the adapter at load are folded; the rest run inside vLLM's LoRA shrink and expand, so all five serve in one batch. Author-reported on an RTX A6000: a mixed pool takes 1.05-1.17x the decode time of LoRA-only, and 1.5x to at least 2.5x the request rate of rank-64 conversions.
Low-rank adaptation (LoRA and variants)Parameter-efficient fine-tuningLLM servingllm-servinglorapeftvllmgpu-kernelsmulti-tenant
arXivPDFCodeMarkdown20 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A company that hosts one large language model often serves many customers, each with a small add-on that tunes the model to their task. If every add-on is of the LoRA kind, a server can keep one copy of the big model, run all customers' requests together in one batch, and add each customer's small correction on the way through. Several other popular add-on methods work differently. Today's serving software either refuses them or asks the operator to convert them into LoRA form, which can cost accuracy or a lot of memory.
The paper's idea is to describe every one of these add-ons with a single formula: the LoRA computation, plus six extra knobs. Some knobs can be merged into a customer's own weights once, when the add-on is loaded, and then cost nothing. The others must be applied to every token, but they can be applied inside the same steps the server already runs for LoRA, so they add only a little memory traffic. The result, called Scute, is a patch to the open-source vLLM server. It serves all five methods side by side and matches the reference library's output to within a small fraction of each add-on's effect.
TL;DR
- Scute writes LoRA, DoRA, (IA)^3, VeRA and OFT as one normal form with six slots eq. 1 §3.1 and serves all five in one batch on vLLM 0.19.1 §5 §6.1.
- A slot folds into the adapter's own weights at load when the adapter owns the tensor next to it. Otherwise it stays live and is computed for every token, inside the LoRA shrink and expand where possible §3.2 §4.1.
- Author-reported: a pool of five methods takes 1.17x the decode time of rank-16 LoRA-only, and a pool of four methods without VeRA takes 1.05x, on an RTX A6000 under a burst tab. 4.
- On the agreement test, the mean difference from PEFT is at most 1.5% of each adapter's effect, and all fifteen trained adapters stay within 0.4 accuracy points of PEFT §6.2 app. E.
- The public repository implements the normal form and fold rule. It does not include the training code, and its example adapter generator differs from the appendix description in several constants. See Paper vs code.
Problem
A platform that serves many tenants keeps one base model on the GPU and applies each tenant's adapter on top. The efficient way to do this for LoRA is to run the shared base matrix once for the whole batch, then compute each tenant's small low-rank update §2 §1. Serving systems such as Punica, S-LoRA, vLLM and SGLang are built this way. When an adapter is loaded, the production engines among them turn away adapters from other methods §1.
Converting a non-LoRA adapter to LoRA has two costs. For (IA)^3 and OFT, the update to the base weight has full rank, so a practical LoRA rank can only approximate it, and the accuracy loss depends on the data used to fit the approximation §1 tab. 3. For VeRA, each tenant owns only a few vectors, but a LoRA conversion needs its own factors, about 0.34 GB at rank 64 per adapter §1 §6.5. Giving each tenant a merged copy of the model avoids the conversion error, but then requests for different tenants cannot share one batch §1.
Method
Normal form
For a hooked projection, the base model computes xW. The paper writes the output of adapter a as a single expression with six slots: an input rotation R (block diagonal, block size b_R), an output scale s, an output shift b, and for each of K low-rank terms an inner vector d_k, an outer vector e_k and a scalar c_k §3.1 eq. 1.
F_a(x) = \big((xR)W\big)\odot s + b + \sum_{k=1}^{K} c_k\Big(\big((xA_k)\odot d_k\big)B_k \odot e_k\Big) eq. 1
A slot is the identity (one, a vector of ones, zero, or the identity matrix) when the method does not use it. Each factor is either private to one adapter or common to the whole pool; the base weight W is always shared §3.1. Table 1 lists how each method fits; "folds" and "live" refer to the rule in §3.2.
tab. 1 Members of the normal form. VeRA's A and B are shared; the folds are those the fold rule allows.
| method | owned matrices | folds | live slots |
|---|---|---|---|
| LoRA | A, B | c into A | none |
| DoRA | A, B | e into B; c into A | s |
| (IA)^3 | none | none | s |
| SSF | none | none | s, b |
| VeRA | none | c into d | d, e |
| OFT | R | none | R |
How each method maps to the form §3.1:
- LoRA: one owned term. Its update c·A·B is already a member.
- DoRA: one owned term, with the output scale s equal to a learned magnitude divided by the column norm of W + cAB. The e slot is folded into B at load §5.1 app. B.
- (IA)^3: the vector is the output scale s on the up-projection slice of a gated feed-forward block. The paper uses the identity that scaling the product of gate and up activations is the same as scaling the up branch §3.1.
- VeRA: one term whose A and B are the same frozen random pair for every adapter, with per-adapter vectors d and e that stay live §3.1 §3.2.
- OFT: an input rotation R of block-diagonal, orthogonal blocks, each built from a skew-symmetric matrix by the Cayley map §2 §5.1.
Two methods are not covered. BOFT needs one input transform per factor, which the paper does not implement. HiRA and full fine-tuning need a dense matrix per adapter after the shared GEMM, so they are outside the form §3.1.
Fold rule
A slot can be absorbed into the adapter's private matrices at load time, whenever one of those matrices is the neighbour that the slot multiplies. The identity (xA)⊙d = x(A diag d) shows why an inner vector folds into an owned A, and the analogous identities fold e into B and c into any owned factor §3.2. The paper states this as an "if and only if" that holds for generic parameters and low total rank r §3.2 app. A. The exact conditions are 4r < min(d_in, d_out), or 3r < d_in and 2r < d_out when every A is shared, plus two structural assumptions app. A.
Consequences §3.2:
- s, b and R never fold, because the only tensor next to them is the shared W.
- Exactly folding DoRA's s would require a merged copy of W for each tenant, so DoRA's scale stays live. In the paper's measurements, the matrices W·diag(s−1) have full numerical rank for all sixty adapted projections tested, and the best rank-16 fit still misses 59.7% or more of the matrix's squared Frobenius mass app. B.
- For VeRA, keeping d and e live is what keeps the factors shared. Folding them would give each tenant its own rank-256 copy of A and B.
- Stacking two members at one hook gives another member, with output scale s1·s2 and shift b1·s2 + b2 §3.1. Swapping the two members changes the result in general. The results coincide only under the matching conditions in app. A.
Cost of the live slots
The engine computes each live slot inside vLLM's LoRA schedule through one of three routes §4.1:
- Scale in the expand (DoRA, (IA)^3 scale s). The expand already loads h, applies the low-rank update and stores the sum, working through output tiles. Every tile maps to a single tenant, so the same pass can scale h by that tenant's s before storing it. The extra cost is one read of s per tile §4.1.
- Shared factors in two GEMMs (VeRA d and e). A first GEMM computes xA for every row of the batch and scales each row by its adapter's d. A second GEMM multiplies by B and applies e per tile, accumulating into the output. The factors are read once per step, regardless of how many VeRA tenants are present §4.1.
- Pre-transform pass (OFT R). One pass before the shared GEMM applies each row's rotation blocks. The rotated rows of all adapters then share one GEMM §4.1.
The byte cost of a route is set by the bytes its passes read and write. The paper gives a time model t = t0 + ΔQ/β + ΔL·λ, where t0 is the time measured with none of the added passes, ΔQ and ΔL are the added bytes and launches, β is bandwidth and λ is per-launch cost §4.3. On the RTX A6000 the paper measures β = 680 GB/s, β_L2 = 3205 GB/s (when the copy fits in L2) and λ = 0.96 µs §2.
The paper also compares conversion to LoRA by the bytes read per decode step. With 16 tenants on Llama-3.1-8B in fp16, the shared-factor route reads 1.19 GB for VeRA at rank 256, against 21.47 GB for the rank-256 conversions tab. 2.
Compiler and runtime
Admission compiler §5.1 compile.py. The compiler reads a PEFT checkpoint and writes it back in normal form, using the tensor names vLLM expects for LoRA. vLLM's loader, paging and scheduler then still apply unchanged.
- LoRA and DoRA pass through unchanged. For DoRA, the engine computes s at admission, after vLLM has packed and sharded the weights _fold.py.
- (IA)^3 becomes the vector s on the up slice, with a zero rank-one term so that vLLM admits the adapter compile.py.
- VeRA becomes the per-module vectors d and e plus a reference to one shared pair of factors, stored once compile.py.
- OFT's Cayley-parametrised blocks R are computed at admission compile.py.
Runtime patch §5.2. The patch leaves vLLM's scheduler, paging, attention and batch composition unchanged. For each adapted layer it allocates one buffer per slot, set to identity, and runs four steps: the base GEMM (split by slice when an adapter carries R), vLLM's shrink (skipping adapters without a term), the expand with s applied as each tile finishes, and the shared-factor term.
# One adapted projection, batched over tenants [§5.2] [§4.1]
# Shapes: x (N, d_in), h (N, d_out), v (N, r); tenant index per row
def scute_forward(x, tenant_of_row, pools):
# 1. Input rotation of OFT tenants, one pass before the shared GEMM [§4.1]
x = rotate_blocks(x, pools.R, tenant_of_row)
# 2. One shared base GEMM for every tenant in the batch [§2]
h = x @ base_weight
# 3. Shrink: each row's low-rank factor V = x A [§2]
v = x @ pools.A[tenant_of_row]
# 4. Expand: h' = s * h + V B, with s applied per tile [§5.2]
h = pools.s[tenant_of_row] * h + v @ pools.B[tenant_of_row]
# 5. VeRA shared term, ((x A_shared) * d) B_shared * e [§4.1]
h = h + shared_term(x, tenant_of_row, pools)
return h_expand.py and
vllm_expand_s.py implement the same four steps.
The fused epilogue is tiled_c += tiled_out * s in the paper; the code applies
the same product with a per-slice broadcast §5.2.
DoRA scale at admission §5.1 app. B.
# DoRA magnitude folded to the output scale at admission [§5.1] [app. B]
# Shapes: W (d_in, d_out); A (d_in, r); B (r, d_out); m (d_out,)
def dora_scale(base_weight, a, b, c, magnitude):
# 1. Merge the update, with the scale c already applied [§5.1]
merged = base_weight + c * (a @ b)
# 2. Squared norm of each output column, accumulated in float32 [§5.1]
sq_norm = (merged ** 2).sum(axis=0)
# 3. Row-parallel: all-reduce sq_norm before the sqrt [app. A]
# 4. Output scale per column, stored in fp16 or fp32 (never bf16) [§6.1]
return magnitude / sqrt(sq_norm)_fold.py implements the scale computation, and _fold.py folds e into B. The engine refuses bfloat16 for DoRA _guards.py.
Hyperparameters
tab. A5 reproduces the paper's training learning rates and steps (see Results). Other values are below.
| Name | Value | Where | Notes |
|---|---|---|---|
| normal-form slot R, block size | 32 (OFT) | app. D | the PEFT default block size |
| LoRA and DoRA rank, alpha | 16, 32 | app. D | trained adapters |
| VeRA rank | 256 | app. D | projections saved; library's default seed |
| (IA)^3 target modules | k_proj, v_proj, down_proj | app. D | library default |
| adaptation targets (LoRA, DoRA, VeRA, OFT) | q, k, v, o, gate, up, down | app. D | the four methods adapt these seven projections |
| training precision | bf16 autocast, gradient checkpointing | app. D | |
| sequence length, effective batch | at most 512 tokens, 16 | app. D | |
| optimiser | AdamW, no weight decay | app. D | |
| schedule | cosine, 3% warm-up | app. D | |
| training steps | min(1000, one epoch) | app. D | loss on completion tokens only |
| learning rate selection | lowest held-out loss after 100 GSM8K steps | app. D | per method |
| distillation | 300 steps, lr 1e-4, KL to PEFT's next-token distribution | app. C | last 200 training examples held out |
| ASVD calibration | 128 sequences; covariance regularised by 1e-6 times mean diagonal | app. C | accumulated in float64 |
| ASVD generic calibration | 128 chunks of 160 tokens from WikiText-2 | app. C | |
| evaluation | GSM8K exact match on 500 test problems; MNLI and BoolQ accuracy on 1000 validation examples | app. D | greedy decoding |
| serving precision | fp16 | §6.1 | CUDA graphs and torch.compile on |
| engine settings | 64 sequences, 8192 batched tokens, max length 1024 | app. G | |
| GPU memory fraction | Scute 0.83, other engines 0.90 (RTX A6000); Scute 0.86, others 0.90 (A100) | §6.1 app. H | equalises device memory |
| latency target | first token within one second, each later token within 100 ms | §6.5 | a rate is sustainable if 90% of requests are on time |
| amplified test adapters | (IA)^3 s = 1+0.3z; VeRA d = 0.1(1+0.3z), e = 0.5z; DoRA g = 1+0.3z; OFT skew 0.02z | app. E | used for the exactness test; see Paper vs code |
tab. A5 Trained adapters: learning rate, optimiser steps and task metric, from the PEFT library app. D.
| method | task | lr | steps | metric |
|---|---|---|---|---|
| LoRA | GSM8K | 0.0002 | 467 | 0.664 |
| LoRA | MNLI | 0.0002 | 1000 | 0.908 |
| LoRA | BoolQ | 0.0002 | 588 | 0.900 |
| DoRA | GSM8K | 0.0002 | 467 | 0.660 |
| DoRA | MNLI | 0.0002 | 1000 | 0.919 |
| DoRA | BoolQ | 0.0002 | 588 | 0.887 |
| (IA)^3 | GSM8K | 0.01 | 467 | 0.598 |
| (IA)^3 | MNLI | 0.01 | 1000 | 0.902 |
| (IA)^3 | BoolQ | 0.01 | 588 | 0.879 |
| VeRA | GSM8K | 0.01 | 467 | 0.584 |
| VeRA | MNLI | 0.01 | 1000 | 0.909 |
| VeRA | BoolQ | 0.01 | 588 | 0.895 |
| OFT | GSM8K | 0.0001 | 467 | 0.630 |
| OFT | MNLI | 0.0001 | 1000 | 0.909 |
| OFT | BoolQ | 0.0001 | 588 | 0.897 |
The metric column is the paper's accuracy on the stated task, in the library's setup. The base model scores 26.2% on GSM8K, 36.8% on MNLI and 74.0% on BoolQ app. C tab. A4.
Results
Author-reported throughout. Hardware: NVIDIA RTX A6000 (48 GB), Llama-3.1-8B in fp16 unless stated §6.1. Each cell is the median of six repetitions except where noted. The A100 repeats the main results §6.6.
| Setting | Metric | Reference | This paper | Source |
|---|---|---|---|---|
| Five methods, four tenants each (20 tenants), burst, same GEMM path | decode time | rank-16 LoRA-only, 1.00 | 1.17x | tab. 4 |
| Same pool | time to first token | LoRA-only, 1.00 | 1.18x | tab. 4 |
| Four methods, no VeRA (16 tenants), burst | decode time | LoRA-only, 1.00 | 1.05x | tab. 4 |
| Same pool | time to first token | LoRA-only, 1.00 | 1.04x | tab. 4 |
| Exactness: nine amplified adapters, five methods | mean error, share of adapter effect | PEFT library, float32 | 0.3% to 1.5% | app. E tab. A6 |
| Fifteen trained adapters, Scute | accuracy difference | PEFT library | within 0.4 points | tab. 3 |
| Conversion to LoRA rank 64, truncated SVD | worst accuracy loss | PEFT | 23.2 points ((IA)^3, MNLI) | tab. 3 |
| Conversion to LoRA rank 64, ASVD, own-task data | worst accuracy loss | PEFT | 1.2 points | §6.2 |
| Conversion to LoRA rank 64, ASVD, generic text | worst accuracy loss | PEFT | 19.6 points ((IA)^3, GSM8K) | §6.2 tab. 3 |
| Conversion to LoRA rank 64, distillation | worst accuracy loss | PEFT | 2.0 points | §6.2 |
| Conversion to LoRA rank 16, truncated SVD, VeRA | worst accuracy loss | PEFT | 18.5 points | §6.2 app. C |
| One VeRA tenant | decode time per token | Scute, 35.4 ms | stock with rank-256 conversion, 36.7 ms (1.04x) | §6.3 |
| 16 VeRA tenants | decode time per token | stock with rank-256 conversions, 65.1 ms | Scute 36.1 ms, flat from one tenant (35.4 ms) | §6.3 |
| Mixed pools, conversions on the same GEMM path | decode time | Scute | at least 1.12x (VeRA at rank 64); 1.14x (no VeRA); 1.65x (five methods) | tab. 4 |
| Mixed pools, vLLM as shipped | decode time | Scute, same GEMM path | 1.21x to 1.98x across rates | §6.4 |
| 1024 tenants, split evenly between (IA)^3 and VeRA | sustainable request rate | stock conversions, 4 req/s | Scute 6 req/s (1.5x) | §6.5 |
| 256 tenants | sustainable request rate | stock conversions, 4 req/s | Scute 8 req/s (2x) | §6.5 |
| 64 tenants | sustainable request rate | stock conversions, 4 req/s | Scute at least 10 req/s (at least 2.5x, the highest rate tested) | §6.5 |
| 1024 tenants, 8 req/s | throughput | stock, 464 tokens/s | Scute 932 tokens/s (2.01x) | §6.5 |
Table 2 shows the bytes read per step with 16 tenants, and the time at copy bandwidth.
tab. 2 Bytes read per decode step with 16 tenants, Llama-3.1-8B, fp16, every adapted projection of 32 layers. "exact" marks routes that serve the adapter without approximation. Bold marks the fewest bytes for each method in the paper.
| adapter | path | lossless | GB read | time (ms) |
|---|---|---|---|---|
| LoRA, rank 16 | shrink-expand | yes | 1.34 | 1.97 |
| VeRA, rank 256 | route 2 (shared factors) | yes | 1.19 | 1.75 |
| VeRA, rank 256 | as LoRA, rank 256 | yes | 21.47 | 31.58 |
| VeRA, rank 256 | as LoRA, rank 64 | no | 5.37 | 7.90 |
| OFT, blocks of 32 | route 3 (pre-transform) | yes | 1.28 | 1.88 |
| OFT, blocks of 32 | as LoRA, rank 64 | no | 5.37 | 7.90 |
| (IA)^3 | route 1 (expand) | yes | 0.01 | 0.01 |
| (IA)^3 | as LoRA, rank 64 | no | 1.88 | 2.76 |
tab. 3 Accuracy gaps relative to the reference library, restricted to (IA)^3, OFT and VeRA. The PEFT column is absolute accuracy in percent, and every other column is a signed change in points relative to it. The Scute column runs all fifteen adapters together; the remaining columns are rank-64 LoRA conversions fitted in different ways. The smallest gap in each row is in bold.
| method | task | PEFT | Scute | SVD | ASVD (own task) | ASVD (generic) | distill |
|---|---|---|---|---|---|---|---|
| (IA)^3 | GSM8K | 59.8 | +0.4 | −18.6 | −0.8 | −19.6 | −0.4 |
| (IA)^3 | MNLI | 90.2 | 0.0 | −23.2 | +0.5 | −16.9 | −0.3 |
| (IA)^3 | BoolQ | 87.9 | −0.1 | −4.8 | +0.8 | −1.6 | +0.3 |
| OFT | GSM8K | 63.0 | +0.2 | −5.6 | −1.2 | −4.6 | −2.0 |
| OFT | MNLI | 90.9 | 0.0 | −2.9 | +0.3 | −0.9 | −0.1 |
| OFT | BoolQ | 89.7 | 0.0 | −0.6 | −0.1 | −0.2 | −0.8 |
| VeRA | GSM8K | 58.4 | +0.2 | +0.8 | −0.8 | +3.4 | none |
| VeRA | MNLI | 90.9 | +0.1 | −0.6 | +0.3 | 0.0 | none |
| VeRA | BoolQ | 89.5 | 0.0 | −0.1 | +0.1 | −0.1 | none |
tab. 4 Ratios of medians over six repetitions, measured on an RTX A6000 under a burst. The five-method pool has 20 tenants, four per method, and stock keeps 16 resident. The "VeRA at 64" row comes from a separate run where stock serves VeRA at rank 64. The no-VeRA pool has 16 tenants, four each of four methods that exclude VeRA.
| pool | load | Scute / LoRA-only | conversion / Scute | conversion / Scute (stock) |
|---|---|---|---|---|
| five methods | decode | 1.17 | 1.65 | 1.73 |
| five methods | prefill | 1.18 | 1.14 | 1.12 |
| VeRA at 64 | decode | 1.11 | 1.12 | — |
| VeRA at 64 | prefill | 1.12 | 0.94 | — |
| no VeRA | decode | 1.05 | 1.14 | 1.25 |
| no VeRA | prefill | 1.04 | 1.01 | 0.99 |
The paper's Table A6 gives the exactness test. Nine amplified adapters, served together in one batch, first on a single GPU and then split across two GPUs for tensor parallelism, are checked against the reference library run in float32. "Effect" means the average change in log-probability per token that an adapter makes relative to the base model. The error is expressed as a share of that effect app. E.
| method | effect (nats/token) | error / effect, TP 1 | TP 2 |
|---|---|---|---|
| LoRA | 0.20 | 1.27% | 1.51% |
| DoRA | 1.98 | 0.55% | 0.47% |
| (IA)^3 | 1.09 | 0.54% | 0.55% |
| VeRA | 0.22 | 1.41% | 1.49% |
| OFT | 0.86 | 0.34% | 0.37% |
The rest of the paper's measurements, in short:
- Cost per tenant §6.3: Scute's time per token is flat for VeRA, at 35.4 ms with one tenant and 36.1 ms with 16. Stock grows from 36.7 ms to 65.1 ms with rank-256 conversions. Scute adds 0.05 ms per tenant, while stock adds 1.94 ms per VeRA tenant and 0.53 ms per (IA)^3 or rank-64 OFT tenant tab. A9.
- Many tenants §6.5: all 1024 tenants fit resident in Scute on one RTX A6000 with a 98k-token KV cache, against 64 rank-64 conversions and 28k tokens for stock. The paper's runs keep 64 resident for Scute and 48 for stock, so both must load adapters on demand.
- A100 §6.6: the same ordering holds. At 16 tenants, converted OFT and converted (IA)^3 run 1.31x and 1.34x slower than Scute, and at 1024 tenants Scute sustains 8 req/s, a rate at which stock meets the latency target for 32% of requests app. H.
- DoRA in vLLM §4.3: with 1024 or more tokens, the expand that carries s takes 0.985x to 1.022x as long as vLLM's plain LoRA expand. A pull request that writes a per-row scale and multiplies afterwards costs 2.38x to 2.95x. End to end, four DoRA tenants on Scute take 1.002x the per-token time of four LoRA tenants.
Paper vs code
Official repository: github.com/ThomasWalker1/Scute-deploy, at commit 0999d8f9926e4d83ee51f356e235771d98b5e9e6, linked from §1 of the paper. The code is Apache-2.0 LICENSE. It modifies two vLLM kernels, which NOTICE lists NOTICE.
Discrepancies found:
- Amplified test adapters differ from the appendix. The paper uses DoRA g = 1 + 0.3z app. E. The example generator multiplies the magnitude by 1 + 0.05z make_adapters.py. For VeRA, the paper uses e = 0.5z app. E; the generator sets the outer vector to z make_adapters.py. The paper does not state an initial value for LoRA; the generator sets lora_B to 0.08z for LoRA and DoRA make_adapters.py. The generator also uses LoRA alpha 16 make_adapters.py, while the trained adapters use alpha 32 app. D. The generator is an example, so the paper's Table A6 may have used a different harness. Its numbers cannot be reproduced exactly from this repository make_adapters.py.
- No training code. The repository contains no training script, so the training settings in app. D and tab. A5 cannot be checked against code. The generator's docstring says its adapters are untrained make_adapters.py.
- Cayley variant not stated. The paper trains OFT "in the library's default Cayley form" app. D. The compiler reads use_cayley_neumann and num_cayley_neumann_terms from the checkpoint, defaulting to Neumann with five terms when absent compile.py pretransform.py. The paper does not say which form the trained adapters used.
- Consistent points, for reference. The fold rule and the three routes match the code's decision procedure, where s, b and R stay live and c folds into A or d slots.py slots.py. The one-line epilogue change is in the code vllm_expand_s.py. The zero rank-one term for vector-only methods is in the compiler compile.py.
- Example numbers are not the paper's. The README's table of errors comes from SmolLM2-360M, not the paper's Llama-3.1-8B measurements README.md.
Pitfalls
- The paper's "exact" means up to fp16 rounding. The engine's error is about the same size as the base model's own fp16 error against PEFT README.md. The paper calls a slot that is dropped or applied twice a large error, which the amplified test adapters are built to expose §6.2.
- DoRA needs fp16 or fp32. A trained g lies in about [0.98, 1.02]. bfloat16 resolves that into nine values, and the engine refuses it _guards.py. The default dtype of Llama-3.1-8B in vLLM is bfloat16, so the dtype must be set explicitly _fold.py.
- DoRA's norm must reduce over squares first under tensor parallelism. A row-parallel shard holds part of d_in, so the squared norms must be all-reduced before the square root. Doing it in the other order makes the shards disagree _fold.py _fold.py app. A.
- The DoRA e factor must be folded into B. The patched expand expects B to already carry e. An engine that applies e in the expand as well would apply it twice _fold.py.
- Stacking order matters. Applying (IA)^3 on top of LoRA rescales the LoRA term, but LoRA on top of (IA)^3 leaves the LoRA term at full size. When one member rescales and the other adds a shift or a term, the two results disagree for generic parameters §3.1 app. A.
- The fold rule is generic, not universal. It holds for generic parameters and low rank r, and the appendix gives explicit size conditions. VeRA's key and value projections, with d_out = 1024 = 4r, sit at the boundary app. A.
- Live slots cost memory traffic, and the cost is set by the byte model. The model fits well when the extra bytes dominate. The median error on total time is 0.7% to 14.0% per route, and the error is larger in launch-bound cells, from 2% to 24% app. F tab. A7.
- Padding is part of the baseline's cost. Stock vLLM pads each adapter up to the highest rank the engine serves. In the five-method pool, the paper attributes most of the stock overhead to that padding §6.4.
- Memory accounting must be equal. Scute's slot buffers sit outside vLLM's budget, so the paper uses 0.83 against 0.90 for the others §6.1.
- Many-tenant results use synthetic tenants. Tenants reuse one of 16 synthetic adapter files of their method, under their own names. The engine therefore loads and swaps them as separate adapters, but the weights are not distinct app. G.
- The 2.5x figure is a lower bound. At 64 tenants Scute still met the target at the highest rate tested, so the sustainable rate is at least 10 req/s §6.5.
- Rounding in the headline. The abstract says the mean error is at most 1.5%. Table A6 lists 1.51% for LoRA with tensor parallelism of 2 tab. A6.
- Requirements that the paper does not list. The engine is pinned to vLLM 0.19.1 and refuses other releases unless the caller overrides the check engine.py. It refuses quantised base weights, fully sharded LoRA, LoRA bias, COFT and block-shared OFT README.md. It needs one VeRA shared pair and one OFT block size per engine engine.py.
- Conversion accuracy depends on calibration data. Converted (IA)^3 loses up to 19.6 points when fitted on generic text and up to 1.2 when fitted on the tenant's own data at rank 64 tab. 3 app. C.
Lineage
- Builds on: LoRA, and Punica's segmented schedule that batches adapter rows into segments §2 §7. The base method families are DoRA, (IA)^3, VeRA and OFT §2. The PEFT library is the reference implementation §1. The engine is built on vLLM 0.19.1 §5.
- Compared against: stock vLLM serving LoRA conversions at rank 64, and rank 256 for VeRA §6.1. The rank-16 LoRA-only pool is the reference for the price of exactness §6.1.
- Related work the paper cites: PetS, FLoRA, S-LoRA, VB-LoRA, Tied-LoRA, DeltaZip and BitDelta §7. A DoRA-serving change to vLLM (pull request 47395 in the vllm-project repository) was still pending when the paper was written §7.
- Superseded by: not known from this wiki.
Source and license
Adapted from “Scute: Exact Batched Serving for PEFT Adapters Beyond LoRA” by Thomas Walker, Richard Baraniuk (arXiv:2611.01089), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ThomasWalker1/Scute-deploy at commit 0999d8f9926e (Apache-2.0). Any quoted code is shown under that license.