Paper · 2026-10-16 · cs.AI, cs.LG
ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning
ToPA is a parameter-efficient fine-tuning method that adapts a frozen weight with a chain of block-wise Toeplitz matrices plus an identity residual, avoiding explicit low-rank or sparsity limits. Block structure shortens the chain and aids stability. Author-reported: best commonsense average (86.3 with LLaMA-3-8B) and wins on GSM8K and HumanEval at about 12.6M parameters.
Parameter-efficient fine-tuningLow-rank adaptation (LoRA and variants)Sparse fine-tuningStructured matrix parameterizationfine-tuningparameter-efficientstructured-matrices
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Fine-tuning a large model means changing its weights, but changing all of them is expensive. Common shortcuts restrict the change in one of two ways: LoRA forces the change to be a low-rank product, and OFT/BOFT force it to be a sparse, structured transformation. Both shortcuts leave performance behind full fine-tuning.
ToPA takes a different structure. A Toeplitz matrix is one where every diagonal holds a single repeated value, so it stores far fewer numbers than a full matrix while still being dense. ToPA splits such a matrix into a grid of smaller Toeplitz blocks, which gives each factor more numbers to work with. Several of these factors are multiplied together, and an identity matrix is added so the adapter starts from the original model.
The point is that the block layout lets the chain stay short, and short chains train more stably than long ones. The authors report that this gives a better accuracy-per-parameter trade-off than the low-rank and sparse baselines they test.
TL;DR
- ToPA adapts a frozen linear layer
W0byW = W0 (∏ T_blk + I), where eachT_blkis a block-wise Toeplitz matrix §3.3 eq. 8. - Block-wise Toeplitz factors keep the update dense and not restricted to a fixed rank or sparsity pattern §4.1.
- Author-reported gains are small, over the strongest baseline in each setting. Examples: 86.3 vs 85.9 average on commonsense with LLaMA-3-8B, and +0.9 on GSM8K with LLaMA-3-8B tab. 1 tab. 2.
- Only the LLaMA-3 commonsense and math/code runs have training scripts in the repository. GLUE, subject-driven generation and LLaMA-2 runs have no code here.
- Relevant if you need a PEFT method with more capacity than LoRA at a similar budget and can afford the extra FFT work and training care described under Pitfalls.
Problem
Full fine-tuning updates every weight and is costly in memory and compute §1. Low-rank adapters (LoRA and its variants) bound the rank of the update, and they note that a task can have a low intrinsic dimension while its weight update still has high matrix rank. They measure the rank of full fine-tuning on LLaMA-2-7B at about 3758, against 16, 32 and 64 for LoRA §2.1 fig. 1.
Sparse orthogonal adapters (OFT, BOFT) fix the sparsity pattern before training. Each adapted output column is then confined to a subspace chosen in advance, and the authors report that the non-zero ratio of the adapter R falls to about 0.5% to 6.5% under tight budgets §2.2 fig. 1.
The authors assume that a method which is neither rank-limited nor pattern-limited, while keeping the parameter count low, should close this gap. The stated assumptions are the usual PEFT setting: a frozen pretrained weight, adapters on attention or projection matrices, and a tight parameter budget §2.2 app. E.3.
Method
ToPA builds on three ideas.
- Toeplitz factor. An
n × nToeplitz matrix is determined by a vector of2n − 1numbers, and its product with a vector can be computed with the FFT inO(n log n)§3.1 eq. 4 app. D. - Block-wise Toeplitz factor. Split
nintopequal blocks of sizeb = n/p. Each of thep × psub-blocks is an independent Toeplitz matrix. The total number of parameters isp(2n − p), aboutptimes that of one Toeplitz matrix. A chain oftsuch factors therefore hast · p(2n − p)parameters §3.3 eq. 7 app. C. - Residual chain. The adapter is
W = W0 (∏_{i=1}^{t} T_blk^i + I), with the identity as a residual §3.3 eq. 8. Shorter chains mean fewer factors in the gradient product of Prop. 1, which the authors link to stability §3.2 eq. 6.
The naive version, a plain Toeplitz chain W0 ∏ T_i with T_i initialised to the identity, trains unstably. In the authors' run with t = 16 on LLaMA-3-8B, gradient norm and loss fluctuate heavily and the run collapses §3.2 fig. 2.
Initialisation. The first factor is set to zero and the others are drawn from a normal distribution, so the product is zero and W = W0 at step 0 §3.4 eq. 9. The paper does not state the standard deviation σ.
Expressiveness. The authors prove a universal representability result for block-wise Toeplitz chains. The chain length needed is about ⌊n/(2p)⌋ + 1 generically, and at most about 2n/p + 5 always. They also show that some updates need O(n) ToPA parameters but O(n²) for LoRA and O(n log n) for OFT or BOFT §4.2. These are existence results, not a bound on what training reaches.
Rank. For a generic choice of factors, the update ΔW = W0 ∏ T_blk has the same rank as W0 with probability one §4.1 eq. 10. At initialisation the first factor is zero, so ΔW = 0 at step 0. The result is about generic parameters, not the starting point.
Pseudocode for the forward pass, reconstructed from the paper and the reference code. The code applies factors in a loop, so it is the reference for ordering:
# ToPA forward for one adapted linear layer [§3.3] [eq. 8]
# x: (B, n) input rows; W0: (m, n) frozen weight
# factors: list of t arrays, each (p, p, 2b - 1), with b = n / p
def topa_forward(x, W0, factors):
# Run x through the chain of block-wise Toeplitz factors; the reference
# code applies the highest-index factor first and index 0 last
# [code: peft/tuners/topa/layer_block_toeplitz.py#L214-L217]
h = x
for T in factors_in_code_order(factors):
h = block_toeplitz_apply(T, h)
# Add the identity residual, then apply the frozen weight [eq. 8]
# [code: peft/tuners/topa/layer_block_toeplitz.py#L290]
return (h + x) @ W0.T# Apply one block-wise Toeplitz factor to a batch of vectors [app. D]
# T: (p, p, 2b - 1); x: (B, n) with n = p * b; output: (B, n)
def block_toeplitz_apply(T, x):
# Split each row into p blocks of length b [§3.3]
blocks = reshape(x, (B, p, b))
# Circular-embedding length: smallest power of two >= 2b - 1; the
# kernel is [T[j, i][b-1:], zeros, T[j, i][:b-1]]
# [code: peft/tuners/topa/layer_block_toeplitz.py#L163-L169]
L = next_power_of_two(2 * b - 1)
# Output block i sums, over input blocks j, the circular convolution
# of the kernel T[j, i] with input block j, computed with the FFT
# [app. D] [code: peft/tuners/topa/layer_block_toeplitz.py#L171-L186]
out = sum_over_j(circular_conv(kernel(T[j, i]), blocks[:, j]))
# Keep the first b samples of each output block
# [code: peft/tuners/topa/layer_block_toeplitz.py#L184]
return reshape(out[..., :b], (B, n))# Initialise the chain so that W = W0 at step 0 [§3.4] [eq. 9]
# Paper: first factor zero, others N(0, sigma^2); sigma is not stated.
def init_factors(t, p, n, sigma):
factors = []
for k in range(t):
# Each factor holds p * p Toeplitz blocks of 2b - 1 numbers each
# [§3.3] [app. C]
if k == 0:
factors.append(zeros(p, p, 2 * (n // p) - 1))
else:
# Reference code uses sigma = 0.02
# [code: peft/tuners/topa/layer_block_toeplitz.py#L84-L90]
factors.append(normal(0, sigma, (p, p, 2 * (n // p) - 1)))
return factorsParameter count. For a square n × n layer, a chain of t factors with p blocks per dimension has t · p(2n − p) parameters app. C. Per factor, p(2n − p) equals p² blocks times (2n/p − 1) numbers each, so the two forms agree. The reference code stores the same count as (in/b)² · (2b − 1) · t per adapted layer layer_block_toeplitz.py.
Computation. One block-wise product costs O(p² q log q) with q = n/p, which is O(p n log q), and a chain of t costs O(t p n log q) app. C. The paper reports no measured wall-clock time or memory. Its sections and the LaTeX source contain only these analytic costs.
Hyperparameters
Values as stated in the paper, with script or code values where the paper says "not stated". Where a value is derived from the scripts (for example effective batch size = per-device batch × accumulation × GPUs), the card says so.
| Setting | Adapted modules | t, p | Learning rate | Batch | Epochs | Schedule | Warm-up | Max length | Source |
|---|---|---|---|---|---|---|---|---|---|
| Commonsense, LLaMA-3-8B | query, value | t=3, p=8 (block 512) | 3e-5 | 16 (derived: 1 × 4 × 4 GPUs) | 3 | not stated in paper; code: linear | 100 steps | not stated in paper; code: 256 | §5.2 tab. 1 app. E.3 train_topa.sh |
| Commonsense, LLaMA-2-7B | query, value | t=2, p=16 | 3e-5 | 16 | 3 | not stated | 100 steps (E.3 covers commonsense) | not stated | §5.2 tab. 1 app. E.3 |
| Math, LLaMA-3-8B | query, value | t=3, p=8 (block 512) | 1e-4 | 32 (derived: 1 × 8 × 4 GPUs) | 1 | cosine (code) | not stated in paper; code: 3% of steps | not stated in paper; code: 512 | §5.3 tab. 2 app. E.3 train_topa.sh |
| Math and code, Qwen3-4B | query, value | t=3, p=5 | 1e-4 | 32 | 1 | cosine | not stated | not stated | §5.3 tab. 3 app. E.3 |
| Subject-driven, Stable Diffusion v1.5 | query, key, value, output of U-Net attention | t=2, p=32 | 6e-5 | 1 | about 1,000 steps | not stated | not stated | not stated | §5.4 tab. 4 app. E.3 |
| NLU, DeBERTaV3-base | query, key, value (per E.3) | t=2, p=4 | per task, see table below | 32 | per task | not stated | not stated | per task | §5.5 tab. 5 app. E.3 |
GLUE per-task settings, DeBERTaV3-base app. E.3:
| Dataset | MNLI | SST-2 | CoLA | QQP | QNLI | RTE | MRPC | STS-B |
|---|---|---|---|---|---|---|---|---|
| Epochs | 8 | 15 | 30 | 15 | 5 | 40 | 40 | 25 |
| Learning rate | 3e-4 | 1e-4 | 4e-4 | 1e-4 | 1e-4 | 2e-5 | 3e-4 | 8e-5 |
| Max sequence length | 256 | 128 | 64 | 320 | 512 | 320 | 320 | 128 |
Other settings from the paper and code:
- Optimiser: AdamW for all tasks app. E.3. The code uses
adamw_torchwith weight decay 0 in all three scripts train_topa.py train_topa.sh. - Precision: bf16 in the commonsense and math scripts train_topa.sh.
- Block size vs p: the code takes a block size
b, notp. Withn = 4096andp = 8,b = 512train_topa.sh. - Training subset: math uses the first 100,000 rows of the split (
metamath:100000), not a random sample train_topa.py. - Not stated: the paper does not state seeds, dropout, Adam betas or epsilon, or the random-seed policy for the main tables.
Results
Author-reported. Setup: frozen pretrained backbone; adapter on the listed modules; the paper compares against LoRA, DoRA, PiSSA, HiRA, RidgeLoRA, OFT, BOFT and HRA §5.1 app. E.2. The paper says 21 NLP and CV datasets across 5 architectures; this count is consistent with §5.1 and app. E.1 (8 GLUE + 8 commonsense + 4 math/code + 1 DreamBooth). Each method's number of trainable parameters is listed beside it.
Commonsense reasoning, eight tasks, LLaMA-2-7B and LLaMA-3-8B §5.2 tab. 1
| Backbone | Approach | Trainable params | ARC-c | ARC-e | BoolQ | HellaSwag | OBQA | PIQA | SIQA | WinoGrande | Mean |
|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaMA2-7B | LoRA r=32 | 16.8M | 71.5 | 85.4 | 70.5 | 84.6 | 78.6 | 81.1 | 80.1 | 78.5 | 78.8 |
| LLaMA2-7B | DoRA r=32 | 17.0M | 71.2 | 85.6 | 70.7 | 84.8 | 78.2 | 81.3 | 79.5 | 78.4 | 78.7 |
| LLaMA2-7B | HiRA r=32 | 16.8M | 69.8 | 85.9 | 70.6 | 86.5 | 79.4 | 82.2 | 79.3 | 79.8 | 79.2 |
| LLaMA2-7B | RidgeLoRA r=32 | 17.0M | 71.5 | 87.0 | 71.2 | 87.3 | 79.8 | 83.5 | 81.1 | 82.6 | 80.5 |
| LLaMA2-7B | OFT b=64 | 16.8M | 69.9 | 86.6 | 70.5 | 86.5 | 80.4 | 83.1 | 80.2 | 82.0 | 79.9 |
| LLaMA2-7B | BOFT m=2, b=32 | 17.0M | 70.1 | 87.4 | 70.6 | 82.4 | 77.6 | 83.2 | 79.2 | 78.6 | 78.6 |
| LLaMA2-7B | HRA r=64 | 16.8M | 68.8 | 85.5 | 69.9 | 80.6 | 76.4 | 81.8 | 78.1 | 82.0 | 77.9 |
| LLaMA2-7B | ToPA t=2, p=16 | 16.7M | 72.5 | 85.6 | 71.5 | 87.7 | 83.6 | 84.5 | 80.9 | 84.5 | 81.4 |
| LLaMA3-8B | LoRA r=32 | 13.6M | 80.4 | 92.8 | 73.0 | 93.5 | 84.8 | 88.8 | 80.2 | 86.0 | 84.9 |
| LLaMA3-8B | DoRA r=32 | 13.8M | 80.9 | 93.0 | 72.9 | 93.8 | 85.6 | 88.7 | 80.1 | 86.2 | 85.2 |
| LLaMA3-8B | HiRA r=32 | 13.6M | 83.0 | 92.7 | 74.0 | 95.1 | 86.4 | 89.0 | 81.8 | 86.3 | 86.0 |
| LLaMA3-8B | RidgeLoRA r=32 | 17.0M | 81.5 | 92.9 | 73.7 | 91.7 | 88.2 | 88.3 | 81.7 | 85.2 | 85.4 |
| LLaMA3-8B | OFT b=64 | 16.9M | 82.1 | 93.2 | 74.6 | 94.3 | 87.4 | 88.5 | 81.6 | 85.6 | 85.9 |
| LLaMA3-8B | BOFT m=3, b=16 | 12.8M | 80.5 | 91.8 | 72.9 | 93.5 | 84.6 | 88.6 | 81.2 | 84.8 | 84.7 |
| LLaMA3-8B | HRA r=64 | 13.6M | 79.3 | 92.6 | 71.1 | 92.6 | 84.8 | 88.6 | 79.2 | 83.8 | 84.0 |
| LLaMA3-8B | ToPA t=3, p=8 | 12.6M | 81.5 | 92.2 | 75.1 | 94.4 | 88.4 | 89.9 | 82.1 | 86.5 | 86.3 |
Columns are sorted alphabetically by task; the paper's order is in tab. 1.
Headline: ToPA has the top average for both backbones, 81.4 (LLaMA-2-7B) and 86.3 (LLaMA-3-8B), with the fewest or near-fewest parameters. The margin over the best baseline is 0.9 (LLaMA-2: RidgeLoRA 80.5) and 0.4 (LLaMA-3: OFT 85.9). The table gives one run per method, and the seed is not stated.
Math reasoning and code generation, LLaMA-3-8B §5.3 tab. 2
| Method | # Param | GSM8K | MATH | HumanEval | MBPP |
|---|---|---|---|---|---|
| LoRA r=32 | 13.6M | 70.7 | 22.5 | 50.0 | 67.5 |
| DoRA r=32 | 13.8M | 71.1 | 22.7 | 51.2 | 66.9 |
| PiSSA r=32 | 13.6M | 72.1 | 23.0 | 50.0 | 65.1 |
| HiRA r=32 | 13.6M | 68.9 | 22.1 | 50.6 | 67.2 |
| RidgeLoRA r=32 | 16.9M | 69.8 | 21.7 | 48.2 | 65.9 |
| OFT b=64 | 17.0M | 73.8 | 22.9 | 45.1 | 66.1 |
| BOFT m=3, b=16 | 12.8M | 69.4 | 22.4 | 45.1 | 67.5 |
| ToPA t=3, p=8 | 12.6M | 74.7 | 23.5 | 51.8 | 67.2 |
Math reasoning and code generation, Qwen3-4B §5.3 tab. 3
| Method | # Param | GSM8K | MATH | HumanEval | MBPP |
|---|---|---|---|---|---|
| LoRA r=16 | 5.9M | 82.6 | 47.0 | 69.5 | 71.2 |
| DoRA r=16 | 5.9M | 83.2 | 46.6 | 68.9 | 72.2 |
| PiSSA r=16 | 5.9M | 83.5 | 47.4 | 70.1 | 73.0 |
| HiRA r=16 | 5.9M | 83.6 | 46.0 | 67.7 | 71.2 |
| RidgeLoRA r=16 | 5.9M | 83.7 | 46.5 | 66.5 | 71.2 |
| OFT b=32 | 6.1M | 83.8 | 47.6 | 69.5 | 73.8 |
| BOFT m=2, b=16 | 6.1M | 83.1 | 47.8 | 67.7 | 74.6 |
| ToPA t=3, p=5 | 5.5M | 84.3 | 48.2 | 73.2 | 73.5 |
Headline (LLaMA-3-8B): ToPA leads GSM8K (+0.9 over OFT, the best baseline at 73.8) and HumanEval (+0.6 over DoRA at 51.2). On MBPP it trails BOFT and LoRA (67.5) by 0.3. The paper calls the MBPP gap marginal §5.3.
Multi-seed check, LLaMA-3-8B, three seeds app. F (the table lists mean ± spread; the paper does not state whether the spread is a standard deviation):
| Method | # Param | GSM8K | MATH | HumanEval | MBPP |
|---|---|---|---|---|---|
| LoRA r=32 | 13.6M | 70.5 ± 0.4 | 22.5 ± 0.3 | 50.2 ± 0.3 | 67.2 ± 0.3 |
| DoRA r=32 | 13.8M | 71.1 ± 0.3 | 22.8 ± 0.2 | 51.0 ± 0.3 | 67.0 ± 0.2 |
| OFT b=64 | 17.0M | 73.4 ± 0.3 | 22.9 ± 0.2 | 45.7 ± 0.5 | 66.5 ± 0.2 |
| BOFT m=3, b=16 | 12.8M | 69.4 ± 0.4 | 22.3 ± 0.3 | 45.5 ± 0.3 | 67.1 ± 0.2 |
| ToPA t=3, p=8 | 12.6M | 74.9 ± 0.3 | 23.5 ± 0.2 | 51.6 ± 0.3 | 67.2 ± 0.3 |
Subject-driven generation, Stable Diffusion v1.5, DreamBooth §5.4 tab. 4 fig. 3
| Method | # Param | DINO ↑ | CLIP-I ↑ | CLIP-T ↑ | LPIPS ↑ |
|---|---|---|---|---|---|
| LoRA r=64 | 12.8M | 0.654 | 0.793 | 0.230 | 0.743 |
| DoRA r=64 | 12.8M | 0.654 | 0.791 | 0.231 | 0.740 |
| PiSSA r=64 | 12.8M | 0.634 | 0.782 | 0.228 | 0.739 |
| HiRA r=64 | 12.8M | 0.668 | 0.794 | 0.226 | 0.736 |
| RidgeLoRA r=64 | 12.8M | 0.657 | 0.791 | 0.222 | 0.737 |
| OFT b=64 | 23.3M | 0.653 | 0.793 | 0.225 | 0.735 |
| BOFT m=2, b=32 | 23.4M | 0.664 | 0.796 | 0.225 | 0.734 |
| ToPA t=2, p=32 | 11.5M | 0.676 | 0.796 | 0.234 | 0.740 |
The paper says ToPA is best on DINO, CLIP-I and CLIP-T and marginally behind LoRA on LPIPS (0.740 vs 0.743), which it reads as a small loss in sample diversity §5.4.
Natural language understanding, GLUE, DeBERTaV3-base §5.5 tab. 5
| Method | # Param | MNLI | SST-2 | CoLA | QQP | QNLI | RTE | MRPC | STS-B | Avg |
|---|---|---|---|---|---|---|---|---|---|---|
| LoRA r=8 | 0.44M | 89.92 | 94.72 | 69.31 | 91.68 | 93.99 | 84.48 | 89.46 | 91.12 | 88.09 |
| DoRA r=8 | 0.47M | 89.67 | 94.50 | 68.86 | 91.55 | 94.23 | 85.92 | 89.46 | 91.56 | 88.22 |
| PiSSA r=8 | 0.44M | 89.90 | 95.07 | 68.52 | 91.78 | 93.98 | 86.64 | 89.22 | 91.11 | 88.28 |
| HiRA r=8 | 0.44M | 90.20 | 95.64 | 69.80 | 91.08 | 93.26 | 87.00 | 89.22 | 91.45 | 88.46 |
| RidgeLoRA r=8 | 0.47M | 89.74 | 95.18 | 68.82 | 91.57 | 93.98 | 85.20 | 88.97 | 91.12 | 88.07 |
| OFT b=16 | 0.44M | 89.55 | 94.38 | 68.97 | 91.25 | 93.56 | 84.48 | 88.73 | 91.24 | 87.77 |
| BOFT m=2, b=8 | 0.47M | 89.46 | 94.27 | 68.12 | 91.32 | 93.63 | 85.20 | 88.97 | 90.98 | 87.74 |
| HRA r=16 | 0.44M | 90.24 | 95.41 | 70.80 | 91.24 | 93.70 | 85.20 | 89.71 | 91.11 | 88.43 |
| ToPA t=2, p=4 | 0.44M | 90.15 | 95.18 | 70.88 | 91.85 | 94.05 | 86.28 | 89.95 | 91.68 | 88.75 |
The paper reports a gain of 0.29 on the average over the best baseline (HiRA, 88.46) and first or second place on eight tasks §5.5.
Scaling and ablations (LLaMA-3-8B, math, text only)
- Scale vs LoRA: at matched budgets with
t = 3andp ∈ {2, 4, 8}, ToPA beats LoRA atr ∈ {8, 16, 32}on GSM8K and MATH §5.6 fig. 6. The values are only in the figure. - Block count
p: witht = 3fixed, accuracy rises withp ∈ {1, 2, 4, 8}on the three commonsense tasks the authors plot §5.7 fig. 4. - Chain length
t: withp = 8fixed, accuracy rises fromt = 1tot = 3. Att = 4it drops slightly on OBQA and PIQA, which the authors attribute to optimisation instability §5.7 fig. 4. - Identity residual: the authors report that including the identity matrix improves results on every commonsense task, on each backbone §5.8 fig. 5.
Paper vs code
Repository: https://github.com/scongl/ToPA, commit 85c3c1feaead270ddb0ec9dc10f374aaef6443e8 (one commit, 2026-10-14). The paper does not link it. The owner is the personal account scongl, and the README's author list matches the paper's. The sole commit is by sicongli, the first author. The card treats the repository as official on that basis. Verify the attribution before relying on it. The repository has no LICENSE file, so no code is quoted here.
The paper's checklist says the complete codebase will be released if the paper is accepted [checklist.tex]. The repository here may therefore not be the final release, and its scope is narrower than the paper's experiments.
- Notation
pvsblock_size. The paper usesp(blocks per dimension) §3.3. The code takesblock_size = n/pconfig.py. The paper'sp = 8on a 4096-wide layer isblock_size = 512in the code train_topa.sh. Consistent, but easy to mix up. - Commonsense hyperparameters match. The paper's lr 3e-5, warm-up 100, 3 epochs, batch 16 app. E.3 are the script values train_topa.sh. The script's effective batch of 16 is 1 × 4 × 4 GPUs, derived from the script train_topa.sh train_topa.sh.
- Math hyperparameters match. lr 1e-4, batch 32, one epoch, cosine app. E.3 match the script train_topa.sh. The script adds a 3% warm-up ratio train_topa.sh, which the paper does not state.
- Code-generation data differs. The paper trains code generation on CodeFeedback §5.1. The script trains on sub-task
pythonfrom a dataset directory namedpissa-datasettrain_topa.sh train_topa.sh. The repository does not map one to the other, so this cannot be checked. - Initialisation constant. The paper leaves
σopen §3.4. The code sets it to 0.02 for every factor except factor 0 layer_block_toeplitz.py. - Default values differ between files. The PEFT config defaults to
toep_r = 4,block_size = 128config.py. The training script defaults totoep_r = 4,block_size = 64train_topa.py. The shell scripts override both. - Eval decoding. The paper says HumanEval pass@1 is measured "across multiple samples" app. E.1. The code generates greedily: temperature 0, top-p 1, max 1024 new tokens gen_vllm.py. If you reproduce the table, use greedy decoding.
- Single-run vs multi-seed numbers. The main math table reports GSM8K 74.7 for ToPA. The three-seed table reports 74.9 ± 0.3 app. F. The paper does not say which seed the main table uses.
- Not in the repository. The GLUE, DreamBooth/Stable Diffusion, LLaMA-2 and Qwen runs have no scripts. The repository's
peftpackage is a modified copy with ToPA added, and it pinspeft==0.14.0requirements.txt. - Untested paper claims. The paper states the expressiveness theorem with an
O(n)construction and does not verify it in code. The code has no test for it.
Pitfalls
- Ambiguous factor order. Eq. (8) writes the product
∏ T_blk^iwithout saying how it acts on a row or column vector. The code applies the highest-index factor first to each input row layer_block_toeplitz.py. Check the order numerically against the reference before porting; the two conventions give different functions for non-commuting factors. - The zero factor gives
W = W0at step 0. A zero factor makes the whole product zero. If you init every factor with noise, the adapter starts away from the pretrained model. The paper's "rank is preserved almost surely" result §4.1 concerns generic parameters, not step 0, whereΔW = 0. - Unstated
σand warm-up. Only the code'sσ = 0.02is known. Other values will change convergence, and the paper's Fig. 2 instability is fort = 16only. - Truncation. The math run truncates to 512 tokens and the commonsense run to 256 train_topa.sh train_topa.sh. The paper does not state either limit. Longer examples lose their tails silently.
- Training data subset.
metamath:100000takes the first 100,000 rows of the split, not a random sample train_topa.py. The paper says "a 100k subset" §5.1. - Loss masking. The code masks prompt tokens with
-100and appends EOS to targets train_topa.py train_topa.py. The paper does not state this. The Alpaca-style prompt template is also code-only train_topa.py. - FFT in fp32. Under bf16 weights the code runs the FFT in fp32 layer_block_toeplitz.py. If you cast the whole path to bf16, expect numerical differences.
- Dense merge is costly.
merge()builds a densein × inmatrix by pushing the identity through the chain layer_block_toeplitz.py, so memory scales within². The scripts merge after training (--merge True) train_topa.py. - Unmerge is not implemented.
unmerge()raisesNotImplementedErrorlayer_block_toeplitz.py. Once merged, an adapter cannot be removed. - No scaling.
scale_layerandunscale_layeronly warn and keep the scale at 1 layer_block_toeplitz.py. Adapter-strength tricks used with LoRA do not transfer. - Divisibility.
in_featuresmust be divisible byblock_size, or construction raises an error layer_block_toeplitz.py. Any module you add must respect this. - Parameter totals. Both the paper and the code count one
n × nfactor per adapted layer, applied to the layer's input app. C layer_block_toeplitz.py. The 12.6M in the LLaMA-3-8B row is consistent with that: 2 modules × 32 layers × 3 factors × 65,472 numbers ≈ 12.57M [derived; the 32 layers and hidden size 4096 are model config values, not stated in the paper]. - Cost is analytic only. The paper reports no measured runtime or memory. Budget the FFT overhead yourself. Its published cost is
O(t p n log q)per layer app. C. - Non-square layers. The code applies
Ron the input side and sums with the input before the frozen weight, so it works forout ≠ in. The paper's cost analysis assumes squareW0app. C. - Statistics. The main tables are single-run. The multi-seed table covers only LLaMA-3-8B on math and code app. F. Gains on the commonsense and GLUE tables are below 1 point and may be within run-to-run noise.
Lineage
- Builds on: LoRA (low-rank adapters) §2.1; OFT and BOFT (sparse orthogonal adapters) §2.2 app. B; the Toeplitz decomposition theorem, used for the expressiveness claim §3.1.
- Compared against: LoRA, DoRA, PiSSA, HiRA, RidgeLoRA, OFT, BOFT and HRA §5.1 app. E.2.
- Superseded by: not stated.
Source and license
Adapted from “ToPA: Block-wise Toeplitz Adaptation for Expressive and Efficient Fine-Tuning” by Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Longtao Huang, Xiaochun Cao, Qingming Huang (arXiv:2610.22883), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/scongl/ToPA at commit 85c3c1feaead (none (no LICENSE file in repository)). Any quoted code is shown under that license.