Concept · 1 paper · updated 2026-11-22
Structured matrix parameterization
Adapt a frozen layer with an update built from structured matrices, here a chain of block-wise Toeplitz factors plus an identity residual, instead of a low-rank or sparse update. Each factor stores far fewer numbers than a dense matrix but is not limited to a fixed rank or sparsity pattern. ToPA is the only variant so far; its gains are author-reported and single-run on most tables.
fine-tuningparameter-efficientstructured-matrices
Markdown4 min read
AI-compiled overview, updated 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
Overview
The idea is to write a weight update as a product of matrices that each have a
cheap, fixed structure, so the update has far fewer free numbers than a dense
matrix. The frozen weight W0 stays as it is, and the adapter is a structured
matrix chain sitting next to it. The invariant shared by this family is that
the adapter starts from the pre-trained model: the first factor is zero, so the
update is zero at step 0 2610.22883 §3.4 eq. 9.
Low-rank adapters bound the rank of the update, and sparse orthogonal adapters fix a sparsity pattern in advance. Structured parameterisations aim to avoid both limits while keeping the parameter count low 2610.22883 §2.1 §2.2. Only one paper on this page so far, so the design space shown here is the one that paper explores.
Variants
Block-wise Toeplitz chain (ToPA)
A Toeplitz matrix is constant along each diagonal. An n × n Toeplitz matrix is
set by 2n − 1 numbers, and its product with a vector can be computed with an
FFT in O(n log n) 2610.22883 §3.1 eq. 4 app. D. ToPA splits the
layer into p equal blocks per dimension. Each of the p × p sub-blocks is an
independent Toeplitz matrix, so one block-wise factor holds p(2n − p) numbers
2610.22883 §3.3 eq. 7 app. C. Several such factors are chained and
an identity matrix is added, giving W = W0 (∏ T_blk + I) [arxiv:2610.22883
§3.3] eq. 8. Card: ToPA.
The authors use the block layout to keep the chain short. Their argument is that shorter chains give fewer factors in the gradient product, which they link to training stability 2610.22883 §3.2 eq. 6. Without blocks, a plain Toeplitz chain of length 16 on LLaMA-3-8B shows large gradient-norm and loss swings and collapses in the authors' run 2610.22883 §3.2 fig. 2.
The paper also proves that block-wise chains can represent a given update, with
a chain length of roughly ⌊n/(2p)⌋ + 1 for generic updates and at most about
2n/p + 5 in all cases. These are existence results, not a statement about
what training reaches 2610.22883 §4.2.
When to use which
There is one variant on this page, so the table compares the structured adapter with the two families it is meant to replace, using only the paper's own constraints.
| Situation | Use | Why |
|---|---|---|
| You need an update that is not rank-limited or pattern-limited, under a tight parameter budget | Block-wise Toeplitz chain | The update is dense and generically full rank 2610.22883 §4.1 eq. 10; the paper's commonsense run uses 12.6M parameters against 13.6M for LoRA r=32 2610.22883 tab. 1 |
| You can afford extra FFT work per layer and need no measured speed numbers | Block-wise Toeplitz chain, with a benchmark of your own | Cost is given only analytically, O(t p n log q) per layer; no wall-clock or memory is reported 2610.22883 app. C |
| You must merge the adapter and later remove it | Not ToPA as released | Merging builds a dense in × in matrix, so memory grows with in², and unmerge() is not implemented layer_block_toeplitz.py layer_block_toeplitz.py |
| You need adapter-strength scaling of the LoRA kind | Not ToPA as released | Scaling only warns and keeps the scale at 1 layer_block_toeplitz.py |
| Your layer width is not divisible by the block size | Not ToPA as released | Construction raises an error layer_block_toeplitz.py |
| The gap to the best baseline is under one point, from a single run | Treat the gain as unconfirmed | Commonsense and GLUE tables are single runs, and the three-seed check covers only math and code on LLaMA-3-8B 2610.22883 app. F |
Reported accuracy, author-reported and single-run unless noted: on eight commonsense tasks, the mean is 86.3 for ToPA on LLaMA-3-8B against 85.9 for OFT, the best baseline in that table 2610.22883 tab. 1. On GSM8K with LLaMA-3-8B, ToPA scores 74.7 against 73.8 for OFT 2610.22883 tab. 2; the three-seed mean is 74.9 ± 0.3 2610.22883 app. F. On GLUE with DeBERTaV3-base, the average is 88.75 against 88.46 for HiRA [arxiv:2610.22883 tab. 5].
Implementation sketch
Language-agnostic core of one adapted linear layer, with the points where the reference code and the paper differ noted in the comments.
# Structured-matrix adapter for one frozen linear layer [§3.3] [eq. 8]
# x: (batch, n) activations; W0: (m, n) frozen weight
# factors: list of t structured factors, each a p x p grid of Toeplitz blocks
def adapted_forward(x, W0, factors):
# Apply each structured factor to the activations. The paper's equation
# does not fix the order; the reference code applies them in a fixed
# order, so check it numerically before porting
# [code: peft/tuners/topa/layer_block_toeplitz.py#L214-L217]
h = x
for T in factors:
h = apply_block_toeplitz(T, h)
# Identity residual, then the frozen weight [eq. 8]
return (h + x) @ W0.T
def init_factors(t, p, n, sigma):
# First factor zero, so the update is zero at step 0 [§3.4] [eq. 9]
# The paper leaves sigma unstated; the reference code uses 0.02
# [code: peft/tuners/topa/layer_block_toeplitz.py#L84-L90]
first = zeros(p, p, 2 * (n // p) - 1)
rest = [normal(0, sigma, (p, p, 2 * (n // p) - 1)) for _ in range(t - 1)]
return [first] + restApplying one block-wise factor is a sum of per-block circular convolutions computed with the FFT app. D. The reference code embeds each kernel in a circular buffer whose length is the smallest power of two that holds the kernel layer_block_toeplitz.py; the paper does not state this.
Contradictions
None known across papers, since only one card is on this page. Within the one paper, the main math table reports ToPA at 74.7 on GSM8K and the three-seed table at 74.9 ± 0.3. The paper does not say which seed the main table uses 2610.22883 tab. 2 app. F.
Papers
- ToPA: block-wise Toeplitz factor chains with an identity residual as a PEFT adapter, with analytic cost and author-reported gains over LoRA, OFT and BOFT.
Source and license
Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.