wikillm

Concept · 1 paper · updated 2026-11-22

Structured matrix parameterization

Adapt a frozen layer with an update built from structured matrices, here a chain of block-wise Toeplitz factors plus an identity residual, instead of a low-rank or sparse update. Each factor stores far fewer numbers than a dense matrix but is not limited to a fixed rank or sparsity pattern. ToPA is the only variant so far; its gains are author-reported and single-run on most tables.

fine-tuningparameter-efficientstructured-matrices

Markdown4 min read

AI-compiled overview, updated 2026-11-22, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

The idea is to write a weight update as a product of matrices that each have a cheap, fixed structure, so the update has far fewer free numbers than a dense matrix. The frozen weight W0 stays as it is, and the adapter is a structured matrix chain sitting next to it. The invariant shared by this family is that the adapter starts from the pre-trained model: the first factor is zero, so the update is zero at step 0 2610.22883 §3.4 eq. 9.

Low-rank adapters bound the rank of the update, and sparse orthogonal adapters fix a sparsity pattern in advance. Structured parameterisations aim to avoid both limits while keeping the parameter count low 2610.22883 §2.1 §2.2. Only one paper on this page so far, so the design space shown here is the one that paper explores.

Variants

Block-wise Toeplitz chain (ToPA)

A Toeplitz matrix is constant along each diagonal. An n × n Toeplitz matrix is set by 2n − 1 numbers, and its product with a vector can be computed with an FFT in O(n log n) 2610.22883 §3.1 eq. 4 app. D. ToPA splits the layer into p equal blocks per dimension. Each of the p × p sub-blocks is an independent Toeplitz matrix, so one block-wise factor holds p(2n − p) numbers 2610.22883 §3.3 eq. 7 app. C. Several such factors are chained and an identity matrix is added, giving W = W0 (∏ T_blk + I) [arxiv:2610.22883 §3.3] eq. 8. Card: ToPA.

The authors use the block layout to keep the chain short. Their argument is that shorter chains give fewer factors in the gradient product, which they link to training stability 2610.22883 §3.2 eq. 6. Without blocks, a plain Toeplitz chain of length 16 on LLaMA-3-8B shows large gradient-norm and loss swings and collapses in the authors' run 2610.22883 §3.2 fig. 2.

The paper also proves that block-wise chains can represent a given update, with a chain length of roughly ⌊n/(2p)⌋ + 1 for generic updates and at most about 2n/p + 5 in all cases. These are existence results, not a statement about what training reaches 2610.22883 §4.2.

When to use which

There is one variant on this page, so the table compares the structured adapter with the two families it is meant to replace, using only the paper's own constraints.

SituationUseWhy
You need an update that is not rank-limited or pattern-limited, under a tight parameter budgetBlock-wise Toeplitz chainThe update is dense and generically full rank 2610.22883 §4.1 eq. 10; the paper's commonsense run uses 12.6M parameters against 13.6M for LoRA r=32 2610.22883 tab. 1
You can afford extra FFT work per layer and need no measured speed numbersBlock-wise Toeplitz chain, with a benchmark of your ownCost is given only analytically, O(t p n log q) per layer; no wall-clock or memory is reported 2610.22883 app. C
You must merge the adapter and later remove itNot ToPA as releasedMerging builds a dense in × in matrix, so memory grows with in², and unmerge() is not implemented layer_block_toeplitz.py layer_block_toeplitz.py
You need adapter-strength scaling of the LoRA kindNot ToPA as releasedScaling only warns and keeps the scale at 1 layer_block_toeplitz.py
Your layer width is not divisible by the block sizeNot ToPA as releasedConstruction raises an error layer_block_toeplitz.py
The gap to the best baseline is under one point, from a single runTreat the gain as unconfirmedCommonsense and GLUE tables are single runs, and the three-seed check covers only math and code on LLaMA-3-8B 2610.22883 app. F

Reported accuracy, author-reported and single-run unless noted: on eight commonsense tasks, the mean is 86.3 for ToPA on LLaMA-3-8B against 85.9 for OFT, the best baseline in that table 2610.22883 tab. 1. On GSM8K with LLaMA-3-8B, ToPA scores 74.7 against 73.8 for OFT 2610.22883 tab. 2; the three-seed mean is 74.9 ± 0.3 2610.22883 app. F. On GLUE with DeBERTaV3-base, the average is 88.75 against 88.46 for HiRA [arxiv:2610.22883 tab. 5].

Implementation sketch

Language-agnostic core of one adapted linear layer, with the points where the reference code and the paper differ noted in the comments.

Pseudocode
# Structured-matrix adapter for one frozen linear layer  [§3.3] [eq. 8]
# x: (batch, n) activations; W0: (m, n) frozen weight
# factors: list of t structured factors, each a p x p grid of Toeplitz blocks

def adapted_forward(x, W0, factors):
    # Apply each structured factor to the activations. The paper's equation
    # does not fix the order; the reference code applies them in a fixed
    # order, so check it numerically before porting
    # [code: peft/tuners/topa/layer_block_toeplitz.py#L214-L217]
    h = x
    for T in factors:
        h = apply_block_toeplitz(T, h)
    # Identity residual, then the frozen weight  [eq. 8]
    return (h + x) @ W0.T

def init_factors(t, p, n, sigma):
    # First factor zero, so the update is zero at step 0  [§3.4] [eq. 9]
    # The paper leaves sigma unstated; the reference code uses 0.02
    # [code: peft/tuners/topa/layer_block_toeplitz.py#L84-L90]
    first = zeros(p, p, 2 * (n // p) - 1)
    rest = [normal(0, sigma, (p, p, 2 * (n // p) - 1)) for _ in range(t - 1)]
    return [first] + rest

Applying one block-wise factor is a sum of per-block circular convolutions computed with the FFT app. D. The reference code embeds each kernel in a circular buffer whose length is the smallest power of two that holds the kernel layer_block_toeplitz.py; the paper does not state this.

Contradictions

None known across papers, since only one card is on this page. Within the one paper, the main math table reports ToPA at 74.7 on GSM8K and the three-seed table at 74.9 ± 0.3. The paper does not say which seed the main table uses 2610.22883 tab. 2 app. F.

Papers

  • ToPA: block-wise Toeplitz factor chains with an identity residual as a PEFT adapter, with analytic cost and author-reported gains over LoRA, OFT and BOFT.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.