---
kind: concept
slug: structured-matrix-parameterization
title: Structured matrix parameterization
summary: >-
  Adapt a frozen layer with an update built from structured matrices, here a chain of block-wise Toeplitz factors plus an identity residual, instead of a low-rank or sparse update. Each factor stores far fewer numbers than a dense matrix but is not limited to a fixed rank or sparsity pattern. ToPA is the only variant so far; its gains are author-reported and single-run on most tables.
tags: [fine-tuning, parameter-efficient, structured-matrices]
papers: ["2610.22883"]
related: [parameter-efficient-fine-tuning, low-rank-adaptation, sparse-fine-tuning]
status: draft
updated_at: 2026-11-22
---

## Overview

The idea is to write a weight update as a product of matrices that each have a
cheap, fixed structure, so the update has far fewer free numbers than a dense
matrix. The frozen weight `W0` stays as it is, and the adapter is a structured
matrix chain sitting next to it. The invariant shared by this family is that
the adapter starts from the pre-trained model: the first factor is zero, so the
update is zero at step 0 [arxiv:2610.22883 §3.4] [eq. 9].

Low-rank adapters bound the rank of the update, and sparse orthogonal adapters
fix a sparsity pattern in advance. Structured parameterisations aim to avoid
both limits while keeping the parameter count low [arxiv:2610.22883 §2.1]
[§2.2]. Only one paper on this page so far, so the design space shown here is
the one that paper explores.

## Variants

### Block-wise Toeplitz chain (ToPA)

A Toeplitz matrix is constant along each diagonal. An `n × n` Toeplitz matrix is
set by `2n − 1` numbers, and its product with a vector can be computed with an
FFT in `O(n log n)` [arxiv:2610.22883 §3.1] [eq. 4] [app. D]. ToPA splits the
layer into `p` equal blocks per dimension. Each of the `p × p` sub-blocks is an
independent Toeplitz matrix, so one block-wise factor holds `p(2n − p)` numbers
[arxiv:2610.22883 §3.3] [eq. 7] [app. C]. Several such factors are chained and
an identity matrix is added, giving `W = W0 (∏ T_blk + I)` [arxiv:2610.22883
§3.3] [eq. 8]. Card: [ToPA](/papers/2610.22883).

The authors use the block layout to keep the chain short. Their argument is
that shorter chains give fewer factors in the gradient product, which they
link to training stability [arxiv:2610.22883 §3.2] [eq. 6]. Without blocks, a
plain Toeplitz chain of length 16 on LLaMA-3-8B shows large gradient-norm and
loss swings and collapses in the authors' run [arxiv:2610.22883 §3.2] [fig. 2].

The paper also proves that block-wise chains can represent a given update, with
a chain length of roughly `⌊n/(2p)⌋ + 1` for generic updates and at most about
`2n/p + 5` in all cases. These are existence results, not a statement about
what training reaches [arxiv:2610.22883 §4.2].

## When to use which

There is one variant on this page, so the table compares the structured
adapter with the two families it is meant to replace, using only the paper's
own constraints.

| Situation | Use | Why |
|---|---|---|
| You need an update that is not rank-limited or pattern-limited, under a tight parameter budget | Block-wise Toeplitz chain | The update is dense and generically full rank [arxiv:2610.22883 §4.1] [eq. 10]; the paper's commonsense run uses 12.6M parameters against 13.6M for LoRA r=32 [arxiv:2610.22883 tab. 1] |
| You can afford extra FFT work per layer and need no measured speed numbers | Block-wise Toeplitz chain, with a benchmark of your own | Cost is given only analytically, `O(t p n log q)` per layer; no wall-clock or memory is reported [arxiv:2610.22883 app. C] |
| You must merge the adapter and later remove it | Not ToPA as released | Merging builds a dense `in × in` matrix, so memory grows with `in²`, and `unmerge()` is not implemented [code: peft/tuners/topa/layer_block_toeplitz.py#L195-L205] [code: peft/tuners/topa/layer_block_toeplitz.py#L256-L261] |
| You need adapter-strength scaling of the LoRA kind | Not ToPA as released | Scaling only warns and keeps the scale at 1 [code: peft/tuners/topa/layer_block_toeplitz.py#L93-L108] |
| Your layer width is not divisible by the block size | Not ToPA as released | Construction raises an error [code: peft/tuners/topa/layer_block_toeplitz.py#L60-L61] |
| The gap to the best baseline is under one point, from a single run | Treat the gain as unconfirmed | Commonsense and GLUE tables are single runs, and the three-seed check covers only math and code on LLaMA-3-8B [arxiv:2610.22883 app. F] |

Reported accuracy, author-reported and single-run unless noted: on eight
commonsense tasks, the mean is 86.3 for ToPA on LLaMA-3-8B against 85.9 for
OFT, the best baseline in that table [arxiv:2610.22883 tab. 1]. On GSM8K with
LLaMA-3-8B, ToPA scores 74.7 against 73.8 for OFT [arxiv:2610.22883 tab. 2];
the three-seed mean is 74.9 ± 0.3 [arxiv:2610.22883 app. F]. On GLUE with
DeBERTaV3-base, the average is 88.75 against 88.46 for HiRA [arxiv:2610.22883
tab. 5].

## Implementation sketch

Language-agnostic core of one adapted linear layer, with the points where the
reference code and the paper differ noted in the comments.

```pseudocode
# Structured-matrix adapter for one frozen linear layer  [§3.3] [eq. 8]
# x: (batch, n) activations; W0: (m, n) frozen weight
# factors: list of t structured factors, each a p x p grid of Toeplitz blocks

def adapted_forward(x, W0, factors):
    # Apply each structured factor to the activations. The paper's equation
    # does not fix the order; the reference code applies them in a fixed
    # order, so check it numerically before porting
    # [code: peft/tuners/topa/layer_block_toeplitz.py#L214-L217]
    h = x
    for T in factors:
        h = apply_block_toeplitz(T, h)
    # Identity residual, then the frozen weight  [eq. 8]
    return (h + x) @ W0.T

def init_factors(t, p, n, sigma):
    # First factor zero, so the update is zero at step 0  [§3.4] [eq. 9]
    # The paper leaves sigma unstated; the reference code uses 0.02
    # [code: peft/tuners/topa/layer_block_toeplitz.py#L84-L90]
    first = zeros(p, p, 2 * (n // p) - 1)
    rest = [normal(0, sigma, (p, p, 2 * (n // p) - 1)) for _ in range(t - 1)]
    return [first] + rest
```

Applying one block-wise factor is a sum of per-block circular convolutions
computed with the FFT [app. D]. The reference code embeds each kernel in a
circular buffer whose length is the smallest power of two that holds the
kernel [code: peft/tuners/topa/layer_block_toeplitz.py#L163-L169]; the paper
does not state this.

## Contradictions

None known across papers, since only one card is on this page. Within the one
paper, the main math table reports ToPA at 74.7 on GSM8K and the three-seed
table at 74.9 ± 0.3. The paper does not say which seed the main table uses
[arxiv:2610.22883 tab. 2] [app. F].

## Papers

- [ToPA](/papers/2610.22883): block-wise Toeplitz factor chains with an identity residual as a PEFT adapter, with analytic cost and author-reported gains over LoRA, OFT and BOFT.
