wikillm

Paper · 2026-10-06 · cs.LG, cs.AI

MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models

MaRK conditions a frozen 111M Hydra-BERT state space model on the diffusion timestep by adding bounded, low-rank modulations to its A, B, C, D and dt parameters, in three geometries (Hypernet, Chebyshev, DCT). Author-reported CART validation loss averages 2.55 (Chebyshev), 2.59 (DCT) and 3.77 (Hypernet) with 6.3-11M trainable adapter parameters; latency rises 14-16% at length 4096.

State-space model (SSM)Discrete diffusionHypernetworkParameter-efficient fine-tuningstate-space-modelsdiffusion-language-modelsadapters

arXivPDFCodeMarkdown15 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Many modern sequence models carry a fixed-size memory through the input, and their behaviour is set by a few learned matrices. A common way to make such a model follow a changing signal, such as the noise level in an image or text generator, is to feed that signal in as extra input or to rescale activations around the model. This paper changes the model's own memory dynamics instead. A small side network reads the signal and nudges the model's internal parameters by a limited amount.

The base model stays frozen. Only the small side network is trained, so the adaptation costs a few million parameters rather than a full copy of the model. The authors test three ways of shaping those nudges. The one that works best in their language-model experiments uses a smooth set of basis curves, which keeps the adjustments from changing sharply between neighbouring signal values.

The authors report that the method is stable under a stated mathematical condition and that it adapts a pretrained model to a masked-token generation objective. They present their evidence as early and limited. It does not compare quality against other conditioning methods at equal training cost.

TL;DR

MaRK keeps a pretrained 23-layer Hydra-BERT state space model (111M parameters) frozen and trains a per-layer adapter. The adapter maps the diffusion timestep to bounded adjustments of five parameter groups, A, B, C, D and the step size dt, all inside the recurrence §1 §3.4. Three adapter geometries are compared: Hypernet, Chebyshev polynomial and DCT §3.4. The Chebyshev adapter has the lowest average CART validation loss, 2.55, against 2.59 for DCT and 3.77 for Hypernet tab. 1. The results are author-reported, measured on internal validation splits, and compare only the three adapters with each other, not with AdaLN or input injection in quality app. A.5.

Problem

Conditioning a pretrained state space model on a diffusion timestep usually goes through input injection (concatenating the context to the input) or adaptive layer normalisation. Neither changes the state-space parameters A, B, C and dt, so the model's temporal behaviour is fixed while only its inputs or activations change §1. The paper's argument is that conditioning should reach the input-output map itself, which the Markov parameter sequence describes §2. The authors name low-rank adaptation (LoRA) as the static low-rank alternative that their dynamic one is compared against in motivation §1. The method assumes the conditioning variable is continuous and ordered, such as the diffusion timestep. The authors state that they make no claim for autoregressive next-token modelling or for discrete prompt conditioning app. A.4.

Method

Backbone. The base model is a Hydra-BERT model sized like BERT, with about 111M parameters across 23 layers. Its mixer is bidirectional and quasi-separable, a Mamba-style design §3.1. The backbone and its masked-LM head are frozen for every reported experiment app. C.1.

Objective. Training is masked discrete diffusion. A sequence is corrupted by masking each position with probability t, and the model predicts masked tokens from full bidirectional context §3.2 eq. 12. The loss is the negative log-likelihood on masked positions only eq. 13. Each masked token is weighted by CART, a per-token weight that grows with the amount of visible context near it §3.3 app. A.3.

Adapter. Each of the 23 Hydra layers gets its own adapter [code: src/mark.py#L588]. An MLP reads the timestep embedding and produces a hidden state h. Diagonal parameters receive bounded shifts, and dense B and C receive bounded scales and shifts, each in the form of eq. 3 and eq. 4:

\Theta' = \Theta + \beta_{\Theta}\tanh\big(\psi_{\text{shift}}(c)\big) eq. 3

\Phi' = \Phi \odot \big[1 + \alpha_{\Phi}\tanh\big(\psi_{\text{scale}}(c)\big)\big] + \beta_{\Phi}\tanh\big(\psi_{\text{shift}}(c)\big) eq. 4

Here Θ is {A_log, dt bias, D} and Φ is {B, C}. The B and C modulations are low-rank: each factor pair U and V (rank r = 2) is produced from h, and the matrix U times V gives the flattened modulation §3.4 app. A.2.

Stability. The recurrence is written as A = −exp(A_log'), with discrete step dt = softplus(dt_input + dt_bias'), and the discrete eigenvalue Ā = exp(A·Δ) eq. 8 eq. 9 eq. 10. Each eigenvalue therefore lies in (0, 1) for any finite modulation. The authors use this with P = I to give a closed-form Affine Quadratic Stability check §2.1 §4.1 app. B.1 alg. 4. The margin is ε = 1 − max ², and the authors check it at the least stable vertex with Δ_min = 0.001 app. C.4. The proof and the margin rely on Δ having a lower bound, which the modulation does not provide (see Pitfalls).

Three geometries. The adapters differ in how h becomes shifts §3.4:

  • Hypernet uses linear projections from h to all shifts and to the low-rank factors alg. 1. It is the least constrained baseline.
  • Chebyshev maps h to a latent coordinate z and to per-parameter coefficients on the Chebyshev polynomials T_k(z). The shifts are sums of coefficient times T_k(z), and the B and C factors add a base term to a basis term weighted by the mean basis value across heads eq. 16 eq. 17 alg. 2.
  • DCT builds a length-256 signal from h by weighting 8 frequencies with a learned decay 1/(f+1)^(softplus(α)+ε) and multiplying by an orthonormal DCT-II basis. The shifts and low-rank factors are linear in that signal eq. 18 eq. 19 eq. 20 alg. 3.

The authors describe the three as different parameterisations of one bounded operator map. Chebyshev and DCT impose smoothness across timesteps through the basis; Hypernet has no such prior app. A.2 §4.4.

Pseudocode
# Chebyshev MaRK adapter for one Hydra layer  [alg. 2] [app. A.2]
# Input: cond (128,); base A_log (H,), B and C (N, d), dt (H,), D (H,)
# Output: modulated A_log, B, C, dt, D

def chebyshev_adapter(cond, A_log, B, C, dt, D):
    # Hidden state from the timestep embedding: four Linear + SiLU layers
    # [app. A.2] [code: src/mark.py#L181-L190]
    h = mlp(cond)                                    # (256,)

    # Latent coordinate for the basis; not bounded to [-1, 1] in the code
    # [app. A.2] [code: src/mark.py#L268]
    z = z_proj(h)                                    # (H,)

    # Chebyshev basis T_0 .. T_{K-1} at z, with K = 5 functions
    # [alg. 2] [code: src/mark.py#L225-L244]
    basis = chebyshev_basis(z, K)                    # (H, K)

    # Scalar shifts for A, dt and D: learned coefficients times the basis
    # [alg. 2] [code: src/mark.py#L274-L281]
    A_shift = sum(A_proj(h)[k] * basis[k] for k in range(K))
    dt_shift = sum(dt_proj(h)[k] * basis[k] for k in range(K))
    D_shift = sum(D_proj(h)[k] * basis[k] for k in range(K))

    # Bounded shifts: A's bound is 0.5 * sigmoid(learned); dt and D use
    # learned magnitudes  [code: src/mark.py#L284-L285]
    # [code: src/mark.py#L333-L335]
    A_log = A_log + beta_A * tanh(A_shift)
    dt = dt + beta_dt * tanh(dt_shift)        # softplus(dt + dt_bias) follows
    D = D + beta_D * tanh(D_shift)

    # Low-rank factors for B and C: base term plus basis-weighted term
    # [alg. 2] [code: src/mark.py#L246-L255]
    basis_mean = mean(basis, over heads)             # (K,)
    B_scale_uv = base_B_scale(h) + sum_k(basis_mean[k] * basis_B_scale[k])
    B_shift_uv = base_B_shift(h) + sum_k(basis_mean[k] * basis_B_shift[k])
    C_scale_uv = base_C_scale(h) + sum_k(basis_mean[k] * basis_C_scale[k])
    C_shift_uv = base_C_shift(h) + sum_k(basis_mean[k] * basis_C_shift[k])

    # Each (U, V) pair becomes a flattened matrix product U @ V
    # [code: src/mark.py#L252-L255]
    B_scale = uv_product(B_scale_uv)
    B_shift = uv_product(B_shift_uv)
    C_scale = uv_product(C_scale_uv)
    C_shift = uv_product(C_shift_uv)

    # Bounded scale and shift applied to B and C
    # [alg. 2] [code: src/mark.py#L337-L339]
    B = B * (1 + alpha_B * tanh(B_scale)) + beta_B * tanh(B_shift)
    C = C * (1 + alpha_C * tanh(C_scale)) + beta_C * tanh(C_shift)
    return A_log, B, C, dt, D
Pseudocode
# CART token weights for one masked batch  [§3.3] [app. A.3]
# Input: masked (B, L) bool; p = 0.45
# Output: weights (B, L), zero at visible positions

def cart_weights(masked, p):
    # Geometric kernel over distance d = |i - j|; zero at d = 0.
    # Value 0.5 * p * (1 - p)^(d - 1)  [app. A.3] [code: src/train.py#L153-L159]
    kernel = geometric_kernel(p)                     # (L, L)

    # Sum the kernel over visible positions j for each position i
    # [app. A.3] [code: src/megatron.py#L101-L108]
    visible = (~masked).float()                      # (B, L)
    weights = visible @ kernel                       # (B, L)

    # Keep masked positions; the code adds 1e-8 to avoid zero weights
    # [code: src/megatron.py#L108]
    return weights * masked

Hyperparameters

NameValueWhereNotes
BackboneHydra-BERT, 23 layers, 111M params, hidden 768, vocab 30,522app. C.1 training_config_chebyshev_stage1.yamlfrozen; state size 64, convolution width 7, head size 64, expansion 2, chunk length 256 app. C.1
Sequence length4096 tokensapp. C.1
MaRK learning rate6.0e-4app. C.1 training_config_chebyshev_stage1.yaml
Optimizerfused AdamW, β = (0.9, 0.999), weight decay 0.01app. C.1 nemo.py
LR schedulelinear warmup 720 steps, then cosine decay over 14,400 steps to 0.1×app. C.1 training_config_chebyshev_stage1.yamltotal_steps is set by hand, "approximate" in the config
Epochs3app. C.1 training_config_chebyshev_stage1.yaml
Batch size112 (all kernels)app. C.1 training_config_chebyshev_stage1.yamlgradient accumulation of 2 is also set in the config training_config_chebyshev_stage1.yaml; not stated in the paper
Precision and hardwarebf16, DDP on 2 NVIDIA GH200 GPUs, gradient checkpointing, gradient clipping at norm 1.0app. C.1
Adapter rank r2app. C.1 training_config_chebyshev_stage1.yamllow-rank factors for B and C
Adapter MLP width256app. C.1 training_config_chebyshev_stage1.yaml
Timestep embeddingsinusoidal, dimension 128, then two linear layers of width 128§2 hydra_modules.pythe code multiplies t by 1000 before the sinusoid (see Paper vs code)
Chebyshev degree K5app. C.1 training_config_chebyshev_stage1.yamlfive basis functions T_0 to T_4 in the code
DCT basis256 timepoints, 8 frequenciesapp. C.1 training_config_dct_stage1.yaml
Decay exponent α (DCT)learned; ε = 1e-4eq. 18 mark.pyinitial value not stated in the paper; the code uses 1.0
A-shift bound β_A≤ 0.5app. C.2 mark.py
Scale and shift magnitudes α, βinitialised at 0.2, learnedapp. C.2 mark.pysee Paper vs code for the effective β_A at start
CART p0.45§3.3 app. C.2 training_config_chebyshev_stage1.yaml
CART scale1.0app. C.2
Seeds42 + 100iapp. C.310 seeds for Tables 1 and 2; 15 seeds for Table 3
Validation cap500 batches for arXiv onlyapp. C.3
Synthetic LPVstate 4, rank 2, L = 20, N in {50 ... 1600}, σ = 0.1, 15 seeds, Δt = 0.1app. C.5
AQS checkP = I, β_A = 0.5, Δ_min = 0.001app. C.4 alg. 4

Results

Author-reported. Setup: internal Hydra corpus, about 3.54B training and 0.76B validation tokens app. C.1; 111M frozen Hydra-BERT; metric is the CART-weighted validation loss (weighted_nll), not perplexity app. C.3. Lower is better.

CART loss by adapter, mean ± 95% CI over 10 seeds tab. 1. The paper's bold and underline marks are dropped here:

KernelTrainable paramsPTBWikiTextLAMBADAAG NewsarXivAvg.
Hypernet6.3M3.39 ± 0.313.73 ± 0.233.70 ± 0.173.91 ± 0.204.12 ± 0.103.77
Chebyshev11.0M2.42 ± 0.262.55 ± 0.162.40 ± 0.102.85 ± 0.142.55 ± 0.052.55
DCT7.1M2.48 ± 0.272.55 ± 0.142.47 ± 0.112.86 ± 0.142.61 ± 0.072.59

Leave-one-out on WikiText, which modulates each SSM parameter group separately tab. 2; "Full" is all five, "None" is the frozen backbone. Mean ± 95% CI over 10 seeds:

KernelFullWithout AA alonedt aloneB and C aloneD aloneWithout dtNone
Hypernet3.73 ± 0.263.91 ± 0.274.07 ± 0.294.04 ± 0.294.08 ± 0.293.78 ± 0.263.74 ± 0.263.99 ± 0.29
Chebyshev2.55 ± 0.182.78 ± 0.193.44 ± 0.223.92 ± 0.263.06 ± 0.213.90 ± 0.262.56 ± 0.184.00 ± 0.27
DCT2.55 ± 0.162.80 ± 0.163.24 ± 0.173.99 ± 0.243.05 ± 0.173.90 ± 0.242.59 ± 0.164.00 ± 0.24

For Hypernet, the A-alone and B-and-C-alone variants score worse than the frozen backbone (4.07 and 4.08 against 3.99) tab. 2. The paper reads the rows as showing that modulating A is necessary and that modulating Δ alone gives only marginal gains §4.4.

Inference cost at sequence length 4096, batch 8, one NVIDIA A100-SXM4-80GB; 15 seeds, 100 forward passes each; mean ± 95% t-interval tab. 3:

VariantLatency (ms)Memory (MiB)Total params (M)Added params (M)Overhead seq=512Overhead seq=4096
Frozen Hydra267.43 ± 0.064446.58 ± 0.06111.680.00--
Input injection270.86 ± 0.034455.25 ± 0.06113.962.28+3.6%+1.3%
AdaLN Zero276.84 ± 0.024474.52 ± 0.60118.516.84+7.6%+3.5%
Hypernet (MaRK)304.75 ± 0.034470.62 ± 0.06117.956.27+65.6%+14.0%
Chebyshev (MaRK)309.74 ± 0.104488.61 ± 0.06122.6510.97+104.0%+15.8%
DCT (MaRK)307.28 ± 0.054473.93 ± 0.06118.757.07+88.7%+14.9%

Other reported results:

  • The AQS margins are positive for all 23 layers under the stated assumptions §4.1. The per-layer values are only in the figure fig. 1.
  • On synthetic LPV data, the Markov operator error falls with dataset size, at fitted rates of about N^-0.54 (Hypernet), N^-0.66 (Chebyshev) and N^-0.64 (DCT) §4.2 fig. 2.
  • Markov-norm curves for the three adapters at several diffusion timesteps separate from the frozen backbone §4.3 fig. 3. This is qualitative.

Not evaluated, according to the paper: generation quality, downstream accuracy, long-context benchmarks, and matched-budget comparisons against AdaLN or input injection on quality app. A.5.

Paper vs code

Checked against the official repository at code_commit (721c935d02facfb96bbb682fec9ad5ae2a8d3639). Repository source is Apache-2.0. The paper links the repository only from a commented-out checklist in its LaTeX source. The GitHub owner's profile shows the first author's name, and the project website is under the same owner. Adapter code is in src/mark.py; training freezing and loss are in src/nemo.py, src/megatron.py, src/train.py and src/hydra_model.py.

  • Chebyshev latent coordinate. The paper says z is a latent coordinate in [-1, 1] app. A.2, and its synthetic version uses z = 2τ − 1 app. C.5. The language-model adapter computes z = z_proj(h) with no bound [code: src/mark.py#L192] mark.py. Outside [-1, 1] the polynomial values grow. The tanh that follows bounds the shifts, not the basis.
  • Chebyshev degree. The main text sums T_k for k = 1 to K eq. 16, while Algorithm 2 sums k = 0 to K − 1 alg. 2. The code uses K functions, T_0 to T_{K−1}, with K = 5 mark.py [code: configs/training_config_chebyshev_stage1.yaml#L105]. Five functions means polynomial degree four.
  • Hypernet shifts. The main text says the Hypernet shifts are "optionally left unconstrained" §3.4. The code bounds every Hypernet shift with β tanh mark.py mark.py. Algorithm 1 also uses tanh alg. 1, so the main text is the outlier.
  • Initial A bound. The paper initialises every magnitude to 0.2 app. C.2. The code sets β_A = 0.5 · sigmoid(param) with param initialised to 0.2 mark.py mark.py. At the start, β_A is therefore about 0.275, not 0.2.
  • Timestep embedding scale. The paper describes a standard sinusoidal encoding of the diffusion timestep §2. The code uses the ratio s/T multiplied by 1000 before the sinusoid hydra_modules.py.
  • Timestep and masking ratio. The paper says masking uses probability t, with t drawn from U(0, 1) and the same t drives the conditioning §3.2 §2. The Megatron training path samples T in [10, 1000) and s in [0, T), sets the masking ratio to 1 − s/T and conditions on s/T megatron.py. The NeMo path samples the same way nemo.py. The masking routine then adds N(0, 0.05²) noise to the ratio and clamps it train.py. The discrete grid and the noise are not in the paper.
  • Backbone freezing and training stages. The paper says the backbone and classification head stay fixed for every reported experiment and training runs three epochs app. C.1. The repository has three stages. Stage 1 trains only the adapters nemo.py. Stage 2 unfreezes the last 30% of layers (layers 16 to 22 of 23) at learning rate 3e-5 for five epochs nemo.py nemo.py [code: configs/training_config_chebyshev_stage2.yaml#L3]. Stage 3 trains only the classifier nemo.py. The paper does not say which stage produced the reported checkpoints.
  • Gradient accumulation. The configs set accumulate_grad_batches to 2 training_config_chebyshev_stage1.yaml. The paper does not mention accumulation, so the effective batch size is not stated.
  • CART weight formula. The paper's formula app. A.3 matches the code's kernel, including the factor 0.5 and the sum over visible positions train.py megatron.py. No discrepancy found.

Pitfalls

  • Stability margin needs a lower bound on Δ. The paper says the margin ε = 1 − max ² is positive "for all admissible conditioning contexts" §2.1. But Ā = exp(A·Δ) approaches 1 as Δ goes to 0, and dt is softplus(dt_input + dt_bias'), so it is positive but has no positive lower bound. The bound on A does not bound Δ. The certificate uses Δ_min = 0.001 as an evaluation point app. C.4, which the model does not enforce. Treat the uniform margin as conditional on a Δ floor that the paper does not implement.
  • Modulation bounds are not uniform across parameters. Only A is capped (β_A ≤ 0.5, via a sigmoid in the code). The other magnitudes are learned without a cap app. C.2. The certificate depends only on A, because only A is raised to powers app. C.2. The other parameters sit outside it, so the certificate says nothing about them.
  • Unstated initial value of the DCT decay exponent. The exponent α is learned, and the paper does not give its initial value. The code starts it at 1.0 mark.py, which makes the decay 1/(f+1)^(softplus(1) + ε) at start.
  • Masking details the paper does not state. The code masks only positions that are not padding and not the special tokens 101 and 102; the mask token is 103 train.py train.py. The floor on the masking probability is 1e-3 train.py.
  • CART loss is not a perplexity and not normalised by weight. The reported metric divides the weighted sum by the number of masked tokens, not by the sum of weights hydra_model.py. Each weight is at most 1: on one side of a masked token, the kernel summed over all distances gives 0.5, so the two sides together give at most 1 (from the formula in app. A.3). So the reported loss is smaller than or equal to the mean masked NLL on the same batch, and values from other papers are not directly comparable.
  • Heavily masked sequences get little weight. A masked position with no visible neighbours has weight near zero (the code adds 1e-8 [code: src/megatron.py#L108]). At high mask ratios, many masked tokens therefore contribute almost nothing to the loss. This is inferred from the formula and code, not measured in the paper.
  • Chebyshev basis outside [-1, 1]. The paper's smoothness argument for the Chebyshev kernel assumes z in [-1, 1] app. A.2. In the code z is unbounded (see Paper vs code). Results at timesteps far from the training distribution could behave differently from the paper's analysis.
  • Discrete timestep grid. The conditioning uses s/T with T random in [10, 1000). A continuous timestep t in [0, 1] is an idealisation of this discrete grid §2.
  • Which checkpoint produced the numbers. Tables 1 and 2 come from released per-kernel runs initialised from kernel-specific checkpoints app. C.1. The paper does not say which of the three training stages produced them, so a reproduction should match the stage before comparing numbers.
  • Costs the abstract hides. Overhead is about 37 to 42 ms per forward pass, roughly constant in sequence length §4.4. It is 65% to 104% at length 512 and 14% to 16% at length 4096 tab. 3. The authors also note the cost scales with the number of diffusion sampling steps §4.4.
  • Trainable parameter range. The abstract's "6.3-11M" is the adapter size for the three geometries tab. 1. The backbone itself is 111M and frozen tab. 3.

Lineage

  • Builds on: Hydra (Hwang et al., 2024), the bidirectional quasi-separable mixer whose SSM parameters MaRK modulates §3.1. Not linked: no arXiv id in the bibliography.
  • Builds on: Mamba, whose selective state-space recurrence defines the A, B, C and dt parameters §2 §3.1.
  • Builds on: Scalable Diffusion Models with Transformers, whose sinusoidal timestep encoding with a two-layer MLP the conditioning vector follows §2. The AdaLN baseline is cited to the same work §1.
  • Builds on: CART is adapted from Dream 7B (Ye et al., 2025), which introduced the token-level weighting §3.3 app. A.3. Not linked: no arXiv id in the bibliography.
  • Compared against: AdaLN Zero and input injection, on inference latency, memory and parameter count only tab. 3 app. A.5. No quality comparison at matched budget.
  • Compared against, between adapters only: Hypernet, Chebyshev and DCT, all inside this paper tab. 1 tab. 2.
  • Superseded by: not known.

Source and license

Adapted from “MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models” by Syed Ibrahim Omer, Ginny Y. Wong, Xiangyu Zhao (arXiv:2610.09092), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/ibitec7/mark at commit 721c935d02fa (Apache-2.0). Any quoted code is shown under that license.