On Predictability of Reinforcement Learning Dynamics for Large Language Models

In plain words

Reinforcement learning is what teaches many language models to reason better. This paper looks at how the weights change during that training. Taking the full change between the starting model and the trained one, and keeping only its single strongest direction, already brings back most of the reasoning gain. The paper calls this the rank-1 part of the update.

That strongest direction also changes in a steady, almost straight-line way from one checkpoint to the next. So an early snapshot is enough to guess where it will end up. The method, AlphaRL, does exactly that: it looks at a few early checkpoints, draws the line, and jumps ahead to a predicted late-stage update, which can shorten the training run.

TL;DR

• Rank-1 Dominance: the top singular component of the RL weight update carries nearly all of the reasoning gain, per the authors §2 tab. 8.
• Rank-1 Linear Dynamics: that component evolves close to linearly over training, so later states can be predicted from early ones §3.
• AlphaRL extrapolates the late update from an early window, with no extra modules and no tuned hyperparameters; the authors report up to 2.5x fewer steps while keeping over 96% of performance §4.2 tab. 1.

Problem

RL fine-tuning of LLMs gives large reasoning gains, but the parameter changes behind them are poorly understood §1. Full-parameter RL is also expensive, since every checkpoint must be trained through the whole schedule §4. The paper asks whether the RL update has a simple low-rank structure that is stable over training and therefore predictable from early checkpoints §1.

Assumptions: the base and RL-trained models share an architecture; the update is taken as the weight difference per matrix; reasoning is measured with accuracy on math benchmarks (author setup in §2.2 and app. B).

Method

Rank-1 analysis (Property 1). For each weight matrix, take the update \Delta W = W_{\text{RL}} - W_{\text{base}} and its singular value decomposition §2.1 eq. 1. Keep only the top singular triple eq. 2. Rescale this rank-1 part to the norm of the full update eq. 3. Add it to the base model and compare accuracy with the full model §2.1 §2.2.

\Delta W = \sum_{i=1}^{r} \sigma_i \, u_i v_i^\top, \quad r = \operatorname{rank}(\Delta W) \quad \text{[eq. 1]}

\Delta W^{(1)} = \sigma_1 \, u_1 v_1^\top \quad \text{[eq. 2]}

\Delta \hat W^{(1)} = \alpha \, \Delta W^{(1)}, \quad \alpha = \frac{\lVert \Delta W \rVert_2}{\lVert \Delta W^{(1)} \rVert_2} \quad \text{[eq. 3]}

The paper also evaluates Rank-k% subspaces, keeping the top k percent of singular triples, to study the collective effect §2.1 tab. 8.

Linear dynamics (Property 2). For each module, collect the top left singular vector u_1^{(t)} at checkpoints t = 1 \dots T app. E. Fit a one-component PLS regression from these vectors to the checkpoint accuracy, and measure the fit with R^2 §3.1 app. E eq. 5 eq. 6 eq. 7 eq. 8 eq. 9 eq. 10. A module whose trajectory is smooth and monotone gets an R^2 near 1 §3.1. The authors also tie R^2 to module importance with a sliding-window injection test §3.2 fig. 6.

AlphaRL (acceleration). For a checkpoint at a chosen early step, build a scaled rank-1 vector per module, \alpha^{(t)} \sigma_1^{(t)} u_1^{(t)} §4.1. Fit a single-component PLS model from those vectors to accuracy. Invert the fitted line at a target accuracy y^* and map the score back to a vector. The predicted update is that vector combined with v_1 §4.1. The paper sets y^* = 1 on MATH-500, in relative accuracy, and applies the same prediction to all benchmarks §4.1.

Equations used in the appendix (regression on the trajectory):

\Delta y^{(1)} = \Delta \hat W^{(1)} h = \alpha \, \sigma_1 \, u_1 \langle v_1, h \rangle \quad \text{[eq. 4]}

\mathcal{U}_1 = \{ u_1^{(t)} \}_{t=1}^{T} \quad \text{[eq. 5]}

\mathcal{D} = \{ (u^{(t)}, y^{(t)}) \}_{t=1}^{T} \quad \text{[eq. 6]}

z_1 = \tilde{\mathcal U}_1 \, w_1 \quad \text{[eq. 7]}

y^{(t)} = \alpha z_1^{(t)} + \beta + \varepsilon^{(t)} \quad \text{[eq. 8]}

(\hat\alpha, \hat\beta) = \arg\min_{\alpha,\beta} \sum_{t=1}^{T} \bigl(y^{(t)} - (\alpha z_1^{(t)} + \beta)\bigr)^2 \quad \text{[eq. 9]}

R^2 = 1 - \frac{\sum_t (y^{(t)} - \hat y^{(t)})^2}{\sum_t (y^{(t)} - \bar y)^2} \quad \text{[eq. 10]}

The rank-1 computation per module, as used by AlphaRL:

# Rank-1 part of one module's RL update  [§2.1] [eq. 1] [eq. 2]
# Input:  base weight (d_out, d_in); RL weight of the same shape
# Output: sigma1, u1 (d_out,), v1 (d_in,)

def rank1_of_update(base_weight, rl_weight):
    # 1. Form the weight difference  [§2.1]
    delta = rl_weight - base_weight
    # 2. Full SVD; keep the top singular triple  [eq. 1] [eq. 2]
    u, s, vt = svd(delta)
    return s[0], u[:, 0], vt[0, :]

The AlphaRL predictor, one module at a time:

# AlphaRL: predict a late-stage rank-1 update  [§4.1] [app. E] [eq. 8]
# Input:  scaled vectors xs (T, d_out) from early checkpoints
#         accuracies ys (T,); target accuracy y_star
# Output: predicted vector x_star (d_out,)

def predict_vector(xs, ys, y_star):
    # 1. Fit one PLS component from vectors to accuracy  [app. E]
    pls = PLSRegression(n_components=1)
    scores = pls.fit_transform(xs, ys)[0]    # (T, 1)
    # 2. Fit accuracy as a line in the score  [eq. 8] [eq. 9]
    slope, intercept = ols(scores, ys)
    # 3. Invert the line at the target accuracy  [§4.1]
    score_star = (y_star - intercept) / slope
    # 4. Map the score back to vector space  [§4.1]
    return pls.inverse_transform(score_star)

Applying the prediction to form the model:

# Build the AlphaRL model  [§4.1]
# Code: eval/analysis/AlphaRLBuildPerdictModel.py#L49-L98
# Input:  model weights per module; predicted x_star[m]; v1[m]; flip[m]

def build_model(model, x_star, v1, flip):
    for module in modules:
        # 1. Combine the predicted vector with v1  [§4.1]
        update = outer(x_star[module], v1[module])
        # 2. Undo the module's sign alignment  [code: ...#L65-L66]
        if flip[module]:
            update = -update
        # 3. Add the update to the weights  [§4.1] (see Paper vs code)
        model[module] += update
    return model

The sign alignment that the code applies before the regression (not described in the paper):

# Sign alignment across checkpoints  [code: extract_rank1_au.py#L36-L62]
# Input:  per-checkpoint vectors for one module, ordered by step

def align_signs(vectors):
    # 1. Walk from the last checkpoint back to the first  [code: ...#L89-L100]
    for step in reversed(range(1, len(vectors))):
        # 2. Flip this vector if it points against the next-later step
        if dot(vectors[step - 1], vectors[step]) < 0:
            vectors[step - 1] = -vectors[step - 1]
    return vectors

Hyperparameters

Training values are from the paper's appendix. The official repository contains evaluation and analysis code only, so none of the training values below can be checked against code repository tree.

Results

Author-reported. Setup: six math and science benchmarks (AIME24, AIME25, MATH-500, Minerva, GPQA, GSM8K), 32 samples per question at temperature 0.6, reported as average accuracy. Models: Qwen3-8B-Base trained with GRPO, RLOO and DAPO §4.2 app. B. Baseline: the fully trained model at each algorithm.

Source: tab. 1. Relative accuracy (computed here from the averages in tab. 1, not stated in the paper): at 40% + AlphaRL, DAPO 99.9%, GRPO 95.9%, RLOO 95.5% of the fully trained average.

Other reported results:

• Rank-1 alone: an average of 99.17% of reasoning capability across the RL runs §1; the Rank-1 numbers are shown in a figure, not in a table fig. 3.
• Other models at AlphaRL 40%: Qwen3-14B-Base 59.18 vs 60.87 fully trained; GLM-4-9B-0414 58.89 vs 59.98 tab. 3. Relative accuracy computed here: 97.2% and 98.2%.
• DeepSeek-R1-Distill-Qwen-7B (after distillation) and Llama3-8B-Instruct (after SFT) tab. 4; the table lists both the 50% AlphaRL row and the fully trained row.
• Adversarial self-play on Kuhn Poker (Qwen3-4B-Base, Spiral setup) tab. 5, and MATH-500 after that training tab. 6.
• Update size and rank-1 fraction per method tab. 7; Rank-k% accuracy on MATH-500 per method tab. 8.

Paper vs code

Repository: https://github.com/caiyuchen-ustc/Alpha-RL at commit 2d8861e40632429d5247be1c0f2714ce452e4e17. Repository owner matches the first author. No LICENSE file; GitHub reports no license. The repository contains evaluation and analysis code only, with no RL training code.

• AlphaRL scale. The paper scales each vector by \alpha^{(t)}\sigma_1^{(t)} §4.1. The code uses only \sigma_1 u_1 (U[:, 0] * S[0]), with no \alpha extract_rank1_au.py. The predictor then reads that vector directly AlphaPredVector.py.
• Where the update is added. The paper's evaluated model adds the rescaled rank-1 part to the base model §2.1. For AlphaRL the code loads <RL>-step-N as the model to edit and adds the predicted update to it AlphaRLBuildPerdictModel.py AlphaRLBuildPerdictModel.py. Its docstring is silent on the base. I did not run the code, so the effect of this difference is not verified.
• Sign alignment. The code flips each module's u_1 to agree with the next-later checkpoint extract_rank1_au.py extract_rank1_au.py. The paper does not describe this step.
• Norm used for \alpha. The paper writes \lVert\cdot\rVert_2 in eq. 3 §2.1. The rank-k code uses torch.norm on the full matrix, which is the Frobenius norm for 2-D tensors upd_rank.py. The ratio is computed per module.
• Samples per question. The paper reports 32 responses per question §4.2. The evaluation script runs with n_sampling 4 and k 1 reasoning_eval.sh. The predictor reads the k4 files, and the PLS analysis reads the k1 files AlphaPredVector.py AlphaPLS.py.
• Accuracy scale and target. The paper's target is y^* = 1 in relative accuracy §4.1. The code's acc is a raw fraction, correct over total examples reasoning_eval.py, not relative to the full model. The shipped script passes --y_predict 1.7 AlphaPredVector.sh, which lies outside the raw 0 to 1 range.
• Checkpoint used for prediction. The shipped script predicts from step 9 of 27, described as about 30% of training AlphaPredVector.sh. The paper's table rows are 10% and 40% tab. 1.
• Feature scaling. The appendix says features are standardised app. E. The code does not set scale explicitly and relies on the scikit-learn default AlphaPredVector.py.
• R^2. The code computes R^2 in-sample on the same checkpoints used for the fit AlphaPredVector.py. I found no held-out error computation for the "under 5%" claim in the repository §1.
• Single-component option. Only_Rank_1_Space is passed as a string from the command line, and the branch that uses it refers to undefined U and S AlphaRLBuildPerdictModel.py AlphaRLBuildPerdictModel.py. The CLI always passes a string, so the branch never runs. The docstring claims it improves performance; that is not verified.
• Device handling. svd.py hard-codes .to('cuda') for the difference matrix, so --device cpu fails without CUDA svd.py svd.py. The commented commands in svd.sh use --device cpu.
• Shell script. AlphaPredVector.sh has a space after the backslash on the --end 9 line, which breaks line continuation AlphaPredVector.sh.
• Training code. Absent. The RL training values in app. B cannot be verified against code.

Pitfalls

• Notation clash. \alpha is the norm rescale in eq. 3 §2.1 and the regression slope in eq. 8 app. E. Keep them separate when implementing.
• Speedup arithmetic. "2.5x" equals 1/0.4, the ratio of the full run to the 40% checkpoint §4.2 tab. 1. The count covers RL steps only. It leaves out the cost of a full SVD of every attention and MLP weight matrix at each checkpoint svd.py, the PLS fits and the rebuild.
• Where the claim is stronger than the table. The text says RLOO and GRPO surpass the full model on MATH-500 at 40% §4.2. The table shows GRPO at 78.45 against 78.25, but RLOO at 76.00 against 78.25 tab. 1. The text also says 10% + AlphaRL is comparable to 40% without AlphaRL §4.2. That holds for DAPO only (46.47 vs 46.30); GRPO is 43.43 vs 45.30 and RLOO 43.74 vs 46.44.
• Headline figures. The abstract says "over 99% of performance gains" [abstract]. The introduction says "99.17% of the reasoning capability" §1. The 99.17% average has no table in the text.
• Inconsistent counts. The abstract says 8 LLMs and 7 algorithms. The introduction says 13 LLMs and 10 algorithms §1. §2.2 says eight models, five RL algorithms, plus DIST and SFT §2.2.
• Update-size claim. The text says DIST and SFT updates are "one to two orders of magnitude" larger than RL §2.3. The table gives 21.24 (offline DIST) and 10.75 (offline SFT) against 0.01 (online RL), which is about three orders tab. 7. The units are not given.
• Table 8. The RL column reads 100% at every rank, so it shows no rank effect. Use fig. 3 for the RL curve.
• Table averages. Two averages do not match their rows (spot-checked): tab. 3 14B at 40% + AlphaRL lists 59.18, its row mean is 59.84; tab. 4 DeepSeek-R1-Distill-Qwen-7B at 20% + AlphaRL lists 48.23, its row mean is 68.23. Recompute from the rows before use.
• SFT batch arithmetic. Batch 1,024 with mini-batch 4 gives 256 accumulation steps, not the 512 stated app. B.
• Dataset name. The SFT dataset is called DeepMath-103K in one place and DeepMath-107K in another app. B.
• Noise. No error bars or seeds are reported for the main table. Differences of one or two points on the smaller benchmarks may be within sampling noise; treat them cautiously.
• Sign ambiguity. SVD signs are arbitrary. Without aligning signs across checkpoints, the fitted trajectory may flip; the code aligns them as described above.
• Scope of the claim. The limitations section says the method is constrained by the design and stability of the RL algorithm §6.
• Scope. Attention and MLP weight matrices are analysed svd.py. Embeddings are reported as nearly unchanged under RL §2.3 fig. 5.

Lineage

• Builds on: DAPO, GRPO, RLOO and Dr.GRPO, which are the RL algorithms studied §2.2. Not linked: no cards for them exist yet.
• Related analysis: the paper also studies distillation and SFT as control paradigms §2.2 §2.3; on-policy distillation and DPO show similar low-rank structure app. C tab. 7 tab. 8.
• Compared against: full-parameter training at each checkpoint (baseline) tab. 1.
• Superseded by: none known.