DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

In plain words

Reasoning models are often trained by letting them try a math problem several times, checking each attempt against the known answer, and nudging the model toward the attempts that were right. A popular recipe for this, GRPO, works by comparing each attempt with the average of its own group. The authors show that this comparison quietly gives too little training signal to questions the model almost always gets right or almost always gets wrong.

DisCO takes a different starting point. It treats the job as the one a classifier faces: push the score of each correct attempt above the score of each wrong one, for every question. Two more changes keep training stable. The scores are not clipped, which the authors link to the model's answers collapsing into a narrow set of outputs. And the amount the model may move away from its previous version is limited by a constraint rather than by a hand-tuned penalty.

The point of the method is steadier training and better math accuracy than the baselines compared in the paper, under the same 8k response limit.

TL;DR

• DisCO is a reinforcement learning objective for fine-tuning reasoning models on binary-reward tasks such as math. It replaces GRPO's group-relative advantage with a discriminative loss over positive and negative rollouts §4 §5.1.
• It uses non-clipped scores (likelihood ratio or log-likelihood) and a squared-hinge penalty that holds the KL divergence to the previous policy below a threshold §5.1.
• A second variant (DRO-style, §5.2) handles the many-negatives imbalance.
• Author-reported: on DeepSeek-R1-Distill-Qwen-1.5B with 8k training and testing, the log-likelihood variant averages 0.533 over six benchmarks, against 0.457 for GRPO and 0.473 for DAPO tab. 2.
• The official code is a fork of the verl framework. Several details differ from the paper, so read ## Paper vs code before reimplementing.

Problem

Under a binary reward, GRPO's per-question weight is the square root of p(1-p), where p is the fraction of correct rollouts for that question §4 app. B.1. That weight is near zero for questions that are nearly always right or nearly always wrong, so those questions contribute little to learning. The authors call this difficulty bias. Dr. GRPO removes part of the normalisation but keeps a p(1-p) weight, so the bias remains §4 app. B.3.

Clipped surrogates can cause entropy collapse, and a KL penalty with a fixed coefficient is hard to tune across problems and over training §5.1. Negative rollouts also outnumber positive ones on hard questions, a data imbalance that a uniform average over negatives handles poorly §5.2.

The analysis assumes 0 < p(q) < 1 for each question §4.

Method

Setting. A rollout o is sampled from the old policy pi_old for question q. The reward r(o|q) is 1 or 0. Any expectation over o splits into a positive part and a negative part weighted by p(q) and 1-p(q) eq. 1 §3.

GRPO as a discriminative objective. For GRPO's advantage and clipped surrogate, the objective can be rewritten as a question-weighted difference between the mean score of positive rollouts and the mean score of negative rollouts. The question weight is sqrt(p(q)(1-p(q))) eq. 4 §4 app. B.1. The GRPO-specific scoring functions are the clipped ratios f+ and f- eq. 5.

Removing the weight. Dropping the question weight from GRPO gives a variant the authors call GRPO_RW eq. 6. In their experiment on a 1.5B model, this variant reaches a higher share of fully correct questions and a lower share of fully wrong questions faster fig. 1 §4.

Basic discriminative objective (DisCO-b). The authors maximise the expected surrogate of score differences between positive and negative rollouts for each question, using one score function s for both sides eq. 7 §5.1. The surrogate is the identity in the main runs. Two scores are studied:

• L-ratio: the mean over response tokens of pi_theta / pi_old §5.1.
• log-L: the mean over response tokens of log pi_theta §5.1.

Trust region as a constraint. The objective is maximised subject to KL(pi_old || pi_theta) <= delta eq. 8. The authors solve it by the squared-hinge penalty with coefficient beta eq. 9. The penalty's gradient is zero while the constraint holds, so the KL term acts only after the threshold is crossed. A plain KL regulariser, by contrast, always pulls §5.1.

Imbalanced rollouts (DisCO). The per-question AUC-style objective treats all negatives equally. The authors replace it with a DRO-based objective that emphasises the hardest negatives, derived as a lower bound §5.2:

# Per-positive DRO term; J_2 is its expectation over questions  [§5.2] [eq. 10]
def dro_term(pos_score, neg_scores, tau):
    # Log of the mean exponential over negatives, sharpened by tau
    gap = (neg_scores - pos_score) / tau
    return tau * log(mean(exp(gap)))

The authors show J_2 is at most J_1 by Jensen's inequality, so maximising J_2 also increases J_1 §5.2. DisCO maximises J_2 under the same KL constraint eq. 11; DisCO-b maximises J_1 under the constraint eq. 8 §5.2.

Algorithm. The full loop is Algorithm 1 in Appendix B alg. 1 app. B. Per step it samples 128 questions, freezes pi_old, draws n responses per question, and splits them into positives and negatives by reward. It then runs one AdamW update per mini-batch on the combined gradient G = G1 + G2, where G1 is the DRO-term gradient and G2 = 2 beta [KL - delta]_+ grad KL alg. 1.

# One mini-batch loss for DisCO (L-ratio score)  [alg. 1] [§5.2]
# Inputs: per question, rollouts with reward, new and old token log-probs
def disco_loss(groups, delta, beta, tau):
    # Token-averaged KL(old || new) estimate  [alg. 1]
    kl = mean_over_tokens(old_logp - new_logp)
    total = 0.0
    for group in groups:
        # L-ratio score per rollout: mean over tokens of pi / pi_old  [§5.1]
        scores = mean_over_tokens(exp(new_logp - old_logp))
        pos = scores[group.reward == 1]
        neg = scores[group.reward == 0]
        # Questions with no positive or no negative rollout add zero;
        # the paper does not say this  [code: verl/verl/trainer/ppo/core_algos.py#L473-L476]
        if len(pos) == 0 or len(neg) == 0:
            continue
        # Mean over positives of s - tau * log sum over negatives of exp(s/tau)  [alg. 1]
        total += mean(pos) - tau * logsumexp(neg / tau)
    # Average over the questions in the mini-batch  [alg. 1]
    objective = total / len(groups)
    # Squared-hinge penalty on the KL constraint  [§5.1] [eq. 9]
    penalty = beta * max(kl - delta, 0.0) ** 2
    return -objective + penalty

For the log-L variant, replace the scores with the mean token log-probability and use tau = 10 §6 run_disco_logL_1.5b_8k.sh.

Key equations.

Basic objective eq. 7:

\mathcal J_1(\theta)=\mathbb E_q\,\mathbb E_{o\sim\pi^+_{old},\,o'\sim\pi^-_{old}}\,\ell\big(s_\theta(o,q)-s_\theta(o',q)\big)

Constrained form eq. 8:

\max_\theta \mathcal J_1(\theta)\quad\text{s.t.}\quad D_{KL}(\pi_{old}\,\|\,\pi_\theta)\le\delta

Squared-hinge penalty eq. 9:

\max_\theta\ \mathbb E\big[\ell(\cdot)\big]-\beta\,[D_{KL}(\pi_{old}\,\|\,\pi_\theta)-\delta]_+^2

DRO objective J_2 eq. 10:

\mathcal J_2(\theta)=-\mathbb E_q\mathbb E_{o\sim\pi^+_{old}}\,\tau\log\Big(\mathbb E_{o'\sim\pi^-_{old}}\exp\Big(\tfrac{s_\theta(o',q)-s_\theta(o,q)}{\tau}\Big)\Big)

Final DisCO problem eq. 11: maximise J_2 subject to the same constraint as eq. 8.

Method comparison. tab. 1 summarises how DisCO differs from GRPO, Dr. GRPO, DAPO, GPG and TRPA on difficulty bias, clipping, KL handling, score function, and handling of imbalanced rollouts tab. 1 §5.2.

Connections. The L-ratio score is the TRPO-style surrogate with the question weight removed; the log-L score is the vanilla policy-gradient surrogate with the weight removed app. B.2. Appendix B.3 derives the question-level weights of Dr. GRPO, DAPO and GPG, and tab. 6 lists the weights and score functions of the variants app. B.3 tab. 6.

Hyperparameters

Values are as stated in the paper unless marked "code". Repository paths are relative to the code root at code_commit.

Results

Author-reported. Setup §6: training on the DeepScaleR-Preview dataset, about 40.3k unique problem-answer pairs; six benchmarks (AIME 2024, AIME 2025, MATH 500, AMC 2023, Minerva, OlympiadBench); pass@1 averaged over 16 samples; 8k maximum response length for training and testing unless stated. Base models are DeepSeek-R1-Distill-Qwen-1.5B, -Qwen-7B and -Llama-8B §6.

Rows marked † are shaded in the paper, which the caption says are models trained by other works or numbers taken from their original reports. Other rows are either evaluated on existing models or trained by the authors §6.

1.5B model tab. 2

The paper's text describes the log-L result as about 7 points above GRPO and above DeepScaleR-1.5B-Preview trained with 24k and tested with 32k §6. The table differences are 0.076 (GRPO) and 0.008 (DeepScaleR, 32k test) tab. 2.

7B model tab. 3

8B model tab. 4

1.5B model on DAPO-Math-17k (appendix A.3) tab. 5

The paper does not report DisCO-b numbers in a table. It says DisCO-b with either score is better than all compared baselines on average app. A.1 §6.

Training dynamics fig. 2 fig. 6 app. A.2. Baselines show premature saturation: entropy collapse for GRPO, GRPO-ER and Dr. GRPO, and excessive entropy growth for DAPO. TRPA becomes unstable in generation entropy after about 1100 steps on 1.5B and about 800 steps on 7B. DisCO keeps training reward rising and holds generation entropy near 0.22 §6 (near 0.2 in appendix A.2) app. A.2.

Ablations fig. 3 fig. 4 fig. 5.

• DisCO vs DisCO-b (L-ratio, 7B, 1000 steps): DisCO is better, especially on the AIME sets fig. 3 §6. Appendix A.1 reports the comparison for all settings and finds DisCO better on average in each fig. 5 app. A.1.
• Clipping vs non-clipping: run on DisCO-b with 1.5B, not on DisCO. Clipped L-ratio with eps_high 0.2 collapses entropy; with eps_high 0.28 entropy is too high. Both score worse than the non-clipped scores fig. 3 §6.
• KL regularisation (coefficient 0.001) vs constraint: the constraint is better on both 1.5B and 7B. The regulariser destabilised 7B training fig. 4 §6.
• Sensitivity to tau in {0.5, 1, 5, 10}: not sensitive fig. 4 §6.
• Component ablation on 1.5B: each component helps; the non-clipping score matters most fig. 4 §6.

Paper vs code

The official repository is Optimization-AI/DisCO. Its root has an MIT LICENSE; the verl/ subtree carries Apache-2.0 and deepscaler/ carries its own LICENSE. The DisCO losses sit in the bundled verl copy core_algos.py. No baseline configurations for GRPO, Dr. GRPO, DAPO or TRPA are in the repository, and there is no 8B script.

• Constraint gradient is half the paper's. Algorithm 1 gives G2 = 2 beta [KL - delta]+ grad KL, and the squared-hinge penalty has the same derivative alg. 1 eq. 9. The code computes max(beta*(kl-delta), 0).detach() * kl core_algos.py. That has gradient beta [KL - delta]+ grad KL, without the factor 2. The same pattern appears in the log-L function core_algos.py. Whether the factor matters depends on the intended beta, which the paper does not separately justify.
• Zero-variance questions. Algorithm 1 divides by |S+_q| and sums over negatives, so it is undefined for questions with no positive or no negative. The code masks these questions out of the sum core_algos.py but still divides by the number of unique question ids in the batch, counted before the mask core_algos.py core_algos.py. Skipped questions therefore lower the overall loss scale. The paper does not mention this.
• Reward values. The code treats reward == 1 as positive and reward == 0 as negative core_algos.py. The training reward for DeepScaleR-style data comes from deepscaler_reward_fn, which returns the boolean is_correct math_reward.py main_ppo.py main_ppo.py. The +1/-1 defaults in RewardConfig reward_types.py do not reach this path. A reward scheme with -1 for wrong answers would give no negatives at all.
• Micro-batching may split groups. The loss is computed per micro-batch dp_actor.py, and groups are formed from the uids in that micro-batch. The micro-batch size is ppo_max_token_len_per_gpu // max_seq_len dp_actor.py. The config says max_seq_len is overridden by the program ppo_trainer.yaml, but this repository does not set it in any .py, .sh or .yaml file. Taken at face value, the default of -1 makes the size negative. This is not verified by running the code.
• Logged pg_loss is not the objective. The neg_logmeanexp term is sum / (sum.detach() + eps), so its value is about 1 and only its gradient is meaningful core_algos.py. The gradient matches the log-sum-exp form in Algorithm 1, but the logged value does not.
• Evaluation schedule. The paper evaluates every 200 steps and reports the best evaluation §6. The scripts set test_freq=20 run_disco_Lratio_1.5b_8k.sh. The paper does not say which set selected the best checkpoint.
• Training length. The paper gives 1400 and 1000 steps §6. The scripts set total_epochs=30 and no step cap run_disco_Lratio_1.5b_8k.sh.
• Hardware. Appendix A gives 4 x 2 A100 40GB for 1.5B runs app. A. The 1.5B script says 8 A100-80GB on a single node run_disco_Lratio_1.5b_8k.sh.
• Max prompt length. The code uses 1024 run_disco_Lratio_1.5b_8k.sh; the paper is silent.
• Score definitions match. The L-ratio score is the mean over response tokens of the probability ratio core_algos.py. The log-L score is the mean token log-probability core_algos.py. Both match §5.1.
• Basic and DRO objectives match. DisCO-b uses the uniform mean over negatives core_algos.py core_algos.py (J_1). DisCO uses the log-sum-exp over negatives and the same per-positive form as Algorithm 1 core_algos.py. The two differ only by a constant that has zero gradient.
• KL estimator matches. The code uses the per-token k1 form with the kl option core_algos.py, as in Algorithm 1 alg. 1.
• Advantage unused. The DisCO loss functions take advantages but never use it.

No discrepancies were found for the optimiser, batch sizes, temperature, delta, beta or tau values, which match between the paper §6 and the scripts.

Pitfalls

• Two different betas. GRPO's KL coefficient is 0.001 §6. DisCO's penalty beta is 1e3 §6. The names clash, so check which one an implementation is using.
• The per-step old policy. Algorithm 1 refreshes pi_old once per step and then takes one update per mini-batch, four per step for 128 questions and mini-batch 32 alg. 1 §6. The ratio pi_theta / pi_old therefore drifts across the mini-batches. The code also takes the old log-probs once per batch dp_actor.py. The paper does not discuss this drift.
• Constraint is mostly inactive. The paper reports an average KL of about 2e-5 against delta of 1e-4 §6, so the penalty is zero on most steps by its own numbers. Implementations that use a plain KL term instead behave differently, as the paper's comparison shows fig. 4.
• Non-clipped ratios are unbounded. The L-ratio score has no clip. The paper attributes stability to the non-clipped scores together with the constraint §5.1, so dropping the constraint is a different method.
• Division by the number of questions. The mini-batch loss averages over all questions in the mini-batch alg. 1. The code also counts questions with no positive or no negative rollout in that average, while they add zero to the sum core_algos.py. Batches with many all-correct or all-wrong questions therefore get a smaller loss scale in the code.
• Tau differs by score. tau = 1 for L-ratio and tau = 10 for log-L, because the two scores have different scales §6. Using the same tau for both changes the method.
• Abstract rounding. The abstract says 7% over GRPO and 6% over DAPO for the 1.5B model; §6 says 7% over GRPO. The table averages differ by 7.6 and 6.0 points tab. 2.
• Entropy numbers. The text says generation entropy stays near 0.22 §6; the appendix says near 0.2 app. A.2. Treat it as approximate.
• Baselines are not reproducible from this repo. Only DisCO and DisCO-b scripts are present . The baseline numbers depend on the authors' settings for GRPO, DAPO, TRPA and Dr. GRPO, which are described only in prose §6.
• Scope. Experiments are limited to binary-reward math with distilled DeepSeek-R1 models of 1.5B to 8B parameters §6 §7. The paper says extension to non-binary rewards is future work §7.
• Figure-based claims. Several results (training dynamics, ablations) are only shown in figures. The card does not give numeric values for them.

Lineage

• Builds on: GRPO, which the paper analyses as a discriminative objective with a question weight §4. The clipping comes from PPO and the trust-region idea from TRPO §2 §5.1.
• Analysed variants: Dr. GRPO (arXiv 2503.20783) and DAPO (arXiv 2503.14476), whose question-level weights are derived in appendix B.3 app. B.3.
• Compared against: GRPO, GRPO-ER, Dr. GRPO, DAPO, TRPA, STILL-3-1.5B-preview, DeepScaleR-1.5B-Preview and GRPO-LEAD-7B §6.
• Superseded by: not stated.