Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling

In plain words

Turning a word problem into an optimization model (for example, how many tables and chairs to make to earn the most profit) is hard for language models. They often write code that does not run, or code that runs and gives the wrong answer. A solver is a program that finds the best answer to such a model, so it can also check whether generated code is valid.

This paper trains a model with reinforcement learning, where the reward comes from running the model's code in a solver. The reward asks three questions: is the answer laid out in the expected format, does the code run, and is the objective value right. The same checks also help pick the most consistent answer among several samples when building training data.

The authors report that their 7B and 32B models beat several larger general-purpose models on the benchmarks they tested. These results are their own measurements.

TL;DR

• SIRL is an RL fine-tuning method for LLMs that write optimization models and Gurobi Python code. It starts from Qwen2.5 instruct models without SFT.
• The reward is computed by executing the generated code with a solver: a format term, an execution term and an answer term §3.3 §10.
• A "Partial KL" penalty is applied only to the later sections of the answer, not to the reasoning §3.2 eq. 5.
• Author-reported macro accuracy over five benchmark columns: 60.6 for 7B and 68.3 for 32B tab. 1.
• The shipped training script is a single-stage run with a different reward and KL setup from the paper (see Paper vs code).

Problem

Optimization modeling requires turning a natural-language problem into decision variables, an objective and constraints, then into solver code. Offline methods (SFT, DPO, KTO) imitate curated demonstrations and do not guarantee that code runs or that solutions are feasible §1. Prompt-based and agent approaches keep the model frozen and are sensitive to prompt design §1. Reasoning models without tools struggle on these tasks: the paper argues that global structure and numerical computation are better handled by a solver [§4.3.1, "Performance without External Solver Assistance"] tab. 9.

Method

Overall setup §3 §3.2

The policy \pi_\theta is an LLM that writes a chain of thought \mathbf{z} in three tagged parts: reasoning (<think>), a concise mathematical model (<model>) and executable code (<python>). The objective value y = g(x, \mathbf{z}) is the result of deterministically executing the code §3.2. The training goal is to maximise expected reward eq. 1.

Data synthesis §3.1 fig. 1

1. Start from a seed set of 686 industry cases from OR textbooks plus 100 industry scenarios §8 §11. Sample seed questions, combine each with a scenario from a predefined list, and generate structurally similar new problems §3.1.
2. Augment the problems to raise semantic, modeling and solution complexity §3.1.
3. An LLM judge checks each generated problem for practical relevance and semantic consistency §3.1 §9.
4. Assign 10 LLM roles to each problem. Each role writes a model and code. The code is executed to get the objective value, feasibility status and a .lp file §3.1 §9.
5. Repair execution errors and regenerate the model and code when a solution is infeasible, using reflection-style prompts §3.1 §9.
6. Filter out a pair if the baseline Qwen2.5-32B-Instruct gets it right in 8 of 10 attempts, as too easy §3.1 §11. About 70,000 pairs remain; 10,000 are sampled for training §11.

Instance-enhanced self-consistency §3.1 fig. 2 eq. 2 eq. 3

For each question, the roles' answers are scored by consensus. A role's score is a weighted sum of square roots of consensus counts over four features taken from its .lp file:

• O_r: the objective value;
• D_r: the optimisation direction (max or min);
• N_{bin,r}: the number of binary variables;
• N_{int,r}: the number of general integer variables.

The consensus count \psi(X_r) counts the roles that give the same value of feature X eq. 2. The score is S(r) = w_1\sqrt{\psi(O_r)} + w_2\sqrt{\psi(D_r)} + w_3\sqrt{\psi(N_{bin,r})} + w_4\sqrt{\psi(N_{int,r})} eq. 3. All four weights are 1 in the current implementation §3.1. The answer with the highest score is kept §3.1.

Policy update: REINFORCE++ with Partial KL §3.2 eq. 4 eq. 5

The update uses REINFORCE++ with a clipped token-level surrogate §3.2. The per-token reward is the outcome reward placed on the end-of-sequence token, minus a KL penalty term \beta\,\mathrm{KL}(j,t) eq. 4. The advantage is normalised across the mini-batch eq. 4.

The KL term is switched on or off by section index j. The paper omits it for the reasoning sections and keeps it for the model and code sections j \in \{m-1, m\} eq. 5. It uses the estimator \rho - \log \rho - 1, where \rho is the ratio of the current policy to the reference policy eq. 5. The authors argue that dropping KL in reasoning encourages exploration, while keeping it in the formulation and code preserves the output format §3.2.

# SIRL update step  [§3.2] [eq. 4]
# Input:  batch of (problem x, ground truth y*); K rollouts per problem
# Output: updated policy parameters theta

def sirl_update(policy, reference, problems, truths, solver):
    # Sample K complete responses per problem  [§3.2]
    responses = policy.sample(problems, num_samples=K)
    # Score each response by running its code  [§3.3] [§10] [eq. 6]
    rewards = [reward(z, y_star, solver) for z, y_star in zip(responses, truths)]
    # Token-level KL to the reference policy  [§3.2] [eq. 5]
    kl = token_kl(policy, reference, responses)
    # Keep the KL only in the model and code sections  [§3.2]
    kl = kl * partial_kl_mask(responses)
    # Outcome reward at the end-of-sequence token, minus the KL penalty  [eq. 4]
    token_rewards = eos_reward(responses, rewards) - beta * kl
    # Normalise advantages across the mini-batch  [eq. 4]
    advantages = normalise(token_rewards)
    # Clipped policy-gradient step (REINFORCE++)  [§3.2]
    return policy_gradient_step(policy, advantages)

Reward design and two-stage curriculum §3.3 eq. 6 §10

The reward is a sum of terms. Stage 1 has three: a format term, an execution term and an accuracy term eq. 6. Stage 2 adds a bonus term eq. 6.

• Format R_{format}: 0.5 if the response contains all required tags in the correct order, otherwise 0 §10.
• Execution R_{exec}: 1 if the code is executable, otherwise 0 §10.
• Accuracy R_{accur}: 2 if the objective matches y^* within 0.01, otherwise 0 §10 §3.3.
• Bonus R_{bonus} (stage 2 only): 1 only if the answer is correct and the .lp file shows advanced modelling, such as Big-M, binary variables or non-linear forms §10. The stage-2 weight is not stated.

# Stage-1 reward  [§10] [eq. 6]
# Input:  response z, ground truth y*, solver runner
# Output: scalar reward in [0, 3.5]

def stage1_reward(response, truth, solver):
    # 0.5 if all tags appear in the required order  [§10]
    format_part = 0.5 if tags_in_order(response) else 0.0
    # 1 if the extracted code runs without error  [§10]
    run = solver.run(extract_python(response))
    exec_part = 1.0 if run.ok else 0.0
    # 2 if the objective is within 0.01 of y*  [§10] [§3.3]
    accuracy_part = 0.0
    if run.objective is not None:
        if abs(run.objective - truth) <= 0.01:
            accuracy_part = 2.0
    return format_part + exec_part + accuracy_part

Training runs in two stages. Stage 1 teaches the basic formulation and execution; stage 2 adds the bonus to push toward harder models §3.3. Each stage took about 24 hours on 8 H100 GPUs §11.

Hyperparameters

Unless marked, values are for SIRL-Qwen2.5-7B. The paper does not describe a 32B training configuration.

Table 6 and table 7 (appendix) are reproduced here as part of the table above.

Results

Author-reported. Setup: five benchmark columns (NL4OPT, MAMO Easy, MAMO Complex, IndustryOR, OptMATH); pass@1; relative error below 1e-6 §4 §11. Baseline numbers marked * are from the original or reproduced papers tab. 1. Macro AVG is the mean of the five columns (checked: SIRL-7B gives 60.6, SIRL-32B gives 68.3).

Table 1: main results tab. 1 §4.1

Table 2: value-based vs instance-enhanced self-consistency (pass@1 and @5/@10) tab. 2 §4.2

The "Diff" rows are as printed in the paper; the authors do not define them in the text. Their "Average" column is over the five benchmarks.

Table 3: surrogate function (KL design) ablation tab. 3 §4.3.1

ER is the execution rate: the share of solutions that compile and return a valid result §4.3.1.

Table 4: reward design ablation tab. 4 §4.3.2

Table 6: training parameters tab. 6 §11: listed in the Hyperparameters table above.

Table 7: sampling parameters tab. 7 §11: n = 1; temperature 0.5; top-p 0.9; max tokens 8192; repetition penalty 1.02.

Table 8: sampling-based metrics (mean ± std over runs, temperature 0.5, top-p 0.95) tab. 8 §12

Table 9: no solver (textual reasoning only) tab. 9 §12

Table 10: base vs SFT vs RL (7B) tab. 10 §12

The RL row's MAMO Easy value (90.0) differs from Table 1 and Table 4 (91.7); see Pitfalls.

Table 11: error types, Qwen2.5-7B-Instruct tab. 11 §12

Table 12: error types, SIRL-Qwen2.5-7B tab. 12 §12

Table 13: Partial KL full ablation (acc / ER) tab. 13 §13

Figures: fig. 1 shows the data synthesis pipeline; fig. 2 shows the instance-enhanced self-consistency example; fig. 3 shows the SIRL training framework. They are described here and not reproduced.

Other author-reported findings: the 7B model beats the 14B offline baselines and OptiMUS §4.1. The 32B model's macro average exceeds DeepSeek-V3.1, DeepSeek-R1 and OpenAI-o3 §4.1 tab. 1. Sampling at temperature 0.5 and top-p 0.95 gives small standard deviations on the tested benchmarks tab. 8. Two-stage rewards win on most benchmarks, but stage-2-only is best on OptMATH (33.1 vs 30.5) tab. 4.

Paper vs code

Repository: https://github.com/Cardinal-Operations/SIRL at b329ff288e9240ecd41813fa41e7f003fd5ca47d. The paper links it [abstract] and the affiliation is Cardinal Operations [title page], so it is official. The repository has no LICENSE file. Its vendored verl files (SIRLTrain/partial_kl/kl_ppo/*.py) carry Apache-2.0 headers. The top-level license is therefore not stated.

• KL mechanism and size. The paper's table lists a KL loss type low_var_kl with coefficient 0.005 and "Use KL Loss" off tab. 6. The paper's method puts KL in the reward with \beta §3.2 eq. 4. The training script sets use_kl_in_reward=True and kl_coef=0.0005, with no KL loss run_partialKL.sh.
• Which tokens get the KL. The paper applies KL to sections m-1 and m, the model and the code §3.2 eq. 5. The code finds the first <python-style token sequence and zeroes the KL mask before it, so the mask starts at the code section and the <model> section is not penalised ray_trainer.py ray_trainer.py ray_trainer.py ray_trainer.py.
• Full KL is computed, then discarded. apply_kl_penalty first computes token rewards with full KL ray_trainer.py, then recomputes them with the partial mask ray_trainer.py. The partial version is the one used.
• Reward format. The paper's format term is binary, 0.5 for all tags in order §10. The code's format_reward is graded: it returns 2 - penalties, where penalties are 1/8, 1/4 or 1/3 per error, and the score is then multiplied by 0.5, so it can reach 1.0 batch_score_gurobi.py batch_score_gurobi.py.
• Accuracy term. The paper gives 2 for a correct answer §10. The code gives 1.0 times a boolean batch_score_gurobi.py. The paper's tolerance is |y - y^*| \le 0.01 in §3.3 and 10^{-6} relative error in §11. The code checks |y - \hat y| / (|y| + 1) < 10^{-6} batch_score_gurobi.py.
• Execution term. Both match: 1 if the code finishes with status Done batch_score_gurobi.py.
• Stage-2 bonus. The paper's bonus reads the .lp file for Big-M, binary or non-linear forms §10. stage2_reward_gurobi.py does not use the .lp file for the bonus. It adds a solution-match term sol_reward, which applies only when the data has a sol field stage2_reward_gurobi.py stage2_reward_gurobi.py. The training script points at batch_score_gurobi.py, which has no bonus term run_partialKL.sh.
• Batch and rollout sizes. Paper: batch 128, 8 rollouts, mini-batch 8, micro-batch 4 tab. 6. Script: batch 32, 16 rollouts, mini-batch 32, micro-batch 2 run_partialKL.sh run_partialKL.sh run_partialKL.sh run_partialKL.sh.
• Clip ratio high. Paper 0.28 tab. 6; script 0.30 run_partialKL.sh.
• Matches. Learning rate 1e-6, prompt length 2048, response length 8192, left truncation, clip low 0.20 and REINFORCE++ all agree run_partialKL.sh run_partialKL.sh.
• Not checked. The self-consistency voting code, the sampling temperature at rollout time, and the data-filtering pipeline were not found or not read.

Pitfalls

• Benchmark count. The text says "four benchmarks" §4. Table 1 has five columns, because MAMO is split into Easy and Complex tab. 1.
• Test-set corrections. The authors corrected NL4OPT, MAMO and IndustryOR test sets and say so §11. Baseline rows marked * use the original or reproduced numbers on uncorrected or differently curated data, so the comparison is not like for like.
• Internal inconsistency. Table 10's RL row gives MAMO Easy 90.0%; Table 1 and Table 4 give 91.7% for the same model tab. 10 tab. 1 tab. 4.
• Appendix section numbers. The appendix sections in the source have no \appendix command. Their numbers here (§6 to §13) are counted from the source order. Check them against the compiled PDF before citing.
• Missing 32B setup. Only the 7B training configuration is given §11. The 32B training recipe is not stated, but 32B results are reported tab. 1.
• Stage-2 weight and curriculum. The stage-2 bonus weight is not stated. The paper does not say how the switch from stage 1 to stage 2 is scheduled.
• Overwritten KL. The code computes full KL and then overwrites token rewards with the partial version ray_trainer.py. Re-implementing from the paper alone would not produce this behaviour.
• Shared mask tensor (not verified). partial_kl_tensor writes zeros into response_mask in place ray_trainer.py, and that tensor is the same batch entry the advantage code reads ray_trainer.py ray_trainer.py. If the policy loss uses the same mask, reasoning and model tokens would drop out of the update. I did not run training to confirm this. Verify before copying the pattern.
• Missing tag fallback (inferred from code). When a tag is not found, the code sets its start index to 0 ray_trainer.py. The KL mask then covers the whole response, which is the Full KL behaviour.
• Solver licences. The training and evaluation need Gurobi or COPT. The README asks users to obtain a solver licence README.md.
• Reward hacking. The conclusion says reward hacking persists and that OptMATH and IndustryOR results remain weak §5.
• Cost. The paper reports 384 GPU-hours for training the 7B model §11. It does not report the cost of data synthesis.
• Stochastic evaluation. The main results use n = 1 tab. 7; variance is reported only for the sampling-based run in tab. 8.

Lineage

• Builds on: REINFORCE++ §3.2, the OR-Instruct data pipeline of ORLM §3.1, self-consistency §3.1 §4.2, and the no-KL practice of DAPO and Skywork-OR1 that the paper's ablation compares against §4.3.1.
• Compared against: GPT-4, DeepSeek-V3.1, DeepSeek-R1, OpenAI-o3, OptiMUS, ORLM, LLMOpt and OptMATH models tab. 1.
• Superseded by: not stated.