Co-Evolving LLM Decision and Skill Bank Agents for Long-Horizon Tasks

In plain words

Games are a good way to test whether an AI agent can plan over many steps. Language models can play them, but they tend to forget what worked in earlier games and cannot reuse it well. This paper adds a library of reusable "skills" (short written playbooks with conditions and expected results) that the model can look up while it plays.

Two model roles work together. One model plays the game and picks a skill from the library to guide each move. A second model reads the play logs, cuts them into pieces that look like a single skill, and writes or updates the library. Both roles are trained with reinforcement learning, so the player learns to use skills well and the library improves from the player's new games.

The authors report that this small 8B model beats several frontier chat models on the average of four single-player games, and stays close to them on two multi-player social games. The social games are still below the strongest frontier model in the paper's table.

TL;DR

• COS-PLAY trains a decision agent and a skill-bank agent in alternating rounds, both on Qwen3-8B with separate LoRA adapters §4.3 §5.
• The decision agent retrieves a skill, updates a short intention, and picks one primitive action per step §4.1.
• The skill-bank agent turns unlabeled rollouts into skills with contracts (expected state changes) and maintains the bank §4.2.
• Authors report 25.1% mean relative reward gain over GPT-5.4 on 2048, Tetris, Candy Crush and Super Mario Bros tab. 1.
• Code is public under MIT. Several hyperparameters differ between paper and code (see Paper vs code).

Problem

Long-horizon games need multi-step reasoning, chaining of skills, and decisions under delayed reward and partial observation [Abstract]. LLM agents struggle to keep consistent long-horizon behaviour because they have no mechanism to discover, keep and reuse structured skills across episodes [Abstract] §1. Hand-written or distilled skills need human effort or a stronger teacher §5.3. The paper asks whether the skill bank and the decision agent should be learned jointly, and what makes a bank useful for decisions §1.

Assumptions: observations are converted to text, actions are discrete text actions, and the agent has a Gym-style interface §5 app. B.

Method

Setup §3. The environment is an episode of horizon T with observations, actions, reward and a done flag. Each experience also stores a latent intention z_t. A skill is stored as a protocol with five parts: Summary, Pre-condition, Plan, Success/Abort Criteria and Contract, where the Contract lists the expected state changes §3.

The method has two components and a loop between them §4.

Decision agent (§4.1). At each step the agent keeps a state summary, an intention and an active skill. It keeps the active skill, or retrieves a new one from the bank when the current skill is unavailable, exhausted or ineffective §4.1. The three roles are a skill retriever, an intention updater and an action policy eq. 1 eq. 2 eq. 3. The skill is a structured prompt (protocol steps, preconditions, termination cues). It guides action generation but does not replace low-level control §4.1. A sharp shift in the intention tag signals a subtask change and triggers a skill switch §4.1.

# Decision step  [§4.1] [eq. 1] [eq. 2] [eq. 3]
# Input:  observation, intention, active_skill, skill_bank
# Output: action, intention, active_skill

def decision_step(observation, intention, active_skill, skill_bank):
    # Keep the active skill unless it is unavailable, exhausted or
    # ineffective; otherwise retrieve a new one  [§4.1] [eq. 1]
    if needs_new_skill(active_skill, observation):
        active_skill = retrieve_skill(observation, skill_bank)
    # Update the intention from the observation and the chosen skill  [eq. 2]
    intention = update_intention(observation, active_skill)
    # Pick one primitive environment action  [§4.1] [eq. 3]
    action = act(observation, intention, active_skill)
    return action, intention, active_skill

Skill bank agent (§4.2). It runs four stages on rollouts from the decision agent.

1. Boundary proposal. For each timestep it computes a score from local signals: a predicate changing truth value between neighbouring states, a shift in the intention label, a jump in reward or in logged events, optional surprise peaks, and a move between acting and choosing a skill. High scorers become candidate cut points, and nearby candidates are merged §4.2.
2. Infer segmentation. It chooses which candidates are real boundaries and labels each segment. Each segment is summarised by its added and deleted predicates. Bank skills are scored by overlap with their contracts, with a small bonus for stronger prior support, and the top candidates are re-ranked by the skill-bank agent. A segment that matches no skill well gets the label "new skill", and its effects are kept as provisional evidence §4.2.
3. Contract learning. For each skill it pools added and deleted predicates across its labelled segments and keeps the consensus effects. An LLM summary may add to the consensus. The contract is verified on held-out segments, and only contracts with a high pass rate are written to the bank. Verified contracts also act as a prior for segmentation §4.2.
4. Bank maintenance. Five operations: refine (update a contract when new verified evidence changes it), materialize (promote grouped new-skill segments once there is enough evidence), merge (remove near-duplicates with overlapping contracts), split (mark overly broad skills for re-segmentation) and retire (drop skills no longer supported by recent trajectories). The bank agent may approve or reject proposed changes §4.2.

# Skill bank update from a batch of rollouts  [§4.2] [eq. 5]
# Input:  bank (B), rollouts (D)
# Output: updated bank

def update_bank(bank, rollouts):
    # Propose candidate skill boundaries, scored and merged  [§4.2]
    cut_points = propose_boundaries(rollouts)
    # Choose true boundaries and label segments with a bank skill
    # or "new skill"  [§4.2]
    segments = infer_segments(rollouts, cut_points, bank)
    # Pool effects per skill; keep consensus; verify on held-out  [§4.2]
    contracts = learn_and_verify_contracts(segments, bank)
    # Refine, materialize, merge, split or retire  [§4.2]
    return maintain(bank, segments, contracts)

Co-evolution (§4.3). The two agents alternate. The decision agent uses the current bank to collect trajectories, the bank agent updates the bank from them, and GRPO updates both §4.3 §5. The bank update is written as B^(u+1) = Φ_S(B^(u), D^(u+1)), where D is the accumulated rollout set eq. 5. Both agents start from one shared cold-start model §5.

# Co-evolution loop  [§5] [§4.3] [eq. 5] [tab. 3]
# Input:  game, teacher_seeds (60 per game), base model (Qwen3-8B)
# Output: decision agent, skill bank

def co_evolve(game, teacher_seeds, base_model, total_steps):
    # Cold start: SFT the shared model on GPT-5.4 seed trajectories  [§5]
    model = sft(base_model, teacher_seeds)
    # Initial bank and how it is built are not stated in the paper  [§5]
    bank = initial_bank
    rollouts = []
    for step in range(total_steps):
        # Decision agent plays episodes under the current bank  [§5]
        rollouts = rollouts + collect(model, bank, game)
        # Skill bank agent updates the bank from all rollouts  [eq. 5]
        bank = update_bank(bank, rollouts)
        # GRPO updates each LoRA adapter of both agents  [§4.3] [app. C]
        model = grpo_update(model, rollouts, bank)
    return model, bank

Adapters §4.3 §5: the decision agent has two LoRA adapters, action-taking and skill-retrieval. The bank agent has three, segmentation, contract and curator. Each is trained with GRPO and its own reward app. F.1 app. F.2.

Rewards (app. F). The action-taking reward is a per-step sum r_t = r_env + λ_f · r_follow + r_cost eq. 6. Here r_env is the game reward, r_follow shapes progress on the active skill's contract, and r_cost penalises skill switching. The paper sets λ_f = 0.1 app. F.1. The skill-retrieval adapter gets a delayed reward at skill switch, from the normalised environment reward during the skill, a time-efficiency term, a contract-completion term, an abort penalty and a small retrieval-confidence prior app. F.1. The bank agent's segmentation reward favours covering positive reward with existing skills and gives new-skill labels partial credit app. F.2. The contract reward is an F1 between predicted and observed predicate changes, plus sparsity, specificity and holdout terms app. F.2.

# Per-step action reward  [eq. 6] [app. F.1]
# Input:  r_env, r_follow, r_cost, lambda_f
# Output: scalar reward for the action-taking adapter

def action_reward(r_env, r_follow, r_cost, lambda_f=0.1):
    # Game reward plus shaping for progress on the active skill  [eq. 6]
    return r_env + lambda_f * r_follow + r_cost

Hyperparameters

Co-evolution settings per game tab. 3 app. C. All runs used an 8×A100 cluster app. C. Columns are as printed in the paper; "Clip" is the GRPO clip ratio and "KL" the KL coefficient.

For 2048 and Candy Crush all GRPO fields are the defaults tab. 3. The defaults are: clip 0.2, at most 4 epochs, no advantage clipping, learning rate 5e-5, KL 0.05 app. C.

Other settings:

Results

Author-reported. Setup: six games, Qwen3-8B base, 16 evaluation rollouts for single-player games and 10 per player for social games, compared with GPT-5.4, Gemini-3.1-Pro, Claude-4.6-Sonnet and gpt-oss-120b §5 tab. 1. Metrics are native game reward (single-player), team win rate (Avalon) and mean supply centres (Diplomacy) §5.

Main table tab. 1

Source: tab. 1. Values are rewards (single-player) or win rate and mean supply centres (social), with 95% intervals as printed.

Headline claims, as the paper states them §5.1:

• The authors state a mean gain of 25.1% against GPT-5.4 across the single-player games §5.1 tab. 1.
• On the social games, the paper says COS-PLAY trails Gemini-3.1-Pro by about 1% in win rate on Avalon, and beats Gemini-3.1-Pro by 8.8% on Diplomacy §5.1. The table values are 39.0 vs 42.0 (Avalon) and 2.96 vs 2.72 (Diplomacy) tab. 1.
• Each game reaches strong performance within at most 25 co-evolution steps §5.1 tab. 3.

Ablations §5.2 tab. 1. The paper reads these as showing that the gain comes from co-evolution. Skills or RL alone do not explain it. Variants with a skill bank that does not match the current policy do worse, and the paper calls them "often harmful" §5.2. Single-player averages: base Qwen3-8B 379.6; SFT without skill 409.5; SFT with first bank 465.7; SFT with final bank 359.5; GRPO without skill 359.9; GRPO with first bank 305.2; full COS-PLAY 924.4.

Merged LoRA app. E tab. 6. On Candy Crush, a variant that merges adapters into one per agent reaches 502.5 ± 22.4, below the stage-specific adapters at 648.8 ± 38.8 tab. 6. The number of rollouts behind tab. 6 is not stated.

Skill reuse §5.3 tab. 2. Reuse statistics per game, as printed:

Source: tab. 2. Gini is the reuse-concentration measure in the caption; Avg. ver counts contract refinement versions.

Diplomacy bank over training fig. 3. The active bank stays at roughly 55 to 70 skills, 121 skills are discovered overall, and periodic curation removes 53 redundant skills by merge and split fig. 3.

General reasoning app. G tab. 7. MMLU-Pro accuracy: Qwen3-8B 61.99%, COS-PLAY 61.15%. Math-500 exact match: 46.40% vs 44.60% tab. 7.

Paper vs code

Official repository: wuxiyang1996/COS-PLAY, linked in the paper's abstract, at commit e1d7dcc. It is MIT-licensed LICENSE. The GitHub owner matches the first author's name in the author list. The release includes the decision and skill-bank agents, GRPO scripts, and per-game run scripts . Discrepancies found:

1. Adapter layout. The paper uses 2 decision adapters and 3 bank adapters (segmentation, contract, curator) §4.3. The skill-bank config lists four adapters: boundary, segment, contract and retrieval skillbank_lora.yaml.
2. LoRA rank. The paper does not state a rank. The trainer sets r = 16 and alpha = 32 config.py.
3. Learning rate. The paper gives one GRPO learning rate per game, 5e-5 by default app. C. The trainer config starts at 1e-4 and decays toward a floor config.py config.py config.py. The CLI sets the start rate to twice the given rate run_coevolution.py. Which schedule the paper's runs used is not verified.
4. KL coefficient. The paper uses 0.05 flat by default app. C. The trainer ramps it from 0.01 to 0.05 config.py.
5. Steps per game. The table gives 10 steps for 2048 and 7 for Tetris tab. 3. The run scripts default to 25 steps for 2048 run_2048.sh and 15 for Tetris run_tetris.sh. Diplomacy matches: 25 steps, 28 episodes, LR 1e-5 run_diplomacy.sh run_diplomacy.sh.
6. Episode length. The appendix caps 2048, Tetris and Mario at 200 steps app. B. The Tetris script says episodes "can run 500 steps" run_tetris.sh, and the Mario script says "~500 steps" run_super_mario.sh.
7. Reward costs. The paper gives λ_f = 0.1, which matches reward_func.py. It gives no values for the cost terms. The code uses skill-switch −0.10, memory query −0.05, skill query −0.05 and skill call −0.02 reward_func.py. Predicate bonus 0.05, completion bonus 0.20 and no-progress penalty −0.01 are in reward_func.py.
8. Contract reward weights. The paper lists the contract reward's components only. The code weights them 0.18 (both-sides coverage), 0.13 (sparsity), 0.13 (specificity) and 0.12 (a content fingerprint tiebreaker) rewards.py. The paper's reward-saturation tiebreaker text is commented out in the source, so it is not part of the published text.
9. Bank thresholds. The paper says only "sufficiently high pass rates". The code sets split pass-rate threshold 0.70 and child threshold 0.80 config.py config.py.
10. Boundary threshold. The paper gives no threshold for keeping candidate cuts. The signal extractor defaults to 0.3 signal_extractors.py.
11. GRPO group size. The paper does not state a group size. A module default of 4 exists config.py.
12. Segmentation decoder. The paper refers to a decode score and margin app. F.2 without naming a decoder. The code has DP and beam decoders dp_decoder.py beam_decoder.py. Which one the reported runs used is not verified.

The paper says it will release the implementation, prompt templates and configs app. A. This card did not check the released prompts against the paper.

Pitfalls

• How "25.1%" is computed. The abstract's figure matches the mean of four per-game relative gains over GPT-5.4 tab. 1 (our arithmetic: 41.1%, 11.5%, 21.8%, 26.2%, mean ≈ 25.1%). The ratio of the two "Avg." cells gives 924.4 / 717.4 ≈ 28.9%. The paper does not define the metric.
• Averages mix reward scales. The "Avg." column averages raw native rewards across games with different scales, so 2048 and Super Mario dominate it tab. 1.
• Social-game claims are narrower than the abstract. Avalon: GPT-5.4 scores 65.0 and COS-PLAY 39.0 tab. 1. The text says "trailing by about 1%", but the table gap is 3.0 points against Gemini, and the intervals (±9.4 and ±13.2) overlap tab. 1 §5.1. Diplomacy: GPT-5.4 scores 4.70 against COS-PLAY's 2.96 tab. 1.
• GPT-5.4 is both opponent and a row. Evaluation uses GPT-5.4 as the opponent app. A. The paper does not say how the GPT-5.4 row was produced in that setup.
• Hidden cost. Cold start needs 60 GPT-5.4 trajectories per game §5. Training uses GPT-5-mini opponents for social games §5. The rollout budget is steps × episodes per step: 80 (2048), 80 (Candy Crush), 56 (Tetris), 160 (Mario), 400 (Avalon), 700 (Diplomacy) tab. 3 (our arithmetic). Wall-clock time and API cost are not stated.
• Few-shot wording. "Within 25 steps" refers to co-evolution rounds of 8 to 28 episodes tab. 3, not single episodes.
• Intention timing is ambiguous. §4.1 says the intention tag is updated after each primitive action, but eq. 2 computes the intention before the action from the observation and skill §4.1 eq. 2. Which order the code uses is not verified here.
• Skill-switch trigger undefined. "Sharply" shifts in the intention tag trigger a switch §4.1, but the paper gives no threshold or test.
• GRPO objective not written. The paper names GRPO and gives clip, KL, epochs and LR app. C, but not the loss or advantage normalisation. Group size is not stated.
• Merged-LoRA ablation is thin. It is one game, and the number of rollouts is not stated app. E tab. 6.
• Only evaluation variance is reported. Intervals come from 16 or 10 evaluation rollouts tab. 1. The paper does not report variance across training seeds or repeated training runs.
• Qualitative and showcase material is extensive. It is in the appendix, but it is not used for any number in this card app. I.

Lineage

• Builds on: GRPO for policy updates §4.3 and LoRA adapters §4.3, both cited by the paper but not compiled here. Base model Qwen3-8B §5.
• Compared against: GPT-5.4, Gemini-3.1-Pro, Claude-4.6-Sonnet and gpt-oss-120b §5 tab. 1.
• Related benchmarks used: 2048, Candy Crush, Tetris and Super Mario via LMGame-Bench and Orak; Avalon and Diplomacy via AgentEvolver §5.