wikillm

Paper · 2026-11-02 · cs.CL

TRIO: Task Refinement with Individual Optimization for Retrieval-Augmented Generation

Three trained LLM agents for retrieval-augmented QA: a query writer (RL, reward = F1 gain), a sufficiency judge (SFT on labels from 16 sampled answers) and an answer writer (two-stage RL). They loop retrieval until the judge says enough. Author-reported EM average on five QA sets: TRIO-8B 0.4294 vs 0.4034 for Search-R1.

Retrieval-augmented generationMulti-agent systemsReinforcement learning with verifiable rewards (RLVR)Multi-hop reasoningretrieval-augmented-generationmulti-agentquery-expansionreinforcement-learningmulti-hop-qa

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Question-answering systems often look up documents before they answer. In this setup the lookup is the weak point. One model is asked to write search queries, decide when it has found enough, and write the answer, and it does all three poorly.

The paper splits that job among three specialised models. One writes the next search query when the evidence looks incomplete. One judges whether the evidence is already enough to answer. The third writes the final answer. Each model is trained on its own signal. The query writer is credited when its query improves the answer. The judge learns from whether the answer model reliably gets a question right. The answer model is rewarded for matching the gold answer and for staying short.

At question time the three take turns: search, check, search again if needed, then answer. The aim is fewer, more useful lookups and short answers. The gains are reported by the authors on their own benchmarks.

TL;DR

  • TRIO runs three trained agents in a loop: a query writer, a sufficiency judge and an answer writer §3 §3.5.
  • The judge's labels come from the answer agent: 16 sampled answers per question, all correct means sufficient, all wrong means insufficient §3.4.
  • Author-reported EM on five QA sets: TRIO-8B has an average of 0.4294 against 0.4034 for Search-R1 tab. 1. The AVG column cannot be reproduced as a plain mean of the five columns (see Pitfalls).
  • Evaluated on short-answer QA only. Live web search and long-form answers are not validated § Limitations.
  • The official code is Apache-2.0 and partial. The reward server and the training data are not in the repository query_train.sh.

Problem

Retrieval-augmented generation (RAG) pipelines retrieve too little or too much, and the generator can hallucinate when the retrieved context is noisy §1. Existing multi-agent RAG systems still share one backbone across roles without adaptive adjustment §1. The paper targets multi-hop question answering, where a system must issue several queries before it has the evidence it needs §3 §4.1. Its experiments assume an E5-base-v2 retriever over the Wikipedia 2018 dump, and training questions from HotpotQA and 2WikiMultiHopQA §4.1 app. B.

Method

TRIO uses three agents, each trained for one subtask §3.1:

  • Answer agent A_g: writes the final answer from the retrieved pool §3.2.
  • Query extension agent A_q: proposes one new keyword query per round §3.3.
  • Judge agent A_j: decides whether the pool is sufficient to answer §3.4.

Training follows Algorithm 1 alg. 1:

  1. Split the training questions into a hard set Q1 and a normal set Q2. The paper does not define the split (see Pitfalls) alg. 1.
  2. Run a retrieval debate on Q1 to build T1. Train the answer agent on T1 with two-stage GRPO-style RL alg. 1 §3.2.
  3. Run a single retrieval per question on Q2 to build T2. Train A_q with GRPO and reward R_q alg. 1 §3.3 eq. 7.
  4. Answer every question in T2 with the trained answer agent, 16 samples each, to build T3 alg. 1 §3.4.
  5. Train the judge A_j by supervised fine-tuning on T3 alg. 1 §3.4.

Answer agent §3.2. Questions come from multi-hop QA sets. Each is given a difficulty label (easy, medium or hard), and the medium and hard ones are kept, following R1-Searcher §3.2. A retrieval-only debate, following DRAG, refines the queries for at most three rounds. The query pool and chunks after each round become one training sample, and the number of queries and chunks is varied per question §3.2 app. C. Training has two RL stages. Stage 1 uses the top-3 chunks per query and the reward in eq. 4. Stage 2 uses the top-5 chunks and the reward in eq. 5. The definitions are in eq. 1, eq. 2, eq. 3 and eq. 6.

Query extension agent §3.3. Each original question is used as the first query. The reward for a new query is the change in F1 of the answer agent's output, computed before and after the chunks from that query are added eq. 7. Training uses GRPO §3.3. The prompt asks for at most eight keywords, no stopwords, and a query that differs from existing ones app. B.

Judge agent §3.4 app. B. The label for each sample comes from 16 stochastic answers by the answer agent, with a test-time labelling method following TTRL §3.4. Sixteen correct answers give "Sufficient Information". Sixteen wrong answers give "Insufficient Information". Mixed outcomes are dropped. The judge is trained by SFT and must output one of the two labels only app. B.

Inference loop §3.5 app. B §4.4. The initial pool comes from a retrieval with the raw question. Each round, the query agent writes a query, the retriever returns chunks for it, and the judge decides whether to stop. The round budget is three. The answer agent then writes the answer from all queries and chunks, and nothing else.

Pseudocode
# Inference loop for one question  [§3.5] [app. B]
# Input:  question x, retriever (top 5 chunks per query)
# Output: final answer y

def answer_question(question, retriever, query_agent, judge, answer_agent):
    # Retrieve with the raw question first  [app. B]
    query_pool = {question: retriever(question, top_k=5)}
    # At most three rounds  [§4.4] [code: config/init_config.py#L53-L54]
    for round_index in range(3):
        # The query agent proposes one keyword query  [app. B]
        new_query = query_agent(question, query_pool)
        # The judge is also called in round 0, but its verdict is
        # ignored there  [code: model/multi_agents.py#L300]
        verdict = judge(question, query_pool)
        if round_index > 0 and "Sufficient Information" in verdict:
            break
        # An empty proposal ends the loop  [code: model/multi_agents.py#L307]
        if new_query == "":
            break
        # No duplicate check; a repeated key overwrites the entry
        # [code: model/multi_agents.py#L320]
        query_pool[new_query] = retriever(new_query, top_k=5)
    # The answer agent sees every query and its chunks  [§3.5] [app. B]
    return answer_agent(question, query_pool)

Reward definitions. All of these are from the paper §3.2 §3.3.

EM(y, y^*) = \begin{cases} 1, & y = y^* \\ 0, & \text{otherwise} \end{cases} eq. 1

PR(y, y^*) = \frac{IN(y, y^*)}{PN(y)} eq. 2

L(y) = \begin{cases} -1, & \text{length of } y > l \\ 0, & \text{otherwise} \end{cases} eq. 3

R_{a1} = EM(y, y^*) + \alpha \cdot PR(y, y^*) + L(y) eq. 4

R_{a2} = EM(y, y^*) + \alpha \cdot F1(y, y^*) + L(y) eq. 5

F1(y, y^*) = \frac{2 \cdot IN(y, y^*)}{PN(y) + PN(y^*)} eq. 6

R_q = F1(y_{n+1}, y^*) - F1(y_n, y^*) eq. 7

Here IN is the word count of the overlap between the two answers and PN is a word count. The paper calls alpha and l "tuned" and does not give their values §3.2.

Pseudocode
# Judge label from 16 sampled answers  [§3.4] [app. B]
# Input:  question, retrieved pool, gold answer
# Output: a label, or None if the samples are mixed

def judge_label(question, pool, gold, answer_agent):
    # Draw 16 stochastic answers; the temperature is not stated  [§3.4]
    answers = [answer_agent(question, pool) for _ in range(16)]
    # Correctness test not stated; EM [eq. 1] is the reward metric
    correct = [is_correct(a, gold) for a in answers]
    if all(correct):
        return "Sufficient Information"
    if not any(correct):
        return "Insufficient Information"
    # Mixed outcomes are dropped, about 10.7% of samples  [app. B]
    return None

Hyperparameters

NameValueWhereNotes
RetrieverE5-base-v2§4.1Index built on the 2018 Wikipedia dump §4.1. Repo uses the wiki18_100w corpus base_config_TRIO.yaml
Chunks per query, inference5base_config_TRIO.yamlNot stated in the paper
Chunk cap per question20§4.1Matches 4 queries x 5 chunks (inferred from code)
Rounds, inferencemax 3§4.4 init_config.pyRound 0 always adds a query (see Paper vs code)
Chunks per query, answer RL stage 13§3.2
Chunks per query, answer RL stage 25§3.2
Debate rounds, data generationmax 3§3.2
BackbonesLlama-3.1-8B-Instruct (TRIO-8B); Qwen3-4B-Instruct (TRIO-4B)§4.1Only Llama scripts are in the repo
Training questions8,148app. BFrom HotpotQA and 2WikiMultiHopQA
Rollouts per prompt (GRPO)16app. B query_train.sh
Learning rate (GRPO)1e-6app. BRepo scripts use 2e-6 (see Paper vs code)
KL coefficient (GRPO)1e-4app. B answer_train.sh
Train batch / rollout batch (GRPO)512 / 32app. BRepo scripts differ (see Paper vs code)
Alpha, length limit lnot stated§3.2Both described as tuned
Judge SFT learning rate, epochs, batchnot stated§3.4Repo: 1e-5, 1 epoch, batch 16 sft_train.sh
Judge training samples44,473 kept of 49,785app. B9,087 sufficient, 35,386 insufficient (about 20% / 80%)
Decoding, inferencenot statedRepo: greedy (do_sample False) base_config_TRIO.yaml
Random seednot statedRepo: 2024 base_config_TRIO.yaml
Output cap, query and judge32 tokensmulti_agents.py multi_agents.pyQuery output is at most 8 keywords app. B
Output cap, answer1024 tokensbase_config_TRIO.yamlAnswer RL used 16 tokens answer_train.sh
Hardware, training2 H100 GPUs; reward on 2 L40 GPUsapp. B
Hardware, evaluation2 L40S GPUsapp. B §4.6
Training cost93.00 H100 GPU-hours (8B); 58.10 (4B)app. ESum of the four stages per model

Results

Author-reported. Setup: exact match (EM) on five QA sets §4.1. Two are in-domain (HotpotQA and 2Wiki, where parts of the training sets were used) and three are out-of-domain (NQ, Musique, Bamboogle) §4.1. Baselines were rerun with FlashRAG on the same retriever and corpus §4.1. Full rows are in tab. 1.

MethodHotpotQA2WikiNQMusiqueBamboogleAVG
Qwen3-4B (single-hop)0.27700.15840.33570.05710.22400.2074
Llama3.1-8B (single-hop)0.30950.16390.37200.05330.16000.2237
Qwen3-30B (single-hop)0.29790.24790.31750.12780.24000.2606
SuRe0.22570.11660.35320.06530.16800.1757
IRCoT0.18410.12660.21710.06900.20800.1505
Iter-RetGen0.33500.16510.33430.07240.19200.2282
DRAG0.30600.20940.35740.11710.28800.2490
Self-RAG0.15960.11590.34570.04220.06400.1530
Search-R10.41160.43010.43820.18360.46400.4034
R1-Searcher0.40120.40460.36810.17790.46400.3779
TRIO-4B0.41740.43730.38780.15470.46400.3988
TRIO-8B0.43500.47100.43390.18570.49600.4294

tab. 1. Search-R1 is ahead on NQ (0.4382 against 0.4339 for TRIO-8B). The paper reports TRIO-8B's average as 6.45% above the second-best method and 72.45% above DRAG §4.2. Both figures are derived from the AVG column (see Pitfalls).

Ablation tab. 2. EM / F1 on the four multi-hop sets, with one trained agent replaced by the base backbone at a time:

VariantHotpotQA2WikiMusiqueBamboogle
w/o query agent0.4235 / 0.55550.4547 / 0.51910.1357 / 0.24560.4560 / 0.5590
w/o judge0.4284 / 0.55870.4765 / 0.54160.1593 / 0.26350.4480 / 0.5643
w/o answer agent0.3735 / 0.49260.2778 / 0.36180.1051 / 0.18550.2800 / 0.3967
TRIO (full)0.4350 / 0.57700.4710 / 0.56600.1857 / 0.28180.4960 / 0.6200

tab. 2. The paper says that without the judge the variant is strongest, and that on 2Wiki it beats full TRIO in EM §4.3. The answer agent matters most.

Rounds tab. 3. EM / F1 on Musique and Bamboogle with a fixed number of rounds against the judge-controlled flexible loop:

SettingMusiqueBamboogle
Fixed 10.1556 / 0.25970.4480 / 0.5614
Fixed 20.1585 / 0.26090.4480 / 0.5587
Fixed 30.1585 / 0.25990.4560 / 0.5811
Fixed 40.1547 / 0.25610.4560 / 0.5816
Fixed 50.1531 / 0.25160.4560 / 0.5818
Flexible (max 3)0.1857 / 0.28180.4960 / 0.6200

tab. 3. The paper reads this as the judge stopping at the right point §4.4.

Efficiency tab. 4. Seconds per 100 queries, on two L40S GPUs, 100 sampled questions per benchmark §4.6. Averages: TRIO-8B 79.9, TRIO-4B 45.6, Search-R1 90.1, R1-Searcher 88.0, DRAG 695.8, SuRe 631.0, Self-RAG 10.2, Iter-RetGen 21.1, IRCoT 80.5 tab. 4.

Reward design fig. 3. The paper compares a single outcome reward with the two-stage reward. It reports that the two-stage version improves generalisation, especially on the out-of-domain sets §4.5 fig. 3.

Other results. Under noise, where one or three of the top 5 chunks are replaced with low-ranked corpus chunks, TRIO-8B keeps the highest EM in both settings on Musique and Bamboogle app. F. On PubMedQA, TRIO-8B scores 0.766, the highest value in that table. Iter-RetGen scores 0.716 and TRIO-4B 0.666 app. G.

Paper vs code

The code was checked at commit a5a12c8 (code_commit). Each item below cites both sides.

  • Learning rate. The paper uses 1e-6 for GRPO app. B. Both RL scripts use 2e-6 query_train.sh answer_train.sh.
  • Batch sizes. The paper uses 512 training and 32 rollout app. B. The query script uses 448 and 28 query_train.sh query_train.sh. The answer script uses 256 and 32 answer_train.sh answer_train.sh.
  • Reward functions. The RL scripts call a remote reward server query_train.sh answer_train.sh. The reward code is not in this repository, so eq. 1 to eq. 7 cannot be checked against code. Alpha and l are not in the repository either.
  • Judge SFT. The script trains Llama-3.1-8B-Instruct at 1e-5 for one epoch sft_train.sh sft_train.sh. It defines LR=2e-6 but never uses it sft_train.sh. The training file is not in the repository sft_train.sh.
  • Stopping rule. The paper describes the judge deciding when to stop §3.5 app. B. In the code, the judge's verdict is ignored in the first round, so at least one generated query is always retrieved multi_agents.py. With the default of three rounds, the pool holds between two and four queries multi_agents.py multi_agents.py.
  • Stop string. The stop test is a case-sensitive substring match on "Sufficient Information" multi_agents.py. Any other output, including a lowercase version, keeps the loop going.
  • Duplicate queries. The paper's prompt asks for a query that differs from the existing ones app. B. The code does not check this. A repeated query overwrites the same key and still counts as a round multi_agents.py.
  • Checkpoints. The paper says TRIO-4B uses Qwen3-4B-Instruct §4.1. The code hard-codes the query, judge and answer checkpoints under /scratch paths multi_agents.py multi_agents.py multi_agents.py. The --generator_model flag only names the output directory init_config.py. No Qwen training script is shipped.
  • Judge checkpoint name. The judge path is named llama-3-8b-combined3 multi_agents.py. The SFT script uses Llama-3.1-8B-Instruct sft_train.sh. The README does not explain the difference.
  • Output caps. The paper does not state these. Query and judge outputs are capped at 32 tokens multi_agents.py multi_agents.py. Query RL generations are capped at 48 query_train.sh. Answer RL generations are capped at 16 answer_train.sh. The inference answer cap is 1024 base_config_TRIO.yaml.
  • Decoding. Inference is greedy, with do_sample set to False base_config_TRIO.yaml. The paper does not state a decoding setting for inference.
  • Evaluation split. The paper says "dev or test" §4.1. The code only loads the dev split main.py.
  • Dead code and flags. The helpers _extract_new_query and maintain_query_pool are never called multi_agents.py multi_agents.py. --max_answer_debate_rounds is parsed but unused init_config.py.
  • README. It refers to my_train.sh and config/base_config.yaml, neither of which exists README.md. The data-generation pipeline is delegated to the DRAG repository README.md. The paper's labelling procedure therefore has no code here.

The prompt wording in the code closely follows app. B. The field names use underscores where the paper uses spaces multi_agents.py.

Pitfalls

  • The AVG column does not follow from the other columns. Read as an unweighted mean, TRIO-8B's five values give 0.4043, not the printed 0.4294. Search-R1's give 0.3855, not 0.4034 tab. 1. The paper does not say how the average is weighted. The 6.45% and 72.45% claims depend on it §4.2.
  • The hard/normal split is undefined. Algorithm 1 uses Q1 (hard) and Q2 (normal) alg. 1. The text keeps medium and hard questions for answer training §3.2. It does not say how a question is assigned to either set.
  • Reward details are missing. Alpha, l, the normalisation used in exact match, and the counting unit for word and length terms are not given. PR's intersection could be a set or a multiset, and this is not stated eq. 2.
  • Judge labels rest on unstated choices. The correctness test, the sampling temperature and the number of samples used by TTRL §3.4 are not fully specified in the paper. The retained training data is about 80% insufficient app. B, which may push the judge towards "Insufficient".
  • The judge is ignored in the first round. At least two queries are retrieved even when the first retrieval already suffices multi_agents.py. The paper does not break this cost out §4.6.
  • Retrieval cost grows with rounds. Each round retrieves 5 chunks per query base_config_TRIO.yaml. Four retrieval calls means 20 chunks per question. Each round runs the query model and the judge. The answer model is loaded only after the first two are freed multi_agents.py.
  • Shared generation config may leak a 32-token cap. The query and judge configs are shallow copies that share one generation_params dict, and the code sets max_tokens to 32 in it multi_agents.py multi_agents.py. The answer config is also a shallow copy multi_agents.py, so it may inherit the 32 cap. This is unverified: FlashRAG is not in the repository and was not run here.
  • Answer RL used very short generations. The answer agent was trained with a 16-token generation cap answer_train.sh. Its behaviour on longer answers is not evaluated, and the paper's limitations section says long answers were not studied § Limitations.
  • Single runs, no variance. The paper does not report repeated runs or error bars for Table 1, Table 2 or Table 3 tab. 1 tab. 2 tab. 3. Differences of about one point should be treated with caution.
  • Baseline numbers are the authors' reruns. Baselines were reproduced in FlashRAG §4.1. Their numbers may differ from the originals.
  • Scope. Evaluation covers short-answer QA over Wikipedia and PubMedQA. Live web search and long-form generation are not tested § Limitations. The implementation depends on checkpoints under /scratch paths that are not provided.

Lineage

  • Builds on: DRAG, the retrieval-debate method used for training data and the multi-agent baseline §3.2 §4.1. R1-Searcher, for the difficulty split and outcome-reward RL §3.2. Search-R1, a reinforcement-learning retrieval baseline §4.1. GRPO, for the policy updates §3.2. TTRL, for test-time labelling §3.4.
  • Compared against: base RAG with Qwen3-4B, Qwen3-30B and Llama3.1-8B; SuRe, Self-RAG, IRCoT, Iter-RetGen, Search-R1, R1-Searcher and DRAG §4.1 tab. 1. Adaptive RAG methods such as RQ-RAG and MA-RAG are discussed in app. B.
  • Superseded by: not stated.

Source and license

Adapted from “TRIO: Task Refinement with Individual Optimization for Retrieval-Augmented Generation” by Yingtao Ren, Xiao Luo, Yiwei Fu, Jinzhao Zhou, Yu-Cheng Chang, Chin-Teng Lin (arXiv:2611.02810), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/yingtaoren/TRIO at commit a5a12c8decbc (Apache-2.0). Any quoted code is shown under that license.