wikillm

Paper · 2026-11-12 · cs.IR, cs.AI, cs.CL

RMIT-ADM+S at TREC RAG 2026: Research Agents, Retriever Ablations, and Open-Weight Replication

TREC RAG 2026 system report on ClimbMix. Seven research agents write cited reports up to 1,024 words, varying the writer (GPT-5.6 variants or open-weight GLM-5), retrieval (BM25, dense, hybrid RRF), a requirements brief and one review pass. Best Batch-1 vital coverage ties the track maximum (0.892); open-weight GLM-5 leads Batch-1 human preference at a fifth of the cost. Author-reported.

Retrieval-augmented generationDense retrievalLLM agentLLM-as-a-judgeragagentic-retrievalevaluationopen-weight-models

arXivPDFCodeMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

This is a team's report on its entries to a shared test, the TREC 2026 retrieval-augmented generation track. The task is to answer a research question with a short report of up to about a thousand words, where every claim must point to a source passage from a large fixed text collection.

The team built seven different answer-writing agents. Each one searches the collection, keeps the passages it decides to use, and writes a cited report. The agents differ in which language model writes the report, how they search, and whether they plan first or check the draft at the end. Some use proprietary models; owa-glm5 and facetrag use open-weight models only.

The main lesson is that the choice of writing model mattered more than most of the other design choices the team tried. Different ways of judging the answers also disagreed with each other, and automatic coverage scores did not match what human assessors preferred. The report is useful as a reminder that a system ranking depends on how it is measured.

TL;DR

  • Seven research-agent runs for TREC RAG 2026 on the ClimbMix corpus, each writing a cited report of up to 1,024 words per topic §1 §2.1.
  • Retrieval options were BM25, a fine-tuned dense encoder and reciprocal rank fusion of both. The writer model had the largest effect in pre-submission sweeps §4.2.
  • Six of seven runs were above the track median on all three official coverage collections §5.1.
  • The open-weight run owa-glm5 was preferred by human assessors in Batch 1 but scored lower on automatic coverage. Results are author-reported.

Problem

The authors ask whether open-weight agents can match proprietary ones at lower cost, which design choices matter most for quality and token cost, and whether multi-stage orchestration beats a single-loop agent §1. A fixed evaluation can rank systems differently depending on judge or assessment protocol, so the report also studies that sensitivity §4.1.

Method

The report does not propose a new algorithm. It describes seven full systems and their components.

End-to-end flow. Each run writes a report from the topic. Each chunk citation is converted to the ID of its parent document before output §2.1. The control flow of the three runs they highlight is in fig. 1.

Shared staged-evidence loop. Five systems share a research loop with an evidence ledger. Retrieved chunks are visible for one turn. The agent calls a commit tool to keep chosen chunks. Uncommitted chunks become markers and expire from the staged batch. Adjacent chunks ±1 are added automatically for the top results when that option is on §2.1 §3.1 fig. 1.

Requirements brief. Before research, an isolated model call lists up to eight requirements the answer must meet, drawn from the request and inferred ones §2.1. The brief is added to the writer's prompt as guidance §2.1.

Blind obligation scout. A separate call sees only the topic and adds up to eight more requirements. In owa-glm5 those extras are not checked in the post-draft review §2.1 fig. 1.

Draft review. A code scan flags uncited sentences. One reviewer call checks the draft against the brief. When the review raises issues, the writer receives them and can run new searches before revising. That revised text skips a second review fig. 1 §2.1.

Facet systems. facets plans facets in one conversation and checks coverage in code. facetrag runs a planner model and an analyser model on 3 to 6 facets concurrently, with a curator that selects up to five items per facet §2.1.

Retrieval.

  • BM25 via a self-hosted Lucene/Anserini index with Pyserini defaults k1=0.9, b=0.4 §3.
  • Dense retrieval with a fine-tuned 256-dimensional encoder, served by DiskANN with maximum inner product search §3.
  • Hybrid retrieval fuses the two lists with unweighted reciprocal rank fusion, κ=60. Each retriever returns at least 50 candidates before fusion §3.
Pseudocode
# Hybrid retrieval: fuse dense and BM25 rankings  [§3]
# Input: query, k (passages wanted)
# Output: top-k chunks by fused score

def hybrid_search(query, k):
    # Each retriever returns at least 50 candidates  [§3]
    # [code: src/utils/search.py#L82]
    depth = max(k, 50)
    dense_hits = dense_index.search(embed(query), depth)
    sparse_hits = bm25_index.search(query, depth)
    # Unweighted RRF with kappa = 60  [§3]
    # [code: src/utils/search.py#L26-L60]
    scores = {}
    for ranking in [dense_hits, sparse_hits]:
        # A chunk counts once per ranking, at its best rank
        # [code: src/utils/search.py#L41-L52]
        for rank, chunk in enumerate(unique(ranking), start=1):
            scores[chunk] = scores.get(chunk, 0.0) + 1.0 / (60 + rank)
    return top_k(scores, k)

Fine-tuned retriever. The encoder was fine-tuned on evidence decisions from agent runs over the official test topics. Chunks committed in a run are positives. Retrieved but unselected chunks are negatives. Training uses InfoNCE with in-batch negatives over the first 256 dimensions, before normalisation §3.1. The training code is not in the repository. Its recipe is documented only in a model-card summary embedding-training.md.

Writer loop limits. A program checks the context size, how many citations a sentence carries, and the length of the report §2.1 fig. 1.

Pseudocode
# Brief-hybrid run, one topic  [§2.1] [fig. 1]
# Input: topic (narrative)
# Output: cited report, at most 1,024 words

def answer(topic):
    # Isolated analyst call, at most 8 requirements  [§2.1]
    brief = analyst_model(topic)
    # Writer researches and drafts in one conversation  [§2.1] [fig. 1]
    draft = writer_loop(topic, brief, max_rounds=100)
    # Code scan for uncited sentences, then one reviewer call  [fig. 1]
    issues = uncited_sentences(draft) + reviewer_model(draft, brief)
    if issues:
        # The writer gets the issues and may run new searches  [fig. 1]
        draft = writer_loop_resume(draft, issues)
    # Map chunk citations to parent document IDs  [§2.1]
    return export(draft)

Hyperparameters

NameValueWhereNotes
Chunk length200–500 tokens, hard cap 700; 1.3 tokens per word§3Chunks are paragraph-aware; ID <docid>_p<page>, 1-based
Corpus size6,543 shards, about 563 million documents; 921,892,634 BM25 chunks§3
BM25k1=0.9, b=0.4§3 CHUNKED-REINDEX-DESIGN.mdLive server settings; src/ does not set them
Dense encoder256-dimensional; base model is the nano-size Jina v5 text encoder§3Model card linked from §3
DiskANNMIPS, graph degree R=64, build list L=100, 32-byte PQ§3 run_ft_index.shDefaults in the build script are PQ 0 and L 100, so the 32 comes from the run script
RRFκ=60, unweighted, depth ≥ 50 per retriever§3 search.py
Adjacent chunks±1 on the top 5 results, duplicates removed§3 §2.1 adjacent_pages.py
Retained chunksup to 10 per batch via commitfig. 1 agent.py
Context budget500,000 tokens§2.1 run.py
Round cap100 (brief and owa runs); 40 (sol-research-first)fig. 1 agent.py run.pySafety caps, not a target
Brief sizeat most 8 requirements, at most 4 implicit§2.1 brief.pyImplicit cap is not in the paper
Scout sizeat most 8 additions§2.1 scout.py
Citationsat most 3 per sentence§2.1 fig. 1Enforced by code (not checked in the repo)
Report lengthup to 1,024 words, no minimum§2.1 fig. 1
Review passesone reviewer call, at most one revision§2.1 fig. 1
Fine-tune optimiserlearning rate 1e-4, batch 64, 4 epochs, bf16§3.1Optimiser class, schedule and warm-up: not stated
Fine-tune lossInfoNCE, temperature τ=0.05, in-batch negatives§3.1
Fine-tune LoRArank 16, alpha 64, dropout 0.1§3.1
Fine-tune split95 topics (1,587 sub-queries, 3,849 positive pairs); 24 held out (410 sub-queries)§3.1Exact topic IDs not stated
Pre-submission judgeGPT-5.6-Terra, 0–2 per criterion over six dimensions§4.1Pairwise judge varies by run
Dev topics30 organiser topics, two disjoint sets of 15§4.1
Official judgeGPT-6.1 Sol, medium reasoning§5.1
Arena seed13tab. 3 captionInternal, not official

Results

Author-reported. Setup: organiser-run NIST evaluation of the TREC 2026 RAG track. Batch 1 uses 17 human-written rubrics, Batch 2 uses 25 topics with manual nuggets and 25 with automatically built nuggets. Coverage uses strict scoring, so partial support earns no credit §5.1.

Official coverage, vital_strict / all_strict tab. 4:

SystemB1 rubric (n=17)B2 manual nuggets (n=25)B2 auto nuggets (n=25)
brief-hybrid0.892 / 0.9090.691 / 0.7230.644 / 0.565
sol-research-first0.892 / 0.8870.684 / 0.7020.649 / 0.560
brief-base0.824 / 0.8620.691 / 0.7210.648 / 0.555
sol-default0.790 / 0.8610.679 / 0.7050.630 / 0.563
facets0.831 / 0.8410.627 / 0.6420.390 / 0.333
owa-glm50.853 / 0.8210.550 / 0.6050.288 / 0.243
facetrag0.600 / 0.6220.460 / 0.4810.139 / 0.110
Track max0.892 / 0.9110.729 / 0.7800.666 / 0.584
Track median0.716 / 0.7330.488 / 0.5560.155 / 0.128

Six of seven submissions scored above the track median on every collection §5.1. Tukey's test separated none of the four top proprietary configurations §5.1.

Human side-by-side, Bradley–Terry ratings fig. 2 §5.2:

BatchHigher ratingLower ratingTopic-level test
Batch 1 (n=17)owa-glm5 1212.8sol-research-first 1062.713 of 17 topics favour owa-glm5, sign-flip p=0.006
Batch 2 (n=25)sol-research-first 1124.7owa-glm5 1049.0inconclusive, p=0.110

Estimated generation cost and effort per topic, averaged over 119 topics tab. 2:

SystemSearchesWordsUSD per topic
sol-default23.9882.51.566
sol-research-first31.4936.62.823
brief-base26.4830.22.356
brief-hybrid22.6834.32.156
facets11.7613.00.040
facetrag33.7388.90.106
owa-glm512.7754.00.524

Pre-submission results, which use different judges and sometimes different topics, so they are not comparable with the official numbers §4:

  • Fine-tuned encoder on 24 held-out topics: nDCG@10 0.751 → 0.793, ranking 4,071 chunks the agents had kept §3.1. The labels come from agent decisions, not human judgments.
  • Hybrid-only versus semantic plus keyword for brief-base: 2.333 vs 2.267 on a 15-topic dev set, with the hybrid gain confirmed on a second 15-topic set (2.333 vs 2.133) §4.2.
  • sol-research-first versus sol-default on 30 dev topics: 0.704 vs 0.651 on the rubric §4.2.
  • owa-glm5 on two 15-topic dev sets: 1.467 and 1.333 out of 3, below the proprietary configurations at 1.867 to 2.333 §4.3.

Batch 1 answer characteristics tab. 5: owa-glm5 reports averaged 805.3 words with 16.6 unique references and Flesch–Kincaid grade 15.7. sol-research-first reports averaged 951.2 words, 28.9 references and grade 22.2. The authors say readability is a plausible but unproven contributor to the preference gap §6.

Paper vs code

Repository: rmit-ir/trec-rag-26, commit bcb50ee, which the paper links in a footnote §2.1. It has no LICENSE file, so no code is quoted here.

Checked and consistent:

Differences and gaps:

  • The retriever training code is not in the repository. The repository documents the recipe from a model card and says the exact split, seed, optimiser, schedule, warm-up and checkpoint are missing embedding-training.md. The paper's recipe is the same as the model-card summary, so it cannot be independently checked.
  • The paper says BM25 uses Pyserini defaults. The source does not set k1 or b. They appear only in a design note for the live server CHUNKED-REINDEX-DESIGN.md.
  • The paper describes a self-hosted index. The client points to a hosted server search_sparse.py, and the index is not in the repository.
  • The brief's implicit cap (4 of 8) is in code and not in the paper brief.py.
  • Not checked: the chunk length rules in §3, the 3-citation limit, the facetrag curator, and HyDE prompts. Those would need the chunking and facet code under tasks/ and src/systems/facet_rag/.
  • The RAGDoll judge is a git submodule from castorini/RAGDoll (third-party, not the authors' code). It is not used for the card.
  • No official code for the retriever training, the official judgments or the human assessment protocol. These are organiser or model-card artefacts.

Pitfalls

  • This is a participant system report, not a method paper. A reader cannot separate the effect of orchestration from the effect of models, prompts and retrieval. The authors say this themselves §7.
  • Official numbers depend on the official judge (GPT-6.1 Sol) and on strict scoring, so partial support earns nothing. In pre-submission tests, changing only the pairwise judge moved one system's win rate against the organiser baseline from 17.5% (Sol judge) to 45.0% (Luna judge) §4.1.
  • The fine-tuned retriever was trained on agent decisions over the official 2026 test topics. The 24-topic validation is inside the same set. It does not establish end-to-end generation quality §3.1.
  • Assessors preferred owa-glm5 in Batch 1 with lower coverage, and the preference reversed in Batch 2. The reversal is not statistically significant §5.2. Do not read the preference as a general ranking.
  • The round caps of 100 and 40 are safety limits. Cost depends on how many rounds run, which varies a lot by system (search calls from 11.7 to 33.7 per topic) tab. 2.
  • The review pass allows one revision, and the harness accepts whatever comes back as the full report. In one failed case (Batch-2 topic 65) a request to add a citation to one sentence returned only a 21-word sentence, which was accepted in place of a 922-word draft. That run lost all eight comparisons §5.3.
  • Position bias is large in the pre-submission arena: one pair's win rate moved from 37.7% to 76.7% when order was swapped §4.3.
  • Readability differences are not associated with preference margins at the topic level §6.
  • sol-research-first processed 81% more tokens than sol-default for little difference in Batch-2 coverage §2.3 §7.

Lineage

  • Builds on: the team's own earlier competition systems (LiveRAG and MMU-RAG), which the paper cites but which are not compiled here §1.
  • Compared against: the organiser's Sol agentic BM25 baseline §4.3 and the sol-default configuration §2.1.
  • Superseded by: not known from content in this wiki.

Source and license

Adapted from “RMIT-ADM+S at TREC RAG 2026: Research Agents, Retriever Ablations, and Open-Weight Replication” by Oleg Zendel, Kun Ran, Chenglong Ma, Nguyen Khoi Vo, Shuai Wang (arXiv:2611.16425), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/rmit-ir/trec-rag-26 at commit bcb50ee6ba42 (none (no LICENSE file in repository)). Any quoted code is shown under that license.