LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion

In plain words

Different open chatbots answer the same question differently, and the best one changes from question to question. LLM-Blender first asks several of them for an answer, then picks the good ones. Its ranker, PairRanker, does not score each answer on its own. It looks at the question together with two answers at a time and decides which of the two is better. Doing this for every pair gives a full ordering of the answers.

The second part, GenFuser, does not stop at picking a winner. It reads the question and the top few answers and writes a new reply that keeps what those answers got right. The paper also builds a benchmark of instructions with answers from 11 open models, so that such ensembles can be measured. The authors report that the combined system beats every single model on that benchmark. Those numbers come from the authors and are not independently checked.

TL;DR

• LLM-Blender is a two-stage ensemble: PairRanker ranks N candidate answers, and GenFuser writes one new answer from the top three §2.3.
• PairRanker is a cross-encoder trained on pairwise preferences; its training labels come from BARTScore against the reference answer §3.2 §5.1.
• On MixInstruct (11 open LLMs), the fused output reaches GPT-Rank 3.01 against 3.90 for the best single model, OpenAssistant tab. 2. Author-reported.
• Use it if you need a per-input choice among several open models, or a fused answer. Check the pitfalls: the released code differs from the paper in several training details.

Problem

Single fixed LLMs are often not the best choice across inputs. The paper shows that the best of 11 open models varies a lot from input to input; Vicuna ranks first on only 21.22% of examples §1 fig. 1. Pointwise rerankers, which give each candidate a score on its own, have trouble with candidates that are all fluent and differ only subtly §1 §3.1. Fusing candidates is proposed to go beyond the best existing candidate, since no candidate is best on every axis §1 §4.

The setting assumes a set of N candidate outputs for each input, and a reference answer only at training time §2.1.

Method

The framework has two modules §2.3 fig. 2.

1. For each input x, run N LLMs to get N candidate outputs. The paper uses N = 11 open models for MixInstruct §2.2.
2. PairRanker scores ordered pairs of candidates. It fills an N × N matrix M, where M[i][j] is the confidence that y_i is better than y_j §3.2 §3.3. The final ranking uses the MaxLogits rule. The paper also describes MaxWins and a single bubble pass, and reports MaxLogits as the best of these §3.3.
3. Keep the top K = 3 candidates §2.3 §4.
4. GenFuser, a seq2seq model based on Flan-T5-XL (3B), takes the instruction, the input and the K candidates as one sequence, separated by special tokens. It is fine-tuned to generate a better response §4.

PairRanker training. Each training example comes with candidate outputs and a reference answer. For each candidate, the quality Q(y, reference) is measured with one or more metrics (BERTScore, BARTScore, BLEURT) §2.2 §5.1. The paper uses BARTScore as the supervision signal because its rankings correlate best with GPT-Rank §5.1 tab. 3. Instead of all N(N-1)/2 pairs, the model samples 5 pairs per input §3.3.

Equations in §3.2 are unnumbered in the source, so they are cited by section.

Pairwise scoring. The ranker reads the sequence <source> x </s> <candidate1> y_i </s> <candidate2> y_j </s> in one pass. It takes the hidden states at the three special tokens as representations of x, y_i and y_j §3.3. A head combines the source with each candidate and gives one score per candidate. It returns the difference of the two scores §3.3 ranker.py.

Pairwise loss (paper form). For each metric Q, with pair-specific scores s^i and s^j, the label is (1, 0) when Q(y_i) ≥ Q(y_j), and (0, 1) otherwise. The per-metric loss is -z_i log σ(s^i) - z_j log σ(s^j). The paper sums the metrics but says it takes the average §3.2.

Ordering the candidates. To place the candidates, the paper uses MaxLogits: the score of y_i is the sum over j of (M[i][j] - M[j][i]). MaxWins counts wins. A bubble pass uses N-1 comparisons and returns one best candidate in O(N) §3.3.

Fusion input. Candidates are concatenated after the instruction and input, each marked with a separator token such as <extra_id_i> §4. The released code uses a plain-text layout with a fixed instruction at the end blender_utils.py.

# LLM-Blender at inference  [§2.3] [fig. 2]
# Input:  instruction x; list of N candidate outputs from N LLMs
# Output: fused response y_hat
def llm_blender(instruction, candidates, top_k=3):
    # Rank every candidate with PairRanker  [§2.3] [§3.3]
    ranked = rank_candidates(instruction, candidates)
    # Keep the K best candidates for fusion, K = 3 in the paper  [§2.3]
    top = ranked[:top_k]
    # Build the GenFuser input from the instruction and top candidates  [§4]
    fuser_input = build_fuser_input(instruction, top)
    # Generate the fused response with the Flan-T5-XL seq2seq model  [§4]
    return generate(fuser_input)

# Pair logit for one ordered pair of candidates  [§3.2] [§3.3]
# Input:  source text; candidates cand_i, cand_j (strings)
# Output: scalar confidence that cand_i is better than cand_j
def pair_logit(source, cand_i, cand_j):
    # Encode source and both candidates in one sequence  [§3.3]
    hidden = encoder(
        f"<source> {source} </s> <candidate1> {cand_i} </s>"
        f" <candidate2> {cand_j} </s>"
    )  # (T, d)
    # Read the hidden states at the three special tokens  [§3.3]
    source_vec = hidden_at("<source>")  # (d,)
    i_vec = hidden_at("<candidate1>")  # (d,)
    j_vec = hidden_at("<candidate2>")  # (d,)
    # Score each candidate against the source, then average over metrics  [§3.3]
    score_i = mean(head(concat(source_vec, i_vec)))
    score_j = mean(head(concat(source_vec, j_vec)))
    return score_i - score_j

# Full comparison matrix and MaxLogits ranking  [§3.3]
# Input:  source text; list of N candidates
# Output: candidate indices, best first
def rank_candidates(source, candidates):
    n_candidates = len(candidates)
    matrix = zeros(n_candidates, n_candidates)
    # Fill M[i][j] for every ordered pair, i != j  [§3.3]
    for i in range(n_candidates):
        for j in range(n_candidates):
            if i != j:
                matrix[i][j] = pair_logit(source, candidates[i], candidates[j])
    # MaxLogits: total margin of each candidate over all others  [§3.3]
    scores = sum(matrix - transpose(matrix), axis=1)
    return argsort(scores, descending=True)

# Single bubble pass, N-1 comparisons  [§3.3]
# Input:  source text; list of N candidates
# Output: one candidate judged best
def bubble_best(source, candidates):
    # Start from a random order  [§3.3]
    order = random_permutation(candidates)
    best = order[0]
    for challenger in order[1:]:
        # Compare both orders, then keep the stronger candidate  [§3.3]
        margin = pair_logit(source, best, challenger)
        margin -= pair_logit(source, challenger, best)
        if margin <= 0:
            best = challenger
    return best

# Pairwise loss for one training pair, per metric  [§3.2]
# Input:  pair scores s_i, s_j (n_metrics,); reference metric values
#         q_i, q_j (n_metrics,)
# Output: scalar loss
def pair_loss(s_i, s_j, q_i, q_j):
    # Label the better candidate per metric; a tie labels y_i as better  [§3.2]
    z_i = (q_i >= q_j)  # (n_metrics,)
    z_j = 1 - z_i  # (n_metrics,)
    # Negative log-likelihood of the labelled preferences  [§3.2]
    loss = -z_i * log(sigmoid(s_i)) - z_j * log(sigmoid(s_j))
    # Paper text says average over metrics; the formula is a sum  [§3.2]
    return mean(loss)

Key equations, as written in the paper (unnumbered in the source) §3.2:

\mathcal{L}_Q = -z_i\log\sigma(s^i_{(i,j)}) - z_j\log\sigma(s^j_{(i,j)}) §3.2

s_{ij} = s^i_{(i,j)} - s^j_{(i,j)} §3.2

s_i^{\text{MaxLogits}} = \sum_j (M_{i*} - M_{*i}) §3.3

Hyperparameters

Values in the paper are in the appendix and §5.1. Values marked "code" come from the released repository and were not stated in the paper.

Results

Author-reported. Setup: MixInstruct test split (5k examples), 11 open LLMs, reference metrics BERTScore, BARTScore and BLEURT, plus GPT-Rank (ChatGPT judges every candidate pair; a candidate's rank is its number of wins) §5.1 app. E. The paper does not state which ChatGPT version it used.

Headline: LLM-Blender scores GPT-Rank 3.01 against 3.90 for the best single model (OpenAssistant), BERTScore 79.09 against 74.68, BARTScore -3.02 against -3.45 and BLEURT -0.17 against -0.39 tab. 2 §5.2.

MixInstruct, full table tab. 2. ↑ higher is better; ↓ lower is better. "≥Vic" and "≥OA" give the share of examples where a method is at least as good as Vicuna or OpenAssistant under GPT-Rank. "Top-3" is the share of examples where the method places in the top three.

The "Oracle" rows use the metric itself to pick the best candidate, so they are upper bounds and not methods. The GPT-Rank oracle is 1.00 by definition.

Ranking correlation with GPT-Rank tab. 3. Pearson, Spearman and Spearman's footrule (lower is better) over all candidates:

Benchmark composition tab. 1. Four sources: Alpaca-GPT4 (22,862 examples), Dolly-15K (7,584), GPT4All-LAION (76,552) and ShareGPT (3,002), totalling 110K examples.

Conventional tasks (author-reported, appendix C; PairRanker is trained on each task's candidates, and GPT-3 outputs are reranked without GPT-3 training data) app. C.4:

• CNN/DM, PEGASUS candidates: Rouge-1 47.39 vs 44.56 base (+6.35%, max logits).
• CommonGen, T5-large candidates: CIDEr 15.86 vs 15.48 base (+2.45%, max logits).
• WMT18 zh-en, Opus-MT candidates: BLEU 20.47 vs 19.29 base (+6.12%, max logits).
• Transfer to GPT-3 (text-davinci-003): CNN/DM +6.64%, CommonGen +26.53%, WMT18 +10.93% with max logits (+11.65% with max wins).

CNN/DM, PEGASUS and GPT-3 rows app. C.3-C.4:

CommonGen and WMT18 zh-en app. C.4:

The CommonGen and WMT columns come from two separate tables in the appendix, and the rows are shown side by side here. The base row for each task is T5-large (CommonGen) and Opus-MT (WMT18).

Paper vs code

Code: llm_blender/ at commit 33204d2712944b6b17996f7c079e74cd963ccc7c (Apache-2.0) LICENSE. The paper's footnote links yuchenlin.xyz/LLM-Blender, and the repository is under the GitHub account yuchenlin, the same handle.

• Loss. Paper: the BCE-style formula in §3.2. Script: --loss_type "instructgpt" train_ranker.sh. That branch is the log-sigmoid of the signed score difference ranker.py. The BCE branch ranker.py is the closest to the paper, and the paper says BCE was "good enough" app. A.
• Head. Paper: five-layer MLP with tanh app. A. Code: two linear layers with tanh and dropout 0.05 ranker.py config.py.
• Backbone. Paper: DeBERTa-v3-large §5.1 app. A. Script default: microsoft/phi-2 train_ranker.sh. The configs/pairrm-phi2.yml file is empty.
• Batch size. Paper: 64 on one GPU app. A. Script: n_gpu=8 with the MixInstruct branch at 4 per device and 16 accumulation steps, so 4 × 16 × 8 = 512 sequences per step train_ranker.sh train_ranker.sh. With n_gpu=1 the same branch gives 64, the paper's batch size.
• Inference. Paper: MaxLogits is the default aggregator §3.3. The released Blender.rank uses the full matrix and MaxLogits blender.py utils.py. The training script evaluates with bubble train_ranker.py.
• MaxLogits formula. Paper: sum of margins §3.3. Code: mean of margins utils.py. The ranking is the same.
• Candidate pool. The code removes duplicate candidates before sampling ranker.py. The paper does not mention this.
• Pair order. The code shuffles the two candidates in each training pair ranker.py, as the paper describes §3.3.
• Fuser checkpoint. The paper fine-tunes Flan-T5-XL §4. The released fuser script starts from a checkpoint at a private NFS path ds_train_xl.sh, so the public Flan-T5-XL alone will not reproduce the released fuser.
• Fuser candidate length. The code truncates each candidate to the candidate limit and appends "..." blender_utils.py. The paper does not state this.
• Fuser decoding. Code: beam search with 4 beams blender.py. Paper: not stated.
• Data. The repository contains no training data. The MixInstruct downloader builds the set from public datasets get_mixinstruct.py get_mixinstruct.py. The paper says ground-truth answers are added to the candidate pool during training §3.3; I did not verify this in the code, and the training file is not in the repository.

No discrepancies were found in the core ranking formula: the pairwise difference, the full matrix, and the MaxLogits order match between paper and code §3.3 ranker.py utils.py.

Pitfalls

• Loss is ambiguous in the paper. The formula in §3.2 sums the per-metric losses, while the text says "average". The label for a tie is (1, 0) in the paper. The released default loss is instructgpt, which sets the loss to log 2 with zero gradient when the two reference scores are equal, because the signed term vanishes ranker.py. Decide which loss you train with and say so.
• Batch size. Paper: 64. The script, as released with 8 GPUs and the MixInstruct settings, gives 512. Match the global batch size before comparing learning rates.
• Full matrix cost. Full comparison is N(N-1) ranker forward passes per input (110 for N = 11). The released bubble mode runs both orders at each step, so it takes 2(N-1) passes ranker.py. The paper's "N-1 comparisons" counts a comparison as one pass §3.3. Plan the budget with the code's count.
• Bubble ties. The paper's bubble update is undefined when the margin is exactly 0. The code moves to the challenger when the margin is 0 or less ranker.py. Exact ties are rare in practice, but a consistency check may depend on the rule.
• Random pair sampling. Training draws 5 random pairs per input, anew each time. The training script's seed is 42 train_ranker.py; the paper does not state a seed.
• Order sensitivity. Swapping candidate 1 and candidate 2 changes the output. The paper reports that the ranker agrees with itself under swapping more than 90% of the time app. C.5. Keep the random-order shuffle during training.
• Labels come from BARTScore, not GPT. PairRanker learns the BARTScore ordering. The GPT-Rank reported in the tables is a separate judgment from ChatGPT §5.1 app. E. The ChatGPT version is not stated.
• The GPT-3 "11.65%" headline is max wins. With max logits the WMT18 gain is 10.93% app. C.4. Quote the matching aggregator.
• Oracle rows are upper bounds. The "Oracle" rows choose the best candidate with the target metric, so they are not achievable methods tab. 2.
• Candidate generation is not stated. The decoding settings for the 11 models are not given in the compiled paper, so the benchmark cannot be regenerated from this card.
• Fusion rests on complementary candidates. The motivation is that top candidates have complementary strengths §1 §4. The sections read do not report cases where fusion lowers quality.
• All numbers are author-reported. The paper's own table is the source; no independent reproduction was run for this card.

Lineage

• Builds on: DeBERTa-v3 (PairRanker backbone), BARTScore (supervision and evaluation metric), Flan-T5 (GenFuser backbone) §3.3 §4 §5.1.
• Compared against: MLM-Scoring, SimCLS, SummaReranker, and the InstructGPT reward model (no cards in this wiki; named only) §3.1 §5.2.
• Superseded: not stated.