MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark

In plain words

Language models are often graded on MMLU, a large set of multiple-choice exam questions. Top models now score so high on it that the test no longer tells good models apart, and a lucky guess or a slightly different prompt can move a score noticeably.

MMLU-Pro is a harder version of that test. Each question gets ten answer choices instead of four, so guessing is much less likely to work. It also adds questions that need several reasoning steps, not just recall, and drops the questions that were too easy or badly worded. Models do worse on it, and their scores move less when the prompt wording changes. It also shows a clear effect of asking the model to reason step by step before answering, which the original test did not show.

TL;DR

MMLU-Pro is a multiple-choice benchmark with 12,032 questions over 14 subjects, built by filtering MMLU and adding questions from STEM websites, TheoremQA and SciBench §3.1 §3.2. Each question has up to ten options §3.2. The authors report that it is harder than MMLU, more discriminative between models and less sensitive to prompt style, and that chain-of-thought prompting helps on it but not on MMLU §1 §6.2 §6.3. Use it as a reasoning-heavy benchmark, and reproduce the evaluation with the official code, because several answer-parsing details differ between the paper and the code (see Paper vs code).

Problem

MMLU §1 has become saturated: the authors state that frontier models settle between 86% and 87% and that the benchmark no longer separates them §1. They attribute this to three causes: four-option questions allow shortcuts, most questions need little reasoning, and a portion of questions is unanswerable or mislabelled §1. Scores on MMLU also shift with prompt wording and scoring function §1 §2.2.

Method

The paper contributes a benchmark plus an evaluation protocol. Both are described below.

Dataset construction §3.2 fig. 2

1. Merge MMLU's 57 subject categories into 14 broader categories §3.2.
2. Remove "too easy" MMLU questions: eight models were run on MMLU (Llama-2-7B and 13B with and without chat tuning, Mistral-7B, Gemma-7B, Yi-6B and Yi-6B-Chat), and a question answered correctly by more than four of them was excluded §3.2. This removed 5,886 of 13,937 original questions app. A.1 tab. 4.
3. Add questions from the STEM Website, TheoremQA and SciBench §3.2. STEM Website problems come with solutions and TheoremQA questions with short answers; GPT-4-Turbo extracts a short answer and writes three plausible distractors for each §3.2 app. A.1, tab. 6.
4. Manually compare the extracted answers with the solutions and drop questions where the answer is incomplete or wrong §3.2.
5. Expand each question from four to ten options with GPT-4-Turbo, adding six distractors §3.2 app. A.1, tab. 6.
6. Expert review, in two phases §3.2:
   • Phase 1: experts verify each answer, remove questions unsuitable for multiple choice, and remove questions that need images, tables or missing information.
   • Phase 2: Gemini-1.5-Pro re-evaluates all options to find false negatives (correct options marked wrong); experts then check each flagged option.
7. Final size: 14 subjects and 12,032 questions §3.1. By source: MMLU 6,810, STEM Website 4,083, TheoremQA 598, SciBench 541 app. A.1, tab. 5.

Evaluation protocol

1. Prompt: a 5-shot chain-of-thought prompt. It holds an instruction, five worked examples from the same subject, and the test question §4 app. A.2. The examples come from the validation split app. A.2.
2. Generate a reasoning answer and end with "The answer is (X)" §4 app. A.2.
3. Parse the letter with two regular expressions, then fall back to a random option §4. See the pseudocode below.
4. Score as the fraction of questions whose parsed letter matches the key.

Procedure for answer extraction, as the paper describes it §4 and as the code implements it (differences marked) evaluate_from_local.py:

# Answer extraction from a model output  [§4]
# Input:  output (text generated after the question)
# Output: a letter A-J, or None

def extract_answer(output):
    # Pattern 1: "answer is (X)", parentheses optional  [§4]
    match = search(r"answer is \(?([A-J])\)?", output)
    if match:
        return match.group(1)
    # Pattern 2: "answer: X". The paper's printed pattern is ambiguous
    # (escaped brackets); the code uses this form
    # [code: evaluate_from_local.py#L122]
    match = search(r".*[aA]nswer:\s*([A-J])", output)
    if match:
        return match.group(1)
    # Code only, not in the paper: last standalone letter A-J
    # [code: evaluate_from_local.py#L129-L135]
    match = search(r"\b[A-J]\b(?!.*\b[A-J]\b)", output, DOTALL)
    if match:
        return match.group(0)
    return None

# Scoring with a random fallback  [§4] [code: evaluate_from_local.py#L152-L171]
# Input:  predictions (letter or None), keys, option counts per question
# Output: accuracy

def accuracy(predictions, keys, option_counts):
    correct = 0
    for index in range(len(keys)):
        letter = predictions[index]
        # Unparsed outputs get a uniform random option from the question's
        # own options; the code seeds this RNG with 12345
        if letter is None:
            letter = random_letter(option_counts[index])
        if letter == keys[index]:
            correct = correct + 1
    return correct / len(keys)

Other evaluation settings, covered in detail under Hyperparameters: zero-shot is used for the two Gemini-1.5 models tab. 2 caption. A CoT-versus-direct-answer comparison is made on five models tab. 3. A robustness study runs 24 different prompt styles §6.3.

Hyperparameters

Results

Author-reported. Setup: 14-subject MMLU-Pro, 5-shot chain of thought unless noted, accuracy in percent tab. 2 caption. The paper states that its leaderboard holds full results for all subjects §5. The table below shows the columns the paper prints. The "Overall" column is labelled as such; how it is aggregated is not stated.

Headline numbers (author-reported)

• GPT-4o is the strongest model at 72.6% overall §5.1 tab. 2.
• The two leading closed models separate more on MMLU-Pro: GPT-4o and GPT-4-Turbo differ by about 1 point on MMLU and about 9 on MMLU-Pro §1.
• Accuracy drops 16% to 33% compared with MMLU [abstract].
• Prompt sensitivity: on MMLU, prompt changes move scores by generally 4–5%, peaking at 10.98%; on MMLU-Pro, generally about 2%, with a maximum of 3.74% §6.3.
• CoT vs direct answer, MMLU-Pro: GPT-4o gains 19.1 points with CoT; on MMLU the gain is 1.5 tab. 3.

Table 2: Models on MMLU-Pro, CoT tab. 2. Values are accuracies in percent. Gemini-1.5-Pro and Gemini-1.5-Flash use 0 shots; all others use 5.

Table 3: CoT vs direct answering tab. 3. Accuracy in percent; the difference column is CoT minus direct answer.

Error analysis of GPT-4o (120 erroneous predictions, expert-labelled) §5.3 app. A.6: reasoning errors 39%, lack of specific knowledge 35%, calculation errors 12%, other 14% (no selection 5%, question understanding 4%, generation 2%, annotation 2%, answer extraction 1%) §5.3.

Dataset issues found in expert review tab. 1:

Paper vs code

Compared against the official repository at code_commit (TIGER-AI-Lab/MMLU-Pro, Apache-2.0). The repository is the code of the authors' lab (the paper links the lab's leaderboard §3.2 §5). It holds evaluation code only; the dataset is loaded from Hugging Face.

• Fallback regex 2. The paper requires parentheses around the letter in its second pattern §4. The code uses .*[aA]nswer:\s*([A-J]) with no parentheses required evaluate_from_local.py.
• Third fallback. The paper names only two patterns and then a random option §4. The code adds a third step that takes the last standalone letter A–J in the output evaluate_from_local.py.
• Random fallback size. The local evaluation draws uniformly over the question's own options evaluate_from_local.py. The compute_accuracy.py helper draws over the fixed letters A–J, regardless of how many options the question has compute_accuracy.py. The paper says "a random option from the answer choices" §4, which matches the first.
• "N/A" options. The code removes options whose text is exactly "N/A" before building prompts evaluate_from_local.py evaluate_from_api.py. The paper does not mention this and reports up to ten options §3.2.
• Prompt wording. The paper's example reads "The answer is (X)" [app. A.2]. The code's instruction file says "the answer is (X)" in lower case initial_prompt.txt. The regex matches case sensitively on "answer is", so the code's prompt matches its own parser.
• Demonstration choice. The paper says the five examples are "representative" §4. The code takes the first five validation items of the same subject evaluate_from_local.py.
• Shot reduction. The code lowers the number of demonstrations below five for a question until the prompt fits the context limit evaluate_from_local.py. The paper does not mention this, so some questions may get fewer than five shots.
• Token limits. The local evaluation uses 2048 new tokens evaluate_from_local.py; the API evaluation uses 4000 evaluate_from_api.py. The paper states neither.
• Repeated call. eval_cot calls save_res twice; the second call overwrites the first with the same output evaluate_from_local.py. No effect on the numbers.

No other discrepancies were found in the parts compared (CoT form, 5-shot default, greedy decoding, the first regex, and the subject-level evaluation loop).

Pitfalls

• Answer-letter parsing decides the score. The first regex is applied with re.search, so the first "answer is (X)" in the output wins, even if the model wrote it earlier in its reasoning evaluate_from_local.py. The paper does not say which match is used.
• Random fallback is part of the score. Unparsed outputs count as random guesses (about 1 in the number of options), and the result depends on the random seed. Report how many outputs fell through to the fallback.
• Option counts vary. 17% of questions have fewer than ten options §3.2. Code that assumes ten letters will mis-score those questions, and so will the compute_accuracy.py fallback described above.
• Dataset construction is LLM-assisted. GPT-4-Turbo wrote distractors and option expansions, and Gemini-1.5-Pro flagged false negatives §3.2. The paper says GPT-4-Turbo gained no advantage from option augmentation §3.2. Reviewers still found 483 incorrect answers in the STEM Website part tab. 1.
• Result comparability. Table 2 mixes 5-shot and 0-shot models, and its rows are closed/open-source groups, not a single protocol tab. 2 caption. Compare only rows with the same setting.
• Only seven of fourteen subjects are printed tab. 2. The remaining subjects are on the external leaderboard §5; their numbers are not in this card.
• Variance of the main table is not stated. The paper gives prompt-variation ranges §6.3 but no confidence intervals or run-to-run spread for Table 2.
• Compute. Closed-model evaluation used about 20M input and 5M output tokens app. A.4. A 7B local model takes about 20–30 minutes app. A.4.
• Scope. Text-only multiple choice. It does not evaluate open-ended answers or multimodal models §7.
• Licence of the data. The appendix says the dataset is MIT-licensed and names the source licences app. A.5. Check those terms before reusing the dataset; this card covers the paper only.

Lineage

• Builds on: MMLU, the 57-subject benchmark this paper filters and extends §1 §3.2. The paper cites it by author name; no arXiv id is given in the paper text.
• Uses: the Chain-of-Thought Hub prompt as a base for its CoT prompt §4.
• Compared against: more than 50 LLMs, open and closed §1; MMLU on difficulty, reasoning and robustness §6.
• Superseded by: not stated in this card's sources.