Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving

In plain words

Many tests of AI models ask multiple-choice questions or school-level maths. A model can then guess, or recognise an answer it has seen, without doing the physics. This benchmark uses open-ended problems taken from the qualifying exams that physics PhD students must pass. Each problem has several parts, and each part has a reference answer the model must produce.

A model writes out its reasoning and a list of final answers. A separate language model checks each answer against the reference and says yes or no. A problem only counts as solved if every one of its answers is right. Most models that reason before answering do well; the best reach about 90% on the problems without figures. Most open-weight models still solve less than half. Adding a figure to the problem lowers the score of the open-weight models, and when the authors examined one model's mistakes, misreading a figure was one of the most common failures.

The practical lesson is that a score on this benchmark depends on the grader and on how the questions are scored, so the numbers only compare well when the protocol matches.

TL;DR

Physics is an evaluation set of 1,034 PhD-qualifying-exam physics problems from six domains, with 2,559 separately graded reference answers §3.4. Responses are graded by GLM-5.3-Flash, one judge call per problem, and the main metric is strict accuracy: a problem counts only if all its answers are judged correct §4.2. The authors report that 90.4% strict accuracy is the best on the 602 test problems without figures, reached by a proprietary model, and that most open-weight models solve fewer than half §5.1. The judge is a model that is also in the evaluated set, so scores move by about five points with the choice of judge app. B.2; code: README.md#L124.

Problem

Multiple-choice benchmarks can be solved by exploiting answer choices, and primary- to high-school benchmarks do not test the multi-step derivations the authors want to measure §1. The paper targets open-ended, graduate-level physics: problems need domain knowledge, several derivation steps and results that are checked as final expressions or values §1, §3. The qualifying exams the problems come from have passing thresholds that the paper gives as 40% to 70% §1, footnote.

The benchmark keeps only subquestions that ask for a final result. Requests for proofs, explanations and drawings are removed §3.3. The problems come from publicly available sources; the paper states that annotators checked copyright and licensing for the selected problems §3.2. Whether the problems appear in model pretraining data is not measured §6, Limitations.

Method

The paper contributes two things: a benchmark built from exam problems with a verification pipeline, and an evaluation protocol that extends the response format and the LLM-judge protocol of Humanity's Last Exam (HLE) to problems with several answers §1, §4.1; arXiv:2501.14249.

1. Benchmark construction

1. Problem selection. Problems come from public sources. The six areas are optics, atomic physics, electromagnetism, quantum mechanics, the mechanics of classical bodies, and heat with statistical mechanics §3.1.
2. Annotation. Seven physics-trained annotators wrote each problem in LaTeX, recorded its conditions and assumptions, included any figure of the physical setup, and wrote a step-by-step solution with final answers. Each annotator passed a qualification test first. The authors reviewed every annotation, and a second annotator checked each problem §3.2.
3. Results-only scope. Subquestions asking for a final result are kept; proof, explanation and drawing requests are deleted without changing the retained wording. Of the 1,297 annotated problems, the subquestions kept number 2,730 and the ones removed number 752 §3.3.
4. Blind solving and audit. A solver first worked each problem with no access to the reference. An auditor then compared that work with the reference, and a separate reviewer checked batches of up to five problems. The main audit corrected reference answers in 245 released problems and worked solutions in 264 more §3.3; app. A.3.3.
5. Answer labelling. Every answer gets its subquestion tag and a kind: number, expression or text. Two independent passes gave identical labels for 94.5% of problems, and an adjudicator settled the other 62 §3.3; app. A.3.4.
6. Model-output screen. Where two open-weight models (Qwen3.5-9B and Qwen3.8-27B) gave the same wrong-by-judge answer, the reference was re-derived. Five references were corrected §3.3; app. A.3.5.
7. Final review. Scope, dependency, duplicate and reference checks removed 188 problems, leaving 1,109. After evaluation, a review of answers missed by all five leading models removed 75 more, giving the released 1,034 problems with 2,559 answers §3.3; app. A.3.2.

Each answer's verification used LLM agents (Claude Opus 5, GPT-5.6 Sol and GPT-6 Astra) under fixed written rules, with independent human-style review §3.3. Three of the models in the verification pipeline are also among the evaluated models (see Pitfalls).

2. Evaluation protocol

1. Response format. The system message asks for three parts: an explanation, the final answers one per line in the order requested (each with a part label where the question is divided), and a confidence from 0% to 100%. The number of expected answers is withheld from the model §4.1; app. B.1.
2. Answering. Each model answers each problem once. Figures are attached only for multimodal models §4.1; app. B.1; code: README.md#L64.
3. Judging. One judge call per problem receives the question text, the model's final response with any reasoning block stripped, and the labelled reference answers. Figures are not passed to the judge §4.2.
4. Rules. For each reference, the judge extracts the response's answer for the same part and decides equivalence under these rules §4.2; app. B.1:
   • symbolic answers must be algebraically equivalent; a wrong factor, sign, exponent or variable dependence is incorrect;
   • numeric answers pass if they lie within a 1% relative gap of the reference, or agree once rounded to the reference's significant figures; units must match after conversion; an order-of-magnitude estimate is correct if the order of magnitude matches;
   • text answers must state the same conclusion, and any contradiction is incorrect;
   • hedged answers, answers to a different quantity, and answers given under a different part are incorrect;
   • a convention counts only if the response itself states it, and then only when the two results agree exactly.
5. Confidence. The confidence is extracted separately, defaults to 100, and does not affect the accuracy metrics §4.2.

# Strip any reasoning block; keep the visible answer text  [§4.2]
def visible_text(response):
    return response.split("</think>")[-1].strip()

# Judge one problem: one call returns one verdict per reference  [§4.2]
def judge_problem(question, response, references, retries=4):
    # Empty visible text is wrong on every answer, without a judge call  [§4.2]
    text = visible_text(response)
    if text == "":
        return [False] * len(references)
    # Figures are not sent to the judge  [§4.2]
    for attempt in range(retries):
        verdicts = call_judge(question, text, references)
        # Accept only output with exactly one yes/no per reference
        # [code: physics_eval/judge.py#L15-L32]
        if verdicts is not None and len(verdicts) == len(references):
            return verdicts
    # No valid verdict after retries: scored as wrong
    # [code: physics_eval/judge.py#L56]
    return [False] * len(references)

# Strict accuracy: a problem counts only if every answer is correct  [§4.2]
def strict_accuracy(verdicts):
    # verdicts: one list of booleans per problem
    solved = [all(problem) for problem in verdicts]
    return sum(solved) / len(solved)

# Answer-level accuracy pools every verdict across problems  [§4.2]
def answer_accuracy(verdicts):
    flat = [v for problem in verdicts for v in problem]
    return sum(flat) / len(flat)

# Partial credit weights each problem equally  [§4.2]
def partial_credit(verdicts):
    fractions = [sum(p) / len(p) for p in verdicts]
    return sum(fractions) / len(fractions)

def bootstrap_interval(verdicts, reps=2000):
    # Resample problems with replacement; recompute strict accuracy  [§4.2]
    # Keep the 2.5th and 97.5th percentiles
    # [code: physics_eval/metrics.py#L30-L33]
    estimates = []
    for _ in range(reps):
        sample = resample_with_replacement(verdicts, len(verdicts))
        estimates.append(strict_accuracy(sample))
    return percentile(estimates, 2.5), percentile(estimates, 97.5)

Key formulas are the metric definitions above; the paper contains no numbered equations.

Hyperparameters

Results

Author-reported. Setup: 53 models, four proprietary (OpenAI) and 49 open-weight. The 27 text-only models are scored on the 786 problems without figures; the 26 multimodal models are scored on all 1,034 problems §4.1. The table below is the full strict-accuracy table (%), sorted by the 602 test problems without figures tab. 1. Column abbreviations: Opt. is optics, Stats. is heat and statistical physics, and the other four are the usual short forms of the field names (AMO, E&M, CM, QM). Domain columns and Test (602) are on the 602 test problems without figures; Val is the 184 validation problems without figures. Test (w/ fig.) covers all 794 test problems; "—" means the model is text-only.

Headline findings, all author-reported on the 602 test problems without figures unless stated §5.1:

• Top of the ranking. GPT-6 Astra scores 90.4%, GPT-6 Sol 90.0% and GPT-5.6 Sol 89.9%. The best open-weight models are close: Qwen3.8-27B 88.5% and DeepSeek-V4.1-Flash 88.2% §5.1; tab. 1; app. B.4.5.
• Thinking. The 17 highest-ranked models all use thinking. Across the table, the thinking group's mean is 59.6% and the non-thinking group's is 14.8% §5.1. The average of the 21 thinking models in the table is consistent with this figure tab. 1.
• Most open-weight models. Forty-one of the 49 open-weight models solve less than half the problems. The strongest non-thinking model reaches 32.4%, and the middle value across the 32 non-thinking models is 11.1% §5.1.
• Scale. Within families, accuracy rises with size: Qwen3.5 gains 60.0 points from 2B to 122B, and most of that gain (54.2 points) comes from the 2B-to-9B step app. B.4.1.
• Figures. All 22 open-weight multimodal models do worse on problems with figures, by 7.3 points on average (8.2 points after reweighting by domain). OpenAI's four models average 1.7 points higher with figures §5.1; tab. 1.
• Completion length. Among the 21 thinking models, the shortest third of completed responses averages 73.9% and the longest third 45.7%. This is an association, not a causal test app. B.4.3.
• Judge agreement. GLM-5.3-Flash returned the expected verdict on 202 of 204 synthetic target answers and on all 481 unchanged answers. A cross-check with DeepSeek-V4.1-Flash agreed on 93.5% of the answers of 200 problems; 27 of the 31 disagreements were adjudicated in favour of GLM-5.3-Flash by four human reviewers app. B.2.

Error analysis (one model)

The authors analysed the wrong answers of Qwen3.5-9B on the 794 test problems §5.2. Of its 208 failed problems, 41 hit the 65,536-token limit and were excluded from the per-answer analysis. The remaining failures gave 173 answers with identifiable model errors (from 118 problems), plus 49 other non-truncated failures.

Source: §5.2. The paper does not say how the categories were assigned.

Domains

Averaged over all 53 models on the figure-free test problems, Quantum Mechanics has the lowest mean strict accuracy (25.7%), followed by Electromagnetism (30.9%) and Classical Mechanics (31.0%). Quantum Mechanics problems carry the most answers (2.91 on average), and strict scoring requires every one of them app. B.4.4.

Paper vs code

Compared at code commit 3b36e33 of the official repository (yale-nlp/physics-verified, MIT), which the paper links [§1; code: README.md]. The repository is the authors' own lab organisation; its LICENSE names Yale-NLP.

• Judge prompt. Wording matches the appendix prompt [app. B.1; code: physics_eval/prompts.py#L14-L40]. The numeric tolerance matches (1%) prompts.py.
• System prompt placement. The paper says the response-format text is placed at the start of the user message for chat templates that reject a system turn app. B.1. The released code always sends it as a system message predict.py. The fallback is not implemented.
• Figures. Figures are attached for multimodal models and not given to the judge, as the paper says [app. B.1; code: physics_eval/predict.py#L10; physics_eval/judge.py#L45].
• Reasoning stripped. The judge sees only the text after the reasoning block §4.2; code: physics_eval/judge.py#L10-L12.
• Empty response. Matches the paper: no visible text is scored wrong without a judge call judge.py.
• API errors. A prediction that failed with an API error is scored wrong without a judge call and without retry, and the paper does not say this judge.py. Rerun predict before judging.
• Judge failure. The paper does not say what happens if no valid verdict is returned. The code retries (default 4) and then scores all answers of that problem as wrong [code: physics_eval/cli.py#L30; physics_eval/judge.py#L46-L56; physics_eval/metrics.py#L41].
• Sampling and seed. The paper defers sampling settings to the released outputs; the code passes them as optional flags and sets no default temperature or seed cli.py.
• Calibration. The code computes an HLE-style RMS calibration error metrics.py. The paper text does not describe it.
• Table versions. The README includes a column "paper_table3_test" with scores from the original version of the paper, under a different protocol and problem set, and says they are not comparable README.md. This card uses arXiv v2 (updated 2026-10-12) and its table.
• Released data. The README describes the Hugging Face release as a cleaned, results-only version of the benchmark with the same counts (1,034 problems, 2,559 answers) README.md.

Pitfalls

• Strict accuracy punishes multi-answer problems. Problems carry 2.47 answers on average, and the quantum-mechanics problems carry 2.91. Report answer-level accuracy next to strict accuracy when comparing domains §3.4; app. B.4.4.
• The judge is a model under test and changes the scores. GLM-5.3-Flash is also evaluated. Judged by DeepSeek-V4.1-Flash, its strict accuracy on all 794 test problems is 80.5%, against 85.3% when judged by itself [app. B.2, Table 3 in the appendix]. The repository reports a similar gap on a shared sample of 187 other-model predictions and attributes it to general judge strictness README.md. Scores from different judges are not comparable README.md.
• The judge sees no figures. For the 248 problems with figures, the judge checks the answer text only. A figure misreading shows up only as a wrong answer, not as a diagnosed cause §4.2.
• Removal used model performance. The final review removed 75 problems after evaluation, based on answers that all five leading models missed §3.3; app. A.3.2. The model-output screen also used Qwen3.5-9B and Qwen3.8-27B outputs to flag references, and five references were corrected §3.3. The paper does not measure whether these steps favour particular models. Treat the references as partly shaped by open-weight outputs.
• Verification used models that are later evaluated. The verification agents were Claude Opus 5, GPT-5.6 Sol and GPT-6 Astra §3.3. GPT-5.6 Sol and GPT-6 Astra are in the results table.
• Single sample, no seeds. Each model answers each problem once [code: README.md#L64]. The confidence intervals resample problems only, so run-to-run variance of the models is not measured.
• Test subsets differ. The headline numbers use 602 test problems without figures. Text-only models cannot be scored on the 192 test problems with figures, so "Test w/ fig." and "Test (602)" are different populations.
• Domain estimates are small. Optics has 51 figure-free test problems and the widest interval app. B.4.4. Read domain differences as tentative.
• Truncation. Responses that reach the 65,536-token limit with no visible text are scored wrong. Responses with visible text are judged normally §4.2. The output budget itself therefore changes scores for long-reasoning models.
• Contamination is unmeasured. Problems come from public sources §3.2; §6, Limitations.
• Error analysis is narrow. It covers one model (Qwen3.5-9B) and 173 wrong answers, and the paper does not describe how categories were assigned §5.2.
• Numeric tolerance and estimates. A 1% relative tolerance is applied to numeric answers, and order-of-magnitude estimates count as correct if the order matches §4.2. Expect rounding and convention disputes at the edge of these rules.
• Answer count is hidden. The model is not told how many answers to give §4.1.

Lineage

• Builds on: Humanity's Last Exam (HLE) response format and LLM-judge protocol, extended to several answers per problem §4.1; see the HLE card for the original protocol.
• Compared against: multiple-choice and primary- to high-school benchmarks cited in §1. The paper gives no quantitative comparison with them.
• Later release: a cleaned, results-only version named PHYSICS-Verified, published by the authors README.md.