wikillm

Paper · 2025-01-24 · cs.LG, cs.AI, cs.CL

Humanity's Last Exam

Long Phan, Alice Gatti, Ziwen Han et al.

Humanity's Last Exam: 2,500 expert-written, closed-ended questions, about 14% with images, built so that frontier LLMs fail and web search does not give the answer. Eight models score 2.7% to 13.4% accuracy with 73% to 89% RMS calibration error (author-reported). Official code: MIT, o3-mini judge.

LLM evaluationCalibrationDataset curationbenchmarkevaluationmultimodalcalibrationhallucination

arXivPDFCodeMarkdown10 min read

AI-compiled by claude-haiku-5-5 on 2026-10-18, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Language models now score above 90% on older exam-style tests such as MMLU, so those tests no longer show how much better the newest models are. This paper builds a harder test, called Humanity's Last Exam (HLE), from questions written by subject experts. Each question has one known, checkable answer, and each was first tried against several top AI models. Questions the models already solved were thrown out.

The test is meant to be the last closed-ended academic exam of its kind: a question either has a short exact answer or one correct choice among several, so a program can grade it. The authors' headline finding is that current models get only a small share right, and they often sound very sure while being wrong.

The questions are public, and a private set is kept to check whether models were tuned to the public one.

TL;DR

  • HLE is a 2,500-question, multi-modal benchmark of exact-match and multiple-choice questions written by subject experts §1 §3.1.
  • Questions were filtered so that frontier models failed them before expert review app. B.1 §3.2.
  • Eight evaluated models score 2.7% to 13.4% accuracy, with RMS calibration error of 73% to 89% (author-reported) tab. 1.
  • The reference code uses an o3-mini judge and a 100-item-bin RMS calibration metric; several details differ from the paper (see Paper vs code).
  • Use it to measure closed-ended expert knowledge, not general capability. Do not train on it, and do not republish its questions.

Problem

Benchmarks such as MMLU are close to saturated: LLMs now achieve over 90% accuracy on them §1. The authors want a closed-ended academic benchmark that still measures progress at the frontier of expert knowledge §1 §2. Their assumptions: answers must be short and checkable by a program, and questions must not be answerable by a quick web search or by memorisation §3.1.

The paper also argues that models should report calibrated confidence, since high accuracy alone does not show whether a model knows when it is wrong §4.2.

Method

The contribution has two parts: a dataset construction pipeline and an evaluation protocol. The paper specifies both in prose; the public code implements the evaluation and not the dataset pipeline README.md.

  1. Question format. Two kinds: exact-match questions (the model writes an exact string) and multiple-choice questions with five or more options. About 14% of questions need both text and an image; 24% are multiple-choice §3.1. Each submission carries the question, the answer specification, a rationale, a subject, and the contributor's name and affiliation §3.1. Open-ended or subjective questions and content about weapons of mass destruction are excluded §3.1.

  2. Automatic difficulty gate. Before submission, each question is tried on several frontier multi-modal models (GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, o1), plus two text-only models for text-only questions app. B.1. Exact-match questions must defeat all models. Multiple-choice questions must defeat all but one model, to allow for lucky guesses app. B.1. Over 70,000 attempts yielded about 13,000 questions that went to expert review §3.2.

  3. Expert review. Reviewers hold a graduate degree. The first round gives each question 1 to 3 reviews and aims to improve it. The second round has organisers and expert reviewers approve the good questions §3.2 app. B.1.

  4. Post-release checks. A community bug bounty collected corrections. An audit team fully solved a sample of questions. Questions that a model with web search answered but a model without search did not were checked by hand and removed if easy to find online app. B.2. Two audit rounds of 200 questions each gave an estimated expert disagreement rate of 15.4% on the public set app. B.3.

  5. Evaluation prompt. Every model gets a fixed system prompt that asks for three lines: Explanation, Answer, and Confidence from 0% to 100%. Models without a system role (o1) get it as a user message app. C.1 run_model_predictions.py run_model_predictions.py.

  6. Judging. o3-mini (o3-mini-2025-01-31) with structured output extracts the final answer and decides whether it matches the reference §4.1 app. C.1. The judge prompt is in app. C.1.

  7. Metrics. Accuracy over all questions, and RMS calibration error between stated confidence and observed correctness §4.2.

Evaluation pseudocode, from the authors' reference code:

Pseudocode
# Ask the model with the fixed format prompt and the question  [app. C.1]
# [code: hle_eval/run_model_predictions.py#L31-L50]
def attempt(question):
    # Text, plus an image URL when the question has one  [§3.1]
    messages = build_messages(question.text, question.image)
    # max_tokens is not stated in the paper; the repo advises 8192 or more
    # for reasoning models  [README.md#L32]
    reply = model.complete(messages, max_tokens=MAX_TOKENS)
    return reply.text
Pseudocode
# Judge the reply against the reference answer  [app. C.1]
# [code: hle_eval/run_judge_results.py#L43-L64]
def judge(question, correct_answer, response):
    # Fill the judge prompt with question, reference answer and reply
    prompt = JUDGE_PROMPT.format(question, correct_answer, response)
    # Structured output: extracted answer, reasoning, correct (yes/no),
    # confidence 0 to 100  [app. C.1]
    verdict = judge_model.parse(prompt, schema=Verdict, max_tokens=4096)
    return verdict
Pseudocode
# Accuracy over all questions; a missing judged reply counts as wrong
# [code: hle_eval/run_judge_results.py#L148]
def accuracy(verdicts, n_questions):
    n_correct = count(v for v in verdicts if v.correct == "yes")
    return 100 * n_correct / n_questions
Pseudocode
# RMS calibration error over stated confidences  [§4.2]
# [code: hle_eval/run_judge_results.py#L97-L127]
def rms_calibration_error(confidence, correct, bin_size=100):
    # Sort items by stated confidence, then cut them into bins  [code: #L99-L103]
    order = argsort(confidence)
    bins = split(order, bin_size)
    error = 0
    # The reference loop covers bins[:-1], so the last bin is skipped
    # [code: hle_eval/run_judge_results.py#L107]
    for each bin in bins[:-1]:
        # Gap between mean stated confidence and mean accuracy in the bin
        gap = abs(mean(confidence[bin]) - mean(correct[bin]))
        # Weight by the bin's share of all n items, then square  [code: #L113-L116]
        error = error + (len(bin) / n) * gap ** 2
    # Square root gives the RMS value  [code: #L124-L125]
    return sqrt(error)

Hyperparameters

NameValueWhereNotes
Question count2,500§1 §3Public release; a private held-out set is kept §1
Question typesexact-match and multiple-choice (5+ options)§3.1MCQ is 24% of questions
Image questionsabout 14%§3.1Multi-modal questions
Difficulty gate, exact matchdefeat all modelsapp. B.1Models: GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, o1; plus o1-mini and o1-preview for text-only
Difficulty gate, MCQdefeat all but one modelapp. B.1Allows for lucky guesses; see Pitfalls for a conflict with §3.2
Reviews per question, round 11 to 3§3.2
Audit sample size200 per round, 2 roundsapp. B.3Estimated disagreement 15.4% (public set)
Model temperature0.0 "when configurable"app. C.5Reference code does not send it; see Paper vs code
Model max output tokensnot stated§4 app. CRepo advises at least 8192 for reasoning models [README.md#L32]
Confidence elicitationverbalised, 0% to 100%app. C.1Written in the answer format, not from logits
Judge modelo3-mini-2025-01-31§4.1 app. C.1Structured output; max completion tokens 4096 in code run_judge_results.py
Calibration bin sizenot stated in paper§4.2Reference code uses 100 items per bin run_judge_results.py
Accuracy intervalnot stated in paper§4.2Reference code reports a Wald 95% half-width run_judge_results.py
Dataset split (HF)testcais/hle, split test run_model_predictions.py

Results

Author-reported. Setup: 2,500-question HLE, zero-shot with the system prompt above, o3-mini as judge, accuracy and RMS calibration error on the full set unless noted §4.2 app. C.1. The paper says the low scores are partly by design, because the filter removed questions models solved, and that residual non-zero accuracy comes from noise and guessing §4.2.

Main results tab. 1. Columns are as in the paper. Rows marked with a footnote in the paper are text-only subsets, not the full set.

ModelAccuracy (%)RMS calibration error (%)Notes
GPT-4o2.789
Grok 23.087
Claude 3.5 Sonnet4.184
Gemini 1.5 Pro4.688
Gemini 2.0 Flash Thinking6.682gemini-2.0-flash-thinking-exp-01-21; see app. C.5 footnote
o18.083
DeepSeek-R18.573Text-only subset
o3-mini (high)13.480Text-only subset

Text-only results app. C.2. Same models on text-only questions: accuracy 2.3 to 13.4%, calibration error 80 to 89%. Note o1 is 7.8% / 84 here against 8.0% / 83 in the main table.

ModelAccuracy (%)RMS calibration error (%)
GPT-4o2.388
Grok 23.289
Claude 3.5 Sonnet4.383
Gemini 1.5 Pro4.687
Gemini 2.0 Flash Thinking6.682
o17.884
DeepSeek-R18.573
o3-mini (high)13.480

Accuracy by subject, full dataset app. C.3. Six models only; DeepSeek-R1 and o3-mini are absent from this block.

ModelMathBio/MedPhysicsCS/AIHumanitiesChemistryEngineeringOther
GPT-4o2.36.41.70.83.23.61.82.6
Grok 23.04.63.93.31.42.43.61.7
Claude 3.5 Sonnet4.04.63.92.55.94.27.22.2
Gemini 1.5 Pro5.25.43.03.74.16.13.63.4
Gemini 2.0 Flash Thinking8.08.24.84.56.45.56.33.0
o17.410.47.08.28.79.76.37.3

Accuracy values are percentages; the paper gives no per-cell interval.

Token use. Reasoning models generate far more completion tokens than non-reasoning models for a modest accuracy gain. The authors argue future models should also be compute-optimal §4.2 fig. 5 app. C.4.

Benchmark difficulty comparison. fig. 1 plots accuracy on HLE against saturated older benchmarks. The sources for those numbers are in app. C.6.

Paper vs code

Code is at centerforaisafety/hle, commit 22ed3074b1e7b134bcbc09028d0ba320839b0655, MIT license LICENSE. The repo contains the two evaluation scripts only. The dataset-construction pipeline and the private set are not in the repo.

  • Judge model and prompt match. The paper's judge prompt and o3-mini default match the code app. C.1 run_judge_results.py run_judge_results.py.
  • System prompt matches. The format prompt is the same text app. C.1 run_model_predictions.py.
  • Temperature differs. The paper says models use temperature 0.0 "when configurable" app. C.5. The code defines a temperature option that defaults to 0.0 run_model_predictions.py, but the request line that would send it is commented out run_model_predictions.py, and the option is never read. The provider default therefore applies in the reference code.
  • Calibration bin size is not stated in the paper. The paper cites the PixMix implementation §4.2. The code's function is marked as taken from the outlier-exposure repository and bins 100 items run_judge_results.py run_judge_results.py.
  • Calibration loop skips the last bin. The bins are built to cover all items run_judge_results.py, but the summation runs over range(len(bins) - 1) run_judge_results.py. The weights still divide by all n items run_judge_results.py. For n = 2,500 that drops the 100 most confident predictions from the sum. This is read from the code and was not run.
  • Sample output does not match the paper. The README sample reports n = 2700, accuracy 3.07% and calibration error 92.3 [README.md#L51-L54]. The paper reports 2,500 questions §1. The sample probably comes from a different dataset version; the repo does not say.

No other discrepancies found in the scoring scripts.

Pitfalls

  • Missing answers count as wrong. Accuracy divides by the total question count, so failed API calls lower the score. The code prints a warning but does not change the denominator run_judge_results.py.
  • Temperature and token limits decide the number. Temperature is not sent by the reference code run_model_predictions.py. The paper gives no max output tokens. The repo says some models collapse without a cap of at least 8192 tokens on reasoning models [README.md#L32]. The Gemini 2.0 Flash Thinking row used a newer model sampled at 0.7, not the temperature-0 model from the first version app. C.5.
  • Two different difficulty rules. §3.2 says multiple-choice questions must do worse than random guessing on average. Appendix B.1 says they must defeat all but one model. Both are stated; they are not the same test.
  • Two different judges. Appendix B.1 says the pre-submission check used a GPT-4o judge. §4.1 and app. C.1 say evaluation used o3-mini. The pre-submission check code is not in the repo.
  • Non-zero floor. Multiple-choice guessing and model noise give non-zero accuracy near the floor. The authors say small changes near zero are not strong evidence of progress §4.2 app. B.1.
  • Mixed model inputs. DeepSeek-R1 and o3-mini are evaluated only on the text-only subset tab. 1. Compare them to other models using app. C.2.
  • Label noise. The estimated expert disagreement is 15.4% on the public set and about 18% on a biology, chemistry and health subset app. B.3. The paper does not derive a score ceiling from these rates. Single-reviewer flagging would raise the health-subset rate to 25% app. B.3.
  • Judge reliability is not reported. The judge prompt allows "a small margin of error" without defining it app. C.1 run_judge_results.py. The paper gives no agreement check between the judge and human graders.
  • Calibration numbers depend on low accuracy. When accuracy is this low, a model that states high confidence has a large gap by construction. Read the calibration error alongside accuracy, not on its own. This is an observation from the definition, not a claim from the paper.
  • Which set is scored? The paper says 2,500 questions are released and a private set is held out §1, but Table 1 does not name the set it used.
  • Contamination. Benchmark data must not appear in training corpora. The dataset carries a canary string for filtering [README.md#L29].
  • Dataset licence is unclear. The Hugging Face dataset page lists an MIT licence, but also asks users not to share, re-upload or distribute the data and requires accepting conditions for access § dataset page, not in the paper. This card includes no questions. Check the terms before using the questions in any product.
  • Successor planned. A dynamic fork, HLE-Rolling, will receive updates after release app. B.3. Numbers from the static set and a rolling version will differ.

Lineage

  • Builds on: MMLU, which the authors say is saturated §1 (arXiv:2009.03300). The paper also cites GPQA (arXiv:2311.12022) and MATH (arXiv:2103.03874) among related expert and mathematical benchmarks §2.
  • Calibration measure: the elicitation follows Wei et al. (arXiv:2411.04368), and the RMS implementation is from PixMix (arXiv:2112.05135) §4.2.
  • Compared against: the benchmarks in fig. 1, with sources in app. C.6.
  • Superseded by: not known. HLE-Rolling is announced as a successor fork app. B.3.

Related

ConceptLLM evaluationHow to measure what a large language model can do, and what changes the reported number: prompt, format, decoding, model snapshot, metric, judge and ground truth. Variants cover fixed-prompt benchmarks, setting and prompt sweeps, complexity curves, minimal-edit probes, LLM and human judges, simulated participants, reading-time fit, bias probes, safety rates, surveys and hallucination checks.ConceptCalibrationWhether a model's stated uncertainty or confidence matches how often it is right. Measured with ECE, RMS calibration error or NLL, or set as a conformal threshold for a success rate. Sources are author-reported and use different binning and settings, so their numbers are not directly comparable. SNGP, MC dropout, ensembles and GPR differ in over- and underconfidence.ConceptDataset curationHow training and evaluation data is sourced, filtered, deduplicated, labelled and documented. Covers web text, image-text pairs, speech, vision, 3D objects, parallel text, robot demonstrations, instruction and benchmark data, the filters each paper used and their reported effects, and the licence and documentation traps that come with reusing them.Paper · 2026-10-13Calibrated Disambiguation for Partial Multi-label LearningPML-CD trains a multi-label image classifier on over-annotated labels by weighting each candidate positive with a calibrator. The calibrator maps a 20-bin histogram of confidences on candidate positives to per-bin reliability, and an extra prototype-cosine loss is added. Author-reported mAP beats nine baselines in 15 of 16 settings on VOC, COCO, NUS-WIDE and CUB at 20-80% noise.Paper · 2026-09-28Word Similarity Datasets for Indian Languages: Annotation and Baseline SystemsBuilds manually annotated word-similarity test sets for six Indian languages (Urdu, Telugu, Marathi, Punjabi, Tamil, Gujarati) by translating and re-annotating RG-65 and WordSim-353, then scores Skip-gram, CBOW and FastText embeddings with Spearman's rho on three of them. FastText with an OOV fallback reaches 45.5 (Urdu), 38.7 (Marathi) and 46.0 (Telugu). Author-reported.Paper · 2026-09-22A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language ModelEstimates the full carbon footprint of building Noor, a 1.3B to 13B Arabic LLM family: data storage and transfer, R&D, pretraining, staff flights and projected inference. Electricity is 59.1 MWh and emissions 36.5 t CO2e, 65% from training. Location of compute, not model size alone, drives most of it. Author-reported.

Source and license

Adapted from “Humanity's Last Exam” by Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, Tung Nguyen, Jaeho Lee, Daron Anderson, Mikhail Doroshenko, Alun Cennyth Stokes, Mobeen Mahmood, Oleksandr Pokutnyi, Oleg Iskra, Jessica P. Wang, John-Clark Levin, Mstyslav Kazakov, Fiona Feng, Steven Y. Feng, Haoran Zhao, Michael Yu, Varun Gangal, Chelsea Zou, Zihan Wang, Serguei Popov, Robert Gerbicz, Geoff Galgon, Johannes Schmitt, Will Yeadon, Yongki Lee, Scott Sauers, Alvaro Sanchez, Fabian Giska, Marc Roth, Søren Riis, Saiteja Utpala, Noah Burns, Gashaw M. Goshu, Mohinder Maheshbhai Naiya, Chidozie Agu, Zachary Giboney, Antrell Cheatom, Francesco Fournier-Facio, Sarah-Jane Crowson, Lennart Finke, Zerui Cheng, Jennifer Zampese, Ryan G. Hoerr, Mark Nandor, Hyunwoo Park, Tim Gehrunger, Jiaqi Cai, Ben McCarty, Alexis C Garretson, Edwin Taylor, Damien Sileo, Qiuyu Ren, Usman Qazi, Lianghui Li, Jungbae Nam, John B. Wydallis, Pavel Arkhipov, Jack Wei Lun Shi, Aras Bacho, Chris G. Willcocks, Hangrui Cao, Sumeet Motwani, Emily de Oliveira Santos, Johannes Veith, Edward Vendrow, Doru Cojoc, Kengo Zenitani, Joshua Robinson, Longke Tang, Yuqi Li, Joshua Vendrow, Natanael Wildner Fraga, Vladyslav Kuchkin, Andrey Pupasov Maksimov, Pierre Marion, Denis Efremov, Jayson Lynch, Kaiqu Liang, Aleksandar Mikov, Andrew Gritsevskiy, Julien Guillod, Gözdenur Demir, Dakotah Martinez, Ben Pageler, Kevin Zhou, Saeed Soori, Ori Press, Henry Tang, Paolo Rissone, Sean R. Green, Lina Brüssel, Moon Twayana, Aymeric Dieuleveut, Joseph Marvin Imperial, Ameya Prabhu, Jinzhou Yang, Nick Crispino, Arun Rao, Dimitri Zvonkine, Gabriel Loiseau, Mikhail Kalinin, Marco Lukas, Ciprian Manolescu, Nate Stambaugh, Subrata Mishra, Tad Hogg, Carlo Bosio, Brian P Coppola, Julian Salazar, Jaehyeok Jin, Rafael Sayous, Stefan Ivanov, Philippe Schwaller, Shaipranesh Senthilkuma, Andres M Bran, Andres Algaba, Kelsey Van den Houte, Lynn Van Der Sypt, Brecht Verbeken, David Noever, Alexei Kopylov, Benjamin Myklebust, Bikun Li, Lisa Schut, Evgenii Zheltonozhskii, Qiaochu Yuan, Derek Lim, Richard Stanley, Tong Yang, John Maar, Julian Wykowski, Martí Oller, Anmol Sahu, Cesare Giulio Ardito, Yuzheng Hu, Ariel Ghislain Kemogne Kamdoum, Alvin Jin, Tobias Garcia Vilchis, Yuexuan Zu, Martin Lackner, James Koppel, Gongbo Sun, Daniil S. Antonenko, Steffi Chern, Bingchen Zhao, Pierrot Arsene, Joseph M Cavanagh, Daofeng Li, Jiawei Shen, Donato Crisostomi, Wenjin Zhang, Ali Dehghan, Sergey Ivanov, David Perrella, Nurdin Kaparov, Allen Zang, Ilia Sucholutsky, Arina Kharlamova, Daniil Orel, Vladislav Poritski, Shalev Ben-David, Zachary Berger, Parker Whitfill, Michael Foster, Daniel Munro, Linh Ho, Shankar Sivarajan, Dan Bar Hava, Aleksey Kuchkin, David Holmes, Alexandra Rodriguez-Romero, Frank Sommerhage, Anji Zhang, Richard Moat, Keith Schneider, Zakayo Kazibwe, Don Clarke, Dae Hyun Kim, Felipe Meneguitti Dias, Sara Fish, Veit Elser, Tobias Kreiman, Victor Efren Guadarrama Vilchis, Immo Klose, Ujjwala Anantheswaran, Adam Zweiger, Kaivalya Rawal, Jeffery Li, Jeremy Nguyen, Nicolas Daans, Haline Heidinger, Maksim Radionov, Václav Rozhoň, Vincent Ginis, Christian Stump, Niv Cohen, Rafał Poświata, Josef Tkadlec, Alan Goldfarb, Chenguang Wang, Piotr Padlewski, Stanislaw Barzowski, Kyle Montgomery, Ryan Stendall, Jamie Tucker-Foltz, Jack Stade, T. Ryan Rogers, Tom Goertzen, Declan Grabb, Abhishek Shukla, Alan Givré, John Arnold Ambay, Archan Sen, Muhammad Fayez Aziz, Mark H Inlow, Hao He, Ling Zhang, Younesse Kaddar, Ivar Ängquist, Yanxu Chen, Harrison K Wang, Kalyan Ramakrishnan, Elliott Thornley, Antonio Terpin, Hailey Schoelkopf, Eric Zheng, Avishy Carmi, Ethan D. L. Brown, Kelin Zhu, Max Bartolo, Richard Wheeler, Martin Stehberger, Peter Bradshaw, JP Heimonen, Kaustubh Sridhar, Ido Akov, Jennifer Sandlin, Yury Makarychev, Joanna Tam, Hieu Hoang, David M. Cunningham, Vladimir Goryachev, Demosthenes Patramanis, Michael Krause, Andrew Redenti, David Aldous, Jesyin Lai, Shannon Coleman, Jiangnan Xu, Sangwon Lee, Ilias Magoulas, Sandy Zhao, Ning Tang, Michael K. Cohen, Orr Paradise, Jan Hendrik Kirchner, Maksym Ovchynnikov, Jason O. Matos, Adithya Shenoy, Michael Wang, Yuzhou Nie, Anna Sztyber-Betley, Paolo Faraboschi, Robin Riblet, Jonathan Crozier, Shiv Halasyamani, Shreyas Verma, Prashant Joshi, Eli Meril, Ziqiao Ma, Jérémy Andréoletti, Raghav Singhal, Jacob Platnick, Volodymyr Nevirkovets, Luke Basler, Alexander Ivanov, Seri Khoury, Nils Gustafsson, Marco Piccardo, Hamid Mostaghimi, Qijia Chen, Virendra Singh, Tran Quoc Khánh, Paul Rosu, Hannah Szlyk, Zachary Brown, Himanshu Narayan, Aline Menezes, Jonathan Roberts, William Alley, Kunyang Sun, Arkil Patel, Max Lamparth, Anka Reuel, Linwei Xin, Hanmeng Xu, Jacob Loader, Freddie Martin, Zixuan Wang, Andrea Achilleos, Thomas Preu, Tomek Korbak, Ida Bosio, Fereshteh Kazemi, Ziye Chen, Biró Bálint, Eve J. Y. Lo, Jiaqi Wang, Maria Inês S. Nunes, Jeremiah Milbauer, M Saiful Bari, Zihao Wang, Behzad Ansarinejad, Yewen Sun, Stephane Durand, Hossam Elgnainy, Guillaume Douville, Daniel Tordera, George Balabanian, Hew Wolff, Lynna Kvistad, Hsiaoyun Milliron, Ahmad Sakor, Murat Eron, Andrew Favre D. O., Shailesh Shah, Xiaoxiang Zhou, Firuz Kamalov, Sherwin Abdoli, Tim Santens, Shaul Barkan, Allison Tee, Robin Zhang, Alessandro Tomasiello, G. Bruno De Luca, Shi-Zhuo Looi, Vinh-Kha Le, Noam Kolt, Jiayi Pan, Emma Rodman, Jacob Drori, Carl J Fossum, Niklas Muennighoff, Milind Jagota, Ronak Pradeep, Honglu Fan, Jonathan Eicher, Michael Chen, Kushal Thaman, William Merrill, Moritz Firsching, Carter Harris, Stefan Ciobâcă, Jason Gross, Rohan Pandey, Ilya Gusev, Adam Jones, Shashank Agnihotri, Pavel Zhelnov, Mohammadreza Mofayezi, Alexander Piperski, David K. Zhang, Kostiantyn Dobarskyi, Roman Leventov, Ignat Soroko, Joshua Duersch, Vage Taamazyan, Andrew Ho, Wenjie Ma, William Held, Ruicheng Xian, Armel Randy Zebaze, Mohanad Mohamed, Julian Noah Leser, Michelle X Yuan, Laila Yacar, Johannes Lengler, Katarzyna Olszewska, Claudio Di Fratta, Edson Oliveira, Joseph W. Jackson, Andy Zou, Muthu Chidambaram, Timothy Manik, Hector Haffenden, Dashiell Stander, Ali Dasouqi, Alexander Shen, Bita Golshani, David Stap, Egor Kretov, Mikalai Uzhou, Alina Borisovna Zhidkovskaya, Nick Winter, Miguel Orbegozo Rodriguez, Robert Lauff, Dustin Wehr, Colin Tang, Zaki Hossain, Shaun Phillips, Fortuna Samuele, Fredrik Ekström, Angela Hammon, Oam Patel, Faraz Farhidi, George Medley, Forough Mohammadzadeh, Madellene Peñaflor, Haile Kassahun, Alena Friedrich, Rayner Hernandez Perez, Daniel Pyda, Taom Sakal, Omkar Dhamane, Ali Khajegili Mirabadi, Eric Hallman, Kenchi Okutsu, Mike Battaglia, Mohammad Maghsoudimehrabani, Alon Amit, Dave Hulbert, Roberto Pereira, Simon Weber, Handoko, Anton Peristyy, Stephen Malina, Mustafa Mehkary, Rami Aly, Frank Reidegeld, Anna-Katharina Dick, Cary Friday, Mukhwinder Singh, Hassan Shapourian, Wanyoung Kim, Mariana Costa, Hubeyb Gurdogan, Harsh Kumar, Chiara Ceconello, Chao Zhuang, Haon Park, Micah Carroll, Andrew R. Tawfeek, Stefan Steinerberger, Daattavya Aggarwal, Michael Kirchhof, Linjie Dai, Evan Kim, Johan Ferret, Jainam Shah, Yuzhou Wang, Minghao Yan, Krzysztof Burdzy, Lixin Zhang, Antonio Franca, Diana T. Pham, Kang Yong Loh, Joshua Robinson, Abram Jackson, Paolo Giordano, Philipp Petersen, Adrian Cosma, Jesus Colino, Colin White, Jacob Votava, Vladimir Vinnikov, Ethan Delaney, Petr Spelda, Vit Stritecky, Syed M. Shahid, Jean-Christophe Mourrat, Lavr Vetoshkin, Koen Sponselee, Renas Bacho, Zheng-Xin Yong, Florencia de la Rosa, Nathan Cho, Xiuyu Li, Guillaume Malod, Orion Weller, Guglielmo Albani, Leon Lang, Julien Laurendeau, Dmitry Kazakov, Fatimah Adesanya, Julien Portier, Lawrence Hollom, Victor Souza, Yuchen Anna Zhou, Julien Degorre, Yiğit Yalın, Gbenga Daniel Obikoya, Rai, Filippo Bigi, M. C. Boscá, Oleg Shumar, Kaniuar Bacho, Gabriel Recchia, Mara Popescu, Nikita Shulga, Ngefor Mildred Tanwie, Thomas C. H. Lux, Ben Rank, Colin Ni, Matthew Brooks, Alesia Yakimchyk, Huanxu, Liu, Stefano Cavalleri, Olle Häggström, Emil Verkama, Joshua Newbould, Hans Gundlach, Leonor Brito-Santana, Brian Amaro, Vivek Vajipey, Rynaa Grover, Ting Wang, Yosi Kratish, Wen-Ding Li, Sivakanth Gopi, Andrea Caciolai, Christian Schroeder de Witt, Pablo Hernández-Cámara, Emanuele Rodolà, Jules Robins, Dominic Williamson, Vincent Cheng, Brad Raynor, Hao Qi, Ben Segev, Jingxuan Fan, Sarah Martinson, Erik Y. Wang, Kaylie Hausknecht, Michael P. Brenner, Mao Mao, Christoph Demian, Peyman Kassani, Xinyu Zhang, David Avagian, Eshawn Jessica Scipio, Alon Ragoler, Justin Tan, Blake Sims, Rebeka Plecnik, Aaron Kirtland, Omer Faruk Bodur, D. P. Shinde, Yan Carlos Leyva Labrador, Zahra Adoul, Mohamed Zekry, Ali Karakoc, Tania C. B. Santos, Samir Shamseldeen, Loukmane Karim, Anna Liakhovitskaia, Nate Resman, Nicholas Farina, Juan Carlos Gonzalez, Gabe Maayan, Earth Anderson, Rodrigo De Oliveira Pena, Elizabeth Kelley, Hodjat Mariji, Rasoul Pouriamanesh, Wentao Wu, Ross Finocchio, Ismail Alarab, Joshua Cole, Danyelle Ferreira, Bryan Johnson, Mohammad Safdari, Liangti Dai, Siriphan Arthornthurasuk, Isaac C. McAlister, Alejandro José Moyano, Alexey Pronin, Jing Fan, Angel Ramirez-Trinidad, Yana Malysheva, Daphiny Pottmaier, Omid Taheri, Stanley Stepanic, Samuel Perry, Luke Askew, Raúl Adrián Huerta Rodríguez, Ali M. R. Minissi, Ricardo Lorena, Krishnamurthy Iyer, Arshad Anil Fasiludeen, Ronald Clark, Josh Ducey, Matheus Piza, Maja Somrak, Eric Vergo, Juehang Qin, Benjámin Borbás, Eric Chu, Jack Lindsey, Antoine Jallon, I. M. J. McInnis, Evan Chen, Avi Semler, Luk Gloor, Tej Shah, Marc Carauleanu, Pascal Lauer, Tran Đuc Huy, Hossein Shahrtash, Emilien Duc, Lukas Lewark, Assaf Brown, Samuel Albanie, Brian Weber, Warren S. Vaz, Pierre Clavier, Yiyang Fan, Gabriel Poesia Reis e Silva, Long, Lian, Marcus Abramovitch, Xi Jiang, Sandra Mendoza, Murat Islam, Juan Gonzalez, Vasilios Mavroudis, Justin Xu, Pawan Kumar, Laxman Prasad Goswami, Daniel Bugas, Nasser Heydari, Ferenc Jeanplong, Thorben Jansen, Antonella Pinto, Archimedes Apronti, Abdallah Galal, Ng Ze-An, Ankit Singh, Tong Jiang, Joan of Arc Xavier, Kanu Priya Agarwal, Mohammed Berkani, Gang Zhang, Zhehang Du, Benedito Alves de Oliveira Junior, Dmitry Malishev, Nicolas Remy, Taylor D. Hartman, Tim Tarver, Stephen Mensah, Gautier Abou Loume, Wiktor Morak, Farzad Habibi, Sarah Hoback, Will Cai, Javier Gimenez, Roselynn Grace Montecillo, Jakub Łucki, Russell Campbell, Asankhaya Sharma, Khalida Meer, Shreen Gul, Daniel Espinosa Gonzalez, Xavier Alapont, Alex Hoover, Gunjan Chhablani, Freddie Vargus, Arunim Agarwal, Yibo Jiang, Deepakkumar Patil, David Outevsky, Kevin Joseph Scaria, Rajat Maheshwari, Abdelkader Dendane, Priti Shukla, Ashley Cartwright, Sergei Bogdanov, Niels Mündler, Sören Möller, Luca Arnaboldi, Kunvar Thaman, Muhammad Rehan Siddiqi, Prajvi Saxena, Himanshu Gupta, Tony Fruhauff, Glen Sherman, Mátyás Vincze, Siranut Usawasutsakorn, Dylan Ler, Anil Radhakrishnan, Innocent Enyekwe, Sk Md Salauddin, Jiang Muzhen, Aleksandr Maksapetyan, Vivien Rossbach, Chris Harjadi, Mohsen Bahaloohoreh, Claire Sparrow, Jasdeep Sidhu, Sam Ali, Song Bian, John Lai, Eric Singer, Justine Leon Uro, Greg Bateman, Mohamed Sayed, Ahmed Menshawy, Darling Duclosel, Dario Bezzi, Yashaswini Jain, Ashley Aaron, Murat Tiryakioglu, Sheeshram Siddh, Keith Krenek, Imad Ali Shah, Jun Jin, Scott Creighton, Denis Peskoff, Zienab EL-Wasif, Ragavendran P, Michael Richmond, Joseph McGowan, Tejal Patwardhan, Hao-Yu Sun, Ting Sun, Nikola Zubić, Samuele Sala, Stephen Ebert, Jean Kaddour, Manuel Schottdorf, Dianzhuo Wang, Gerol Petruzella, Alex Meiburg, Tilen Medved, Ali ElSheikh, S Ashwin Hebbar, Lorenzo Vaquero, Xianjun Yang, Jason Poulos, Vilém Zouhar, Sergey Bogdanik, Mingfang Zhang, Jorge Sanz-Ros, David Anugraha, Yinwei Dai, Anh N. Nhu, Xue Wang, Ali Anil Demircali, Zhibai Jia, Yuyin Zhou, Juncheng Wu, Mike He, Nitin Chandok, Aarush Sinha, Gaoxiang Luo, Long Le, Mickaël Noyé, Michał Perełkiewicz, Ioannis Pantidis, Tianbo Qi, Soham Sachin Purohit, Letitia Parcalabescu, Thai-Hoa Nguyen, Genta Indra Winata, Edoardo M. Ponti, Hanchen Li, Kaustubh Dhole, Jongee Park, Dario Abbondanza, Yuanli Wang, Anupam Nayak, Diogo M. Caetano, Antonio A. W. L. Wong, Maria del Rio-Chanona, Dániel Kondor, Pieter Francois, Ed Chalstrey, Jakob Zsambok, Dan Hoyer, Jenny Reddish, Jakob Hauser, Francisco-Javier Rodrigo-Ginés, Suchandra Datta, Maxwell Shepherd, Thom Kamphuis, Qizheng Zhang, Hyunjun Kim, Ruiji Sun, Jianzhu Yao, Franck Dernoncourt, Satyapriya Krishna, Sina Rismanchian, Bonan Pu, Francesco Pinto, Yingheng Wang, Kumar Shridhar, Kalon J. Overholt, Glib Briia, Hieu Nguyen, David, Soler Bartomeu, Tony CY Pang, Adam Wecker, Yifan Xiong, Fanfei Li, Lukas S. Huber, Joshua Jaeger, Romano De Maddalena, Xing Han Lù, Yuhui Zhang, Claas Beger, Patrick Tser Jern Kon, Sean Li, Vivek Sanker, Ming Yin, Yihao Liang, Xinlu Zhang, Ankit Agrawal, Li S. Yifei, Zechen Zhang, Mu Cai, Yasin Sonmez, Costin Cozianu, Changhao Li, Alex Slen, Shoubin Yu, Hyun Kyu Park, Gabriele Sarti, Marcin Briański, Alessandro Stolfo, Truong An Nguyen, Mike Zhang, Yotam Perlitz, Jose Hernandez-Orallo, Runjia Li, Amin Shabani, Felix Juefei-Xu, Shikhar Dhingra, Orr Zohar, My Chiffon Nguyen, Alexander Pondaven, Abdurrahim Yilmaz, Xuandong Zhao, Chuanyang Jin, Muyan Jiang, Stefan Todoran, Xinyao Han, Jules Kreuer, Brian Rabern, Anna Plassart, Martino Maggetti, Luther Yap, Robert Geirhos, Jonathon Kean, Dingsu Wang, Sina Mollaei, Chenkai Sun, Yifan Yin, Shiqi Wang, Rui Li, Yaowen Chang, Anjiang Wei, Alice Bizeul, Xiaohan Wang, Alexandre Oliveira Arrais, Kushin Mukherjee, Jorge Chamorro-Padial, Jiachen Liu, Xingyu Qu, Junyi Guan, Adam Bouyamourn, Shuyu Wu, Martyna Plomecka, Junda Chen, Mengze Tang, Jiaqi Deng, Shreyas Subramanian, Haocheng Xi, Haoxuan Chen, Weizhi Zhang, Yinuo Ren, Haoqin Tu, Sejong Kim, Yushun Chen, Sara Vera Marjanović, Junwoo Ha, Grzegorz Luczyna, Jeff J. Ma, Zewen Shen, Dawn Song, Cedegao E. Zhang, Zhun Wang, Gaël Gendron, Yunze Xiao, Leo Smucker, Erica Weng, Kwok Hao Lee, Zhe Ye, Stefano Ermon, Ignacio D. Lopez-Miguel, Theo Knights, Anthony Gitter, Namkyu Park, Boyi Wei, Hongzheng Chen, Kunal Pai, Ahmed Elkhanany, Han Lin, Philipp D. Siedler, Jichao Fang, Ritwik Mishra, Károly Zsolnai-Fehér, Xilin Jiang, Shadab Khan, Jun Yuan, Rishab Kumar Jain, Xi Lin, Mike Peterson, Zhe Wang, Aditya Malusare, Maosen Tang, Isha Gupta, Ivan Fosin, Timothy Kang, Barbara Dworakowska, Kazuki Matsumoto, Guangyao Zheng, Gerben Sewuster, Jorge Pretel Villanueva, Ivan Rannev, Igor Chernyavsky, Jiale Chen, Deepayan Banik, Ben Racz, Wenchao Dong, Jianxin Wang, Laila Bashmal, Duarte V. Gonçalves, Wei Hu, Kaushik Bar, Ondrej Bohdal, Atharv Singh Patlan, Shehzaad Dhuliawala, Caroline Geirhos, Julien Wist, Yuval Kansal, Bingsen Chen, Kutay Tire, Atak Talay Yücel, Brandon Christof, Veerupaksh Singla, Zijian Song, Sanxing Chen, Jiaxin Ge, Kaustubh Ponkshe, Isaac Park, Tianneng Shi, Martin Q. Ma, Joshua Mak, Sherwin Lai, Antoine Moulin, Zhuo Cheng, Zhanda Zhu, Ziyi Zhang, Vaidehi Patil, Ketan Jha, Qiutong Men, Jiaxuan Wu, Tianchi Zhang, Bruno Hebling Vieira, Alham Fikri Aji, Jae-Won Chung, Mohammed Mahfoud, Ha Thi Hoang, Marc Sperzel, Wei Hao, Kristof Meding, Sihan Xu, Vassilis Kostakos, Davide Manini, Yueying Liu, Christopher Toukmaji, Jay Paek, Eunmi Yu, Arif Engin Demircali, Zhiyi Sun, Ivan Dewerpe, Hongsen Qin, Roman Pflugfelder, James Bailey, Johnathan Morris, Ville Heilala, Sybille Rosset, Zishun Yu, Peter E. Chen, Woongyeong Yeo, Eeshaan Jain, Ryan Yang, Sreekar Chigurupati, Julia Chernyavsky, Sai Prajwal Reddy, Subhashini Venugopalan, Hunar Batra, Core Francisco Park, Hieu Tran, Guilherme Maximiano, Genghan Zhang, Yizhuo Liang, Hu Shiyu, Rongwu Xu, Rui Pan, Siddharth Suresh, Ziqi Liu, Samaksh Gulati, Songyang Zhang, Peter Turchin, Christopher W. Bartlett, Christopher R. Scotese, Phuong M. Cao, Ben Wu, Jacek Karwowski, Davide Scaramuzza, Aakaash Nattanmai, Gordon McKellips, Anish Cheraku, Asim Suhail, Ethan Luo, Marvin Deng, Jason Luo, Ashley Zhang, Kavin Jindel, Jay Paek, Kasper Halevy, Allen Baranov, Michael Liu, Advaith Avadhanam, David Zhang, Vincent Cheng, Brad Ma, Evan Fu, Liam Do, Joshua Lass, Hubert Yang, Surya Sunkari, Vishruth Bharath, Violet Ai, James Leung, Rishit Agrawal, Alan Zhou, Kevin Chen, Tejas Kalpathi, Ziqi Xu, Gavin Wang, Tyler Xiao, Erik Maung, Sam Lee, Ryan Yang, Roy Yue, Ben Zhao, Julia Yoon, Sunny Sun, Aryan Singh, Ethan Luo, Clark Peng, Tyler Osbey, Taozhi Wang, Daryl Echeazu, Hubert Yang, Timothy Wu, Spandan Patel, Vidhi Kulkarni, Vijaykaarti Sundarapandiyan, Ashley Zhang, Andrew Le, Zafir Nasim, Srikar Yalam, Ritesh Kasamsetty, Soham Samal, Hubert Yang, David Sun, Nihar Shah, Abhijeet Saha, Alex Zhang, Leon Nguyen, Laasya Nagumalli, Kaixin Wang, Alan Zhou, Aidan Wu, Jason Luo, Anwith Telluri, Steven Dillmann, Zhengxiang Wang, Junyu Luo, Hugo Lunn, Artem Gazizov, Haitz Sáez de Ocáriz Borde, Ivan Trus, Morgan Hervault, Zheyu Zhang, Bo Chen, Yuchen Wu, Christopher J. Cordier, Gün Kaynar, Cansin Ayvaz, Polina Avdiunina, Johannes Brust, Xingjian Diao, K. D. Meaney, Yifan Gu, Chenyu Wang, Chenzhuo Dong, William Wright, Simon Brave, Owen Root, Jiayuan Liu, Chow Chun Lok, Tianqin Li, Shiyi Du, Dailan He, Lufeiya Liu, Sina Jamalzadegan, Anil Ramakrishna, Xuanqing Xu, Xin Qing, Xin Luo, Wenkai Li, Shi Bo, Filipp Gusev, Maximos Skandalis, Desheng Ma, Chunhui Zhang, Haoran Qiu, Allen G Hart, Rickard Brüel Gabrielsson, Ido Akov, Artem Lukoianov, Summer Yue, Alexandr Wang, Dan Hendrycks (arXiv:2501.14249), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/centerforaisafety/hle at commit 22ed3074b1e7 (MIT). Any quoted code is shown under that license.