Evaluation of OpenAI o1: Opportunities and Challenges of AGI

In plain words

Language models are often judged on fixed quiz-style benchmarks. This paper asks a wider question: how well does OpenAI's o1-preview model, which is built to think through a problem before it answers, do on a broad set of realistic tasks? The authors wrote 27 tasks covering reasoning, planning, creating things such as code and layouts, medical diagnosis, and reviewing or summarising text, and then ran the model on them.

The model did very well on some tasks, such as a set of competitive programming problems and high-school maths. It did less well on others, such as harder college-level proofs and some advanced statistics. For many tasks the authors judged the answers themselves, often on only a handful of examples.

Read it as a broad, informal survey rather than a controlled benchmark. The numbers are small, much of the grading was done by the authors, and the exact prompts and model settings are not fully given.

TL;DR

The authors ran o1-preview on 27 tasks in several fields. It passed 10 of 12 LeetCode problems and answered all 10 high-school maths cases correctly. On 10 radiology reports it had the best ROUGE-1 and ROUGE-L among six models, but not the best ROUGE-2. It got 7 of 12 college-level maths problems right. Samples are small, grading is by the authors, and the paper alone does not pin down the model settings, so the results are hard to reproduce.

Problem

Standard benchmarks, the authors argue, can be manipulated and do not cover enough of a model's abilities §1. The paper therefore tests the model on multi-step, real-world-style tasks in many domains, to judge its progress towards artificial general intelligence §1. The paper also treats o1 as a model that puts chain-of-thought reasoning into inference itself, rather than only when prompted §1.1.

This is an evaluation study. It proposes no training or inference method, so there is no algorithm to implement. The implementable parts are the scoring rules and the task set.

Method

The authors ran a single model, o1-preview, on 27 tasks grouped into five faculties: Reasoning (6 tasks), Planning (5), Creation and Design (6), Diagnosis (5) and Reflection (5) §1.4 fig. 1. Each section of the scope lists the data used for one task §2.1–§2.27.

Data sources differ by task. Some tasks use public datasets, such as the NLI sets MNLI, ANLI, QNLI, MedNLI and RadQNLI §2.4 tab. 1. Radiology uses a private Chinese report dataset §2.2. Coding uses weekly and biweekly LeetCode contests §2.1. Many other tasks use prompts the authors wrote, sometimes drawn from earlier work §4.10.

1. Pick a task and its data §2 §1.4.
2. Write a domain-specific prompt. Prompts were not published for most tasks. Full prompt text is given only for NLI tab. 4, medical text anonymisation tab. 6 and social media analysis tab. 7. For radiology, the phrasing was chosen by trial runs §2.2.
3. Send the prompt to o1-preview and keep its answer. The paper gives no sampling settings and no number of runs per item §4.1.
4. Grade the answer with the task's rule: an automatic judge for coding §2.1, ROUGE against human-written reports for radiology eq. 1 §2.2, and manual grading by the authors for most other tasks §4.15 §4.16.
5. Report counts, percentages or qualitative examples §4.

Coding is the only task with a fully specified protocol. Each problem allows up to three submissions, and it counts as solved if any submission passes LeetCode's judge §2.1. The pseudocode below follows that protocol. The paper does not say whether later attempts see the judge's feedback §4.2.

# LeetCode protocol: up to three submissions per problem  [§2.1] [§4.2]
# Input:  one contest problem (statement, tests), the model
# Output: True if any submission passes the judge

def score_problem(problem, model):
    # Feedback to the model between attempts is not stated  [§4.2]
    for attempt in range(3):
        code = model.write_solution(problem.statement)
        # Judge checks correctness and efficiency on predefined tests  [§2.1]
        if judge(code, problem.tests):
            return True
    return False

Radiology uses ROUGE. The paper defines only ROUGE-N, as a ratio of matched n-grams to the n-grams of the reference reports eq. 1 §2.2. Reported scores are ROUGE-1, ROUGE-2 and ROUGE-L, but ROUGE-L is not defined anywhere in the paper.

# ROUGE-N for one generated report against one reference  [eq. 1] [§2.2]
# Input:  generated report, reference report, n (1 or 2)
# Output: score between 0 and 1

def rouge_n(generated, reference, n):
    # Count the n-grams of the reference  [eq. 1]
    reference_counts = count_ngrams(reference, n)
    # Count_match: clipping and tokenisation not stated  [eq. 1]
    matched = count_matches(generated, reference_counts, n)
    # Divide by all reference n-grams  [eq. 1]
    return matched / sum(reference_counts.values())

The single equation in the scope section is the ROUGE-N definition eq. 1. Its denominator counts n-grams in the reference, so the score is recall-style. The paper does not say whether any precision or F-measure variant is used.

Hyperparameters

The paper gives no training hyperparameters. The table lists the evaluation settings it does state, and what it leaves out.

Results

Author-reported. Setup: o1-preview alone for most tasks. Only radiology has a model comparison, against gpt-4o, gpt-4-turbo, o1-mini, gpt-4o-mini and gpt-3.5-turbo tab. 3. Chip design compares against ChipNeMo, but only by example §4.11.

Coding failures: one hard problem (Weekly 413) gave a correct answer that exceeded the time limit, then wrong answers. Another hard problem (Biweekly 138) gave wrong answers in all three attempts §4.2 fig. 3 fig. 4.

Paper vs code

The paper links the repository UGA-CAID/AGI-Bench for AGI-Benchmark 1.0 §1.4. The clone at the commit above holds a short README and a spreadsheet, Test-data-v2.xlsx. The spreadsheet was not opened or used. The repository has no evaluation scripts, no prompts and no model-calling code, so the reported protocol cannot be checked against code. The paper says the benchmark "will be available" §1.4, which the repository only partly matches. It has no LICENSE file, so its content may not be reused here.

Pitfalls

• Small samples. Radiology uses 10 reports, NLI 10 cases, high-school maths 10 cases, college maths 12 problems and coding 12 problems. Each coding problem moves the pass rate by 8.3 percentage points tab. 2. Treat each percentage as a count of a few items.
• Abstract claims beyond the tables. The abstract says o1-preview is "outperforming other evaluated models" on radiology reports. Table 3 shows gpt-4-turbo with a higher ROUGE-2 (0.0656 vs 0.0448) and a close ROUGE-1 (0.2999 vs 0.3019) tab. 3. The abstract also says the model surpasses "many human experts" on coding. The text reports no human pass rates, only that generation times were "comparable or faster" than top human competitors §4.2.
• Chip design and other comparisons are by example. The o1-preview vs ChipNeMo comparison rests on hand-picked examples, with no metric §4.11.
• ROUGE is not clinical accuracy. It measures n-gram overlap with the reference. The paper defines only ROUGE-N eq. 1. Tokenisation, stemming, handling of multiple references, and ROUGE-L are not stated.
• Prompts are mostly missing. Full text appears only for NLI tab. 4, anonymisation tab. 6 and social media tab. 7. The radiology prompt was tuned by trial runs §2.2. Results could shift with a different wording, and the paper reports no sensitivity analysis.
• Model settings and version are not stated. The paper says only "o1-preview" and does not give a snapshot date, the access route, sampling settings or the number of runs §4.1. The authors also note that their results are "not comprehensive due to OpenAI's policy restrictions" §4.19. Hosted models change, so the numbers may not reproduce.
• Grading is by the authors, without a published rubric. Non-coding tasks are graded manually, with a "questionable" category in addition to correct and incorrect §4.15 tab. 5. Several tasks (EHR diagnosis, medical anonymisation) are assessed qualitatively §4.16 §4.18.
• Contamination is not analysed. Several tasks use public datasets (e.g. MNLI, LogiQA). The paper says AGI-Benchmark 1.0 "resists manipulation and memorization" §1.4 but gives no contamination check. For college proofs the authors asked for two different proofs to avoid a memorised solution §4.15.
• Stated denominators are missing for some claims. The stochastic-process "80% accuracy" does not give the number of problems §4.17. The analogy result of "10 test questions" does not say how many were asked §4.20.
• Latency and cost. o1-preview averaged 15.051 s per radiology report, against 1.629 s for gpt-4o-mini tab. 3. The paper attributes this to extra verification §4.3.
• Section and table numbers follow the arXiv v5 LaTeX source (updated 2026-09-23 per metadata).

Lineage

• Builds on: none. This is an evaluation of one closed model. Its background covers chain-of-thought §1.1 and RLHF §3.
• Compared against: gpt-4o, gpt-4-turbo, o1-mini, gpt-4o-mini and gpt-3.5-turbo on radiology tab. 3; ChipNeMo on chip design §4.11.
• Superseded by: none recorded in this wiki.