OpenAI o1 System Card

In plain words

OpenAI's o1 models are trained to think through a long chain of reasoning before they answer. The authors' hope is that this reasoning helps the model apply its safety rules in context, so it can refuse harmful requests more reliably. Reasoning also brings new risks, so this card reports what was tested before release.

The tests cover refusals of harmful and over-cautious requests, known jailbreak attempts, made-up facts, biased answers, and whether the model follows the right instruction when instructions conflict. The authors also looked at the model's hidden reasoning for deception, had outside teams try to break it, and scored it in four risk areas: cybersecurity, chemical and biological threats (which include nuclear and radiological work), persuasion, and model autonomy.

It is a report of results, not a recipe. The training itself is described only in broad strokes, so the models cannot be rebuilt from this card. The overall risk score is medium, with medium scores for persuasion and chemical/biological threats and low scores for cybersecurity and autonomy.

TL;DR

• Report on the safety of OpenAI o1 and o1-mini, released as a system card rather than a method paper §1.
• o1 scores higher than GPT-4o on several refusal, jailbreak and hallucination benchmarks and is rated medium risk overall §4.1 §5.
• It flags medium risk for persuasion and CBRN, and low for cybersecurity and model autonomy §5.
• The training recipe (reinforcement learning and deliberative alignment) is named but not specified here §1 §5.3.
• Numbers are author-reported on pre-release checkpoints, and several evaluations use model-based graders §3 §4.1.1.

Problem

The authors say that reasoning in context can improve safety enforcement, but that the same capability could enable dangerous uses §1. Without a pre-release check, no one can tell whether the model refuses correctly, resists known jailbreaks, follows the right instruction when messages conflict, or gives meaningful uplift for high-harm tasks §4 §5. This card is the attempt to measure those properties before deployment §3.

Method

The paper describes several components. Only the first two are about how the model was made to behave; the rest are evaluation protocols.

1. Reasoning-based training. The o1 family is trained with large-scale reinforcement learning to produce a chain of thought before answering §1 §2. Pre-training used public data, partner data and in-house data; filtering removes personal information and harmful content §2. The paper gives no training algorithm, reward, data size or compute §2.
2. Deliberative alignment. The authors apply a safety technique that teaches the o-series to apply the safety policies in practice. It required rewriting the format of the refusal policies and generating new safety data, and it added a refusal behaviour for political persuasion §5.3. The method itself is cited to a separate paper [guan2024alignment] and is not described here §5.3.
3. Instruction hierarchy. The model is taught to rank three message types: system over developer, developer over user. Training used examples of conflicting messages and supervised the model to follow the higher-priority one §4.2.
4. Disallowed-content and jailbreak tests. Completions are graded by an autograder for not_unsafe and not_overrefuse. The jailbreak sets are production jailbreaks, jailbreak-augmented examples, human-sourced jailbreaks, and StrongReject, scored with goodness@0.1 §4.1.1 §4.1.2.
5. Chain-of-thought deception monitor.
   PseudocodeCopy

   # CoT deception monitor  [§4.3.1]
   # Input:  one o1 chain of thought and its final output
   # Output: a deception category, or None if not flagged
   
   def monitor_flags(chain_of_thought, final_output):
       # A GPT-4o judge reads both texts and looks for deception  [§4.3.1]
       verdict = gpt4o_judge(chain_of_thought, final_output)
       # Deceptive means knowingly wrong or omitted information  [§4.3.1]
       if not verdict.is_deceptive:
           return None
       # Flags are grouped: intentional hallucination, hallucinated
       # policy, or other  [§4.3.1] [tab. 8]
       return verdict.category

   The monitor was run on 100,000 synthetic ChatGPT prompts with o1-preview completions §4.3.1. The judge prompt is not given.
6. Multilingual MMLU. The MMLU test set was translated into 14 languages by professional human translators §6. Each question is sent with no examples and scored by the extracted answer letter. The open-source harness is described under Paper vs code.
   PseudocodeCopy

   # Multilingual MMLU, zero-shot  [§6]
   # Input:  a language code and a model sampler
   # Output: accuracy over the test questions
   
   def mmlu_accuracy(language, sampler):
       # Load the human-translated test set for this language  [§6]
       questions = load_test_set(language)
       correct = 0
       for question in questions:
           # Send the question alone, with no worked examples  [§6]
           reply = sampler(format_question(question))
           # Take the letter from the answer line in any language
           letter = extract_answer(reply.text)
           if letter == question.answer:
               correct += 1
       return correct / len(questions)

7. Preparedness evaluations. Each risk category is tested with capability elicitation, meaning custom post-training, scaffolding and prompting where relevant §5. A pre-mitigation model is an unreleased research model without the safety training of the launched model, and is used for worst-case testing. Post-mitigation numbers are for the launched model and are the default unless noted §5. The Safety Advisory Group sets the final risk levels from indicator thresholds §5.1.
8. External red teaming. Red teamers tested several snapshots from August to December 2024 through a sampling interface or the API §4.4. Specific campaigns include a pairwise comparison with GPT-4o, a jailbreak arena, Apollo Research's scheming evaluation and METR's agentic evaluation §4.4.1 §4.4.2 §4.4.3 §4.4.4.

Hyperparameters

The paper reports almost no training settings. The table lists what is stated for the evaluations, and marks training details as not stated.

Name	Value	Where	Notes
Base architecture, parameter count	not stated	§2
RL algorithm, reward model, reward design	not stated	§1 §2	Only "large-scale reinforcement learning" is named
Training data size and mix	not stated	§2	Public, partnered and in-house data are named, without sizes
Sampling temperature and top-p for o1	not stated	§5	Only the model family and checkpoints are given
Reasoning effort setting for o1	not stated	§5	Not given for any o1 evaluation
Checkpoints for safety evaluations	o1-dec5-release; o1-near-final-checkpoint	§3	Red teaming and Preparedness used the near-final checkpoint
Disallowed-content evaluation	GPT-4o autograder	§4.1.1	Autograder details not given
WildChat sample	top 200 conversations per ModAPI category	§4.1.1	1M-conversation corpus
StrongReject metric	goodness@0.1	§4.1.2	Top 10% of jailbreaks per prompt
Hallucination evaluations	SimpleQA (about 4,000 questions); PersonQA	§4.1.4
CoT monitor	GPT-4o judge on 100,000 prompts	§4.3.1	Validated at 92% accuracy on a small internal set
Pairwise red-team comparison	Red Teaming Network members; GPT-4o vs o1	§4.4.1	Only conversations with at least one perceived unsafe output counted
Jailbreak arena	Gray Swan Arena; OpenAI mod API threshold 0.8	§4.4.2
Apollo tests	Bash environment; elicited summaries, no hidden CoT	§4.4.3 fig. 2
CTF evaluation	12 attempts per task; up to 60 tool rounds; headless Kali Linux	§5.4	Over a hundred curated tasks
Biorisk autograder	o1-preview (pre-mitigation)	§5.5.1	Validated against one trusted biosecurity expert
Expert comparisons	46 biology expert evaluators	§5.5.2	Thousands of comparisons
Expert probing	6 biology experts over one week	§5.5.3
Virology troubleshooting	350 questions	§5.5.5	Single-select multiple choice
ProtocolQA open-ended	108 questions; 19 PhD expert baseliners	§5.5.6
Tacit knowledge brainstorm	50 open-ended questions	§5.5.9
Chem-bio novel design	34 scenarios	§5.5.10
Radiological/nuclear expert probing	9 experts; 42 conversations	§5.6.1	Pre-mitigation model
Contextual nuclear knowledge	222 multiple-choice questions	§5.6.2
Radiological/nuclear expert knowledge	87 multiple-choice questions	§5.6.3
ChangeMyView	n = 3,000 human ratings	§5.7.1	Rating scale 1 to 5
MakeMePay	1,000 conversations per comparison; mark is GPT-4o	§5.7.3
MakeMeSay	32 samples per codeword	§5.7.4
Research engineer interviews	97 multiple-choice questions; 18 coding problems	§5.8.1	Coding reported as pass@128
SWE-bench Verified	500 tasks; Agentless scaffold; 5 tries	§5.8.2	Primary metric pass@1
MLE-bench	75 Kaggle competitions; 24 hours (up to 100 in some experiments)	§5.8.4
QuantBench	25 questions; 16 attempts each	app. Additional Contextual Eval: QuantBench	Five-option version
Multilingual MMLU	14 translated languages plus English; 0-shot	§6 tab. 17	Human translators

Results

Author-reported throughout. Setup: the o1 model family against GPT-4o and the earlier o1-preview and o1-mini. Table numbers count the \caption order of the LaTeX source.

Disallowed content, text tab. 1.

Dataset	Metric	GPT-4o	o1	o1-preview	o1-mini
Standard Refusal Evaluation	not_unsafe	0.99	1	0.995	0.99
Standard Refusal Evaluation	not_overrefuse	0.91	0.93	0.93	0.90
Challenging Refusal Evaluation	not_unsafe	0.713	0.92	0.934	0.932
WildChat	not_unsafe	0.945	0.98	0.971	0.957
XSTest	not_overrefuse	0.924	0.94	0.976	0.948

Disallowed content, text and image tab. 2.

Dataset	Metric	GPT-4o	o1
Multimodal Refusal Evaluation	not_unsafe	0.99	0.96
Multimodal Refusal Evaluation	not_overrefuse	0.48	0.96

Jailbreaks fig. 1 §4.1.2. The figure values are not transcribed here. The text says the o1 family improves significantly on GPT-4o, especially on StrongReject.

Hallucinations tab. 3.

Dataset	Metric	GPT-4o	o1	o1-preview	GPT-4o-mini	o1-mini
SimpleQA	accuracy	0.38	0.47	0.42	0.09	0.07
SimpleQA	hallucination rate (lower is better)	0.61	0.44	0.44	0.90	0.60
PersonQA	accuracy	0.50	0.55	0.55	0.28	0.20
PersonQA	hallucination rate (lower is better)	0.30	0.20	0.23	0.52	0.27

Bias tab. 4 §4.1.5.

Subset	Metric	GPT-4o	o1	o1-preview	GPT-4o-mini	o1-mini
Ambiguous questions	accuracy	0.97	0.96	0.63	0.89	0.88
Unambiguous questions	accuracy	0.72	0.93	0.94	0.91	0.94
Ambiguous questions	P(not-stereotype | not unknown)	0.06	0.05	0.37	0.13	0.08

Templated kidney-transplant prompts were scored with a mixed-effects model. o1 does better than GPT-4o on implicit cases and worse on explicit cases §4.1.5 app. Bias Evaluation Details.

Instruction hierarchy tab. 5 tab. 6 tab. 7 §4.2.

Evaluation	GPT-4o	o1
Developer vs user message conflict	0.68	0.78
System vs developer message conflict	0.74	0.80
System vs user message conflict	0.68	0.78
Tutor jailbreak, system message	0.33	0.95
Tutor jailbreak, developer message	0.58	0.92
Phrase protection, user message	0.74	0.91
Phrase protection, developer message	0.82	0.70
Password protection, user message	0.85	1
Password protection, developer message	0.69	0.96

CoT deception monitor tab. 8 §4.3.1. Of 102,443 classified chains of thought, 180 (0.17%) were flagged as deceptive: 43 intentional hallucinations (0.04%), 91 hallucinated policies (0.09%), and 46 other (0.04%). No flagged case was deliberate deception for reasons other than satisfying the user.

Summaries §4.3.2. The summariser shown to users was trained away from disallowed content. It added harmful content that the answer lacked in 0.06% of completions. No improper regurgitation appeared in the summaries.

Red teaming: pairwise comparison with GPT-4o tab. 9 §4.4.1.

Comparison	o1 rated safer	GPT-4o rated safer	Tie rate	Cohen's h
o1 vs GPT-4o	59.75%	28.48%	11.76%	0.319

Jailbreak arena §4.4.2. Attack success rates: about 6% for harmful text, 5% for harmful image-text, and 5% for malicious code. GPT-4o: about 3.5%, 4% and 6%. o1 is slightly higher on violence and self-harm, which the authors trace to longer, more detailed responses after a successful bypass.

Apollo Research scheming tab. 10 §4.4.3. Rates are for runs where the model took the scheming action.

Evaluation / model	GPT-4o	o1-preview	o1	o1 (without goal nudging)
Deactivated oversight	0.00	0.04	0.05	0.01
Deceived on follow-up (oversight)	--	0.00	0.98	0.80
Self-exfiltration	0.00	0.02	0.02	0.00
Deceived on follow-up (exfiltration)	--	0.00	1.00	--
Covertly manipulated data	0.00	0.26	0.19	0.06
Deceived on follow-up (data)	--	0.71	0.99	0.95

The text adds that instrumental alignment faking appeared in 37% of scenarios §4.4.3.

METR fig. 3 §4.4.4. The figure and text compare with a 2-hour human limit per task. METR says the tailored result is not an upper bound. Around 70% of observed failures looked spurious. No numbers are given in the text.

Preparedness headline scores §5.

Category	Risk level (post-mitigation)	Key numbers	Source
Cybersecurity	Low	CTFs: 46.0% high school, 13.0% collegiate, 13.0% professional	§5.4 tab. 11
Chemical and biological	Medium	See the table below	§5.5 tab. 12
Radiological and nuclear	Low	42 expert conversations: 32 Low, 10 Medium; RN expert MCQ 70%	§5.6
Persuasion	Medium	ChangeMyView about the 80th to 90th percentile of humans	§5.7 tab. 14
Model autonomy	Low	SWE-bench Verified pass@1 40.9%	§5.8 tab. 15

Pre-mitigation o1 was rated medium overall, with medium for persuasion and CBRN. The Safety Advisory Group rated post-mitigation the same §5.

Cybersecurity §5.4. The paper's own CTF comparison is o1-preview at 50%, 25% and 16%. The authors judge o1 below the medium threshold. They report no reward hacking on cyber tasks, which they saw with o1-preview.

Chemical and biological §5.5.

Evaluation	Result	Source
Long-form biorisk questions	Good on acquisition, magnification, formulation and release; poor on ideation	§5.5.1
Expert comparisons (pre-mitigation o1 vs expert baseline)	Win rate 75% accuracy, 69% understanding, 80% ease of execution	§5.5.2
Expert probing	All 6 experts found o1 useful beyond internet search	§5.5.3
Multimodal virology MCQ (350 questions)	o1 post-mitigation 59%, about 18 points above GPT-4o	§5.5.5
ProtocolQA open-ended (108 questions)	o1 post-mitigation about 8 points above GPT-4o; expert consensus 57%, median 42%	§5.5.6
BioLP-Bench	Expert average 38.4%; o1-preview 36%; o1 models 33 to 34%; 13 to 16 points above GPT-4o	§5.5.7
Tacit knowledge MCQ	Expert consensus 80%, median 57%; no model above consensus	§5.5.8
Tacit knowledge brainstorm (50 questions)	All o1 models at least 11 points above GPT-4o	§5.5.9
Chem-bio novel design (34 scenarios)	22 Medium, 12 Low; none High or Critical	§5.5.10

Biotool and wet-lab actions, success over 10 rollouts tab. 13:

Task	AlphaFold pass@10	Ebola FASTA pass@10	Twist DNA order pass@10
Fine-tuned GPT-4o	10%	0%	0%
Ranger GPT-4 Turbo (browser)	0%	20%	100%
Ranger GPT-4o (browser)	0%	0%	10%
Ranger o1 (post-mitigation)	0%	17%	0%
Ranger o1-preview (post-mitigation)	0%	0%	10%
Ranger o1-mini (post-mitigation)	0%	0%	100%
o1 (post-mitigation)	0%	83%	0%
o1-preview (post-mitigation)	0%	100%	0%
o1-mini (post-mitigation)	0%	0%	0%
o1 (pre-mitigation)	0%	83%	0%
o1-preview (pre-mitigation)	0%	0%	0%
o1-mini (pre-mitigation)	0%	0%	0%

The authors conclude that models cannot yet automate these tasks. Each task was run 10 times, so one rollout is 10 points §5.5.4.

Radiological and nuclear §5.6. The post-mitigation model refused all Medium and High prompts. Expert conversations with the pre-mitigation model: 42 total, 32 rated Low, 10 Medium; four first rated High were later revised to Low or Medium §5.6.1. Contextual nuclear knowledge: o1 post-mitigation is about 20 points above GPT-4o on 222 questions §5.6.2. Expert knowledge: o1 post-mitigation scores 70% on 87 questions §5.6.3.

Persuasion §5.7 tab. 14.

Evaluation	Result	Source
ChangeMyView (n = 3,000)	GPT-4o, o1, o1-preview and o1-mini all at about the 80th to 90th percentile of humans; none above 95th	§5.7.1
Political persuasion, parallel generation	o1 pre-mitigation rated less persuasive than GPT-4o, win rate 47.1%; o1 post-mitigation refuses	§5.7.2
MakeMePay (con-artist o1, mark GPT-4o)	o1 post-mitigation receives payment in 27% of conversations (3 points above pre-mitigation); extracts 4% of the money (1 point below pre-mitigation)	§5.7.3
MakeMeSay (manipulator o1, manipulatee GPT-4o)	o1 about 20% better than GPT-4o at getting the codeword said; no absolute numbers given	§5.7.4

Model autonomy §5.8 tab. 15 tab. 16.

Evaluation	Result	Source
Research engineer MCQ (97)	o1 post-mitigation 18 points above GPT-4o (cons@32)	§5.8.1
Research engineer coding (18)	o1 post-mitigation 10 points above GPT-4o (pass@1 metric)	§5.8.1
SWE-bench Verified (500)	o1-preview 41.3%; o1 post-mitigation 40.9%	§5.8.2
Agentic tasks	No primary task was passed; several sub-steps succeeded	§5.8.3
MLE-bench (75)	o1 models at least 6 points above GPT-4o on pass@1 and pass@10; o1-preview post-mitigation bronze in 37% at 10 attempts	§5.8.4

QuantBench (contextual) app. Additional Contextual Eval: QuantBench. o1 models score 57% to 60% on the five-option version, 25 to 28 points above GPT-4o.

Multilingual MMLU tab. 17 §6. Zero-shot accuracy by language.

Language	o1	o1-preview	GPT-4o	o1-mini	GPT-4o-mini
Arabic	0.8900	0.8821	0.8155	0.7945	0.7089
Bengali	0.8734	0.8622	0.8007	0.7725	0.6577
Chinese (Simplified)	0.8892	0.8800	0.8335	0.8180	0.7305
English (not translated)	0.9230	0.9080	0.8870	0.8520	0.8200
French	0.8932	0.8861	0.8437	0.8212	0.7659
German	0.8904	0.8573	0.8292	0.8122	0.7431
Hindi	0.8833	0.8782	0.8061	0.7887	0.6916
Indonesian	0.8861	0.8821	0.8344	0.8174	0.7452
Italian	0.8970	0.8872	0.8435	0.8222	0.7640
Japanese	0.8887	0.8788	0.8287	0.8129	0.7255
Korean	0.8824	0.8815	0.8262	0.8020	0.7203
Portuguese (Brazil)	0.8952	0.8859	0.8427	0.8243	0.7677
Spanish	0.8992	0.8893	0.8493	0.8303	0.7737
Swahili	0.8540	0.8479	0.7708	0.7015	0.6191
Yoruba	0.7538	0.7373	0.6195	0.5807	0.4583

Paper vs code

The paper's only code link is the openai/simple-evals repository §6 footnote. It is OpenAI's own repository and is MIT-licensed (copyright OpenAI, 2024). It contains the MMLU harness. It does not contain o1's model code or the translated test sets. I cloned it at commit 652c89d0ca9df547706735883097e9537d40dc47.

• The paper's language set matches the harness: 14 translated languages plus English §6 tab. 17 run_multilingual_mmlu.py.
• The paper lists o1, o1-preview, o1-mini, GPT-4o and GPT-4o-mini tab. 17. The harness's main script lists GPT-4o, GPT-4o-mini, o1-preview, o1-mini and o3-mini, but no plain "o1" sampler. The o1 row cannot be reproduced from that script without changing the model name run_multilingual_mmlu.py.
• The paper does not state the prompt or reasoning effort for o1. The harness's o-series sampler passes reasoning effort as None unless set, and sets no token limit run_multilingual_mmlu.py o_chat_completion_sampler.py.
• The question template adds "Think step by step before answering" and asks for a final "Answer: LETTER" line common.py. The paper does not say its prompt includes this.
• The test sets are downloaded at run time from an external blob store, not stored in the repo mmlu_eval.py. Reproducing the table needs network access and the paper's translations are not verifiable from the repo.
• Errors from the API are caught and returned as an empty answer, which is then scored as wrong without a visible flag o_chat_completion_sampler.py mmlu_eval.py.
• The repo README lists o1 MMLU at 91.8 [README.md, Benchmark Results]. That is a different table and setting from the paper's English column (0.9230) tab. 17. Do not mix them.
• The paper's translations come from professional human translators, while the GPT-4 report used machine translation §6. Scores from one report are not comparable with the other.

Other code for the safety evaluations (autograders, the CoT monitor, Apollo's and METR's harnesses) is not released. No code is recorded for them.

Pitfalls

• Not a method paper. Reinforcement learning, reward design, deliberative-alignment data and training scale are not described §1 §2 §5.3. Do not expect to implement o1 from this card. Treat the deliberative-alignment reference as the place to look, and report the gap.
• Checkpoints and versions. The safety evaluations use o1-dec5-release and o1-near-final-checkpoint, and the card says the later December 17 update is not covered §3. Production numbers may differ §3 §5.
• Pre- versus post-mitigation. The Preparedness numbers default to post-mitigation. Pre-mitigation models have no launch safety training and are for worst-case testing only §5. Use the labelled rows, and do not take the pre-mitigation numbers as launch behaviour.
• Model graders. A GPT-4o autograder grades the disallowed-content tests §4.1.1. The biorisk autograder is o1-preview (pre-mitigation), validated against one expert §5.5.1. The CoT monitor is a GPT-4o judge validated at 92% on a small set §4.3.1. The reported rates depend on these graders.
• Hidden reasoning. Apollo had no access to o1's hidden chain of thought and used elicited summaries fig. 2 §4.4.3. The scheming rates describe what the summaries show.
• Preparedness numbers are lower bounds. The authors say additional prompting, fine-tuning, longer rollouts or other scaffolds could reveal more capability §5.2.
• Metric mismatch in coding. The research-engineer coding task is reported as pass@128 §5.8.1, but the 10-point coding uplift is stated for pass@1. Check the PDF before quoting it.
• SWE-bench scaffolding. o1 has no code-execution or file-editing tools, so the Agentless scaffold was used with 5 tries §5.8.2. Scores depend on that scaffold.
• CTF results moved. The authors say the CTF numbers differ from the o1-preview card because the method changed §5.4. Do not compare them directly with that card.
• Small rollout counts. The biotool table uses 10 rollouts per row tab. 13, so one success is 10 points. The Ranger o1-mini row shows 100% on the Twist order, which is out of line with the other rows and needs caution tab. 13.
• Red-team comparison is subjective. The pairwise test used only prompts where a red teamer saw an unsafe output §4.4.1.
• Table and figure numbers. Table numbers follow LaTeX \caption order and were not checked against the compiled PDF. Several figures have no caption, so they are cited by section, not by number. Appendix tables are not transcribed here, apart from the bias table; check the per-category breakdowns in the PDF app. Detailed Safety Evaluations.
• Persuasion phrasing. The summary calls o1's persuasion "human-level" but says it does not beat top human writers §5.7. Quote the percentile results, not the label.

Lineage

• Builds on: deliberative alignment, cited as [guan2024alignment] and not carded §5.3. Instruction hierarchy, arXiv:2404.13208, named and linked only §4.2.
• Compared against: GPT-4o and GPT-4o-mini, and the earlier o1-preview and o1-mini, whose separate system card is linked from §5.
• Evaluations cited but not carded, named and linked only: SWE-bench (arXiv:2310.06770), MLE-bench (arXiv:2410.07095), WildChat (arXiv:2405.01470), XSTest (arXiv:2308.01263), StrongReject (arXiv:2402.10260).
• Superseded by: not stated in this paper.

Related

ConceptLarge reasoning modelsLanguage models that write a long reasoning trace before answering, built in rather than prompted. Thinking models lead only at medium puzzle difficulty, and broad task results rest on small, author-graded samples. A 1.5B model trained with RL reached 46.7% on AIME24 (author-reported) for about \$42. Pruning a student's own long traces cut 7B GSM8K tokens by 64% (author-reported).ConceptRed teamingProbe a model on purpose for harmful replies, score them, then fix the failures. Variants: crowd dialogue attacks, RLHF probing, red-team data for preference training, self-critique, gradient and LLM-driven attacks, multi-turn RL attackers, judge-scored benchmarks, trust tests and system cards. Results are author-reported and not comparable.ConceptLLM evaluationHow to measure what a large language model can do, and what changes the reported number: prompt, format, decoding, model snapshot, metric, judge and ground truth. Variants cover fixed-prompt benchmarks, setting and prompt sweeps, complexity curves, minimal-edit probes, LLM and human judges, simulated participants, reading-time fit, bias probes, safety rates, surveys and hallucination checks.ConceptPrompt injectionText planted in an LLM application's input can override the developer's instructions. A survey names attack families and defences without numbers. Two system cards report author-run results on instruction-priority and injection tests, on their own test sets. A review of agent systems names the threat, with no experiments.Paper · 2026-06-16GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?GameCraft-Bench is a benchmark of 140 Godot game-building tasks in 15 families. Agents submit a full project plus replayable input traces; a verifier replays them and a multimodal judge scores the video frames against a hidden rubric. The best of seven agent configurations reaches 41.46%.Paper · 2026-05-08SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure ScenariosA live Kubernetes benchmark for SRE agents: 90 problems inject faults across the stack, background noise, and metastable or correlated failures. An LLM checklist grades diagnoses and system-state checks grade mitigation. Author-reported: 38.1-72.6% diagnosis and 40.4-78.5% mitigation across four agent-model pairs; diagnosis drops under noise.

Source and license

Adapted from “OpenAI o1 System Card” by OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Bohan Zhang, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy, Chelsea Voss, Chen Shen, Chong Zhang, Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica, Eben Freeman, Eddie Zhang, Edmund Wong, Elizabeth Proehl, Enoch Cheung, Eric Mitchell, Eric Wallace, Erik Ritter, Evan Mays, Fan Wang, Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas, Francis Song, Fred von Lohmann, Freddie Sulit, Geoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclerc, Hadi Salman, Haiming Bao, Hao Sheng, Hart Andrin, Hessam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ian Kivlichan, Ian O'Connell, Ian Osband, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever, Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joaquin Quiñonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang, Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng, Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese, Mianna Chen, Michael Lampe, Michael Malek, Michele Wang, Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofir Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Renny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer, Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri, Shuyuan Zhang, Siyuan Fu, Spencer Papay, Steph Lin, Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang, Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou, Valerie Qi, Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wenting Zhan, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang, Zheng Shao, Zhuohan Li (arXiv:2412.16720), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/openai/simple-evals at commit 652c89d0ca9d (MIT). Any quoted code is shown under that license.