GPT-4o System Card

In plain words

GPT-4o is a single model that takes in text, audio, images and video and can answer with text, audio or images. Voice is not bolted on through a separate speech-to-text and text-to-speech pipeline, and it can respond to speech in about the time a person would take to reply.

This document is a system card, not a technique paper. It is the developer's public account of what the model can do, where it falls short, and what was done to reduce its risks before release. Most of the safety work concerns voice: stopping the model from copying a speaker's voice, from naming a person it hears, and from guessing traits such as intelligence or accent in a harmful way.

Because the model is closed, nobody outside can re-run these tests. The numbers are the authors' own, so read them as their claims rather than as independent measurements.

TL;DR

• GPT-4o is an end-to-end model over text, audio, image and video. The card focuses on speech-to-speech risks and also covers text and vision §1.
• Audio evaluations were built by converting existing text benchmarks to speech with a TTS system and scoring the text of the reply §3.2.
• Preparedness scores: cybersecurity, biological threats and model autonomy are low; persuasion is borderline medium before mitigations; the overall score is medium §3.4.
• Third-party tests by METR and Apollo Research found no significant jump over GPT-4 on autonomy tasks, and Apollo judged catastrophic scheming unlikely §4.
• There is no code, weights or training recipe to reproduce. Use the card for evaluation design and for the reported risk numbers.

Problem

Before release, the developers had to record what a speech-to-speech model can do and which risks it creates, including risks that text-only evaluations do not cover §1 §3. The audio-specific risks they name are unauthorized voice generation, speaker identification, ungrounded inference and sensitive trait attribution, and gaps in disallowed-content safety for audio output §3.3. The card states that its risk list is illustrative and non-exhaustive, and that it centres on the ChatGPT experience §3.3.

The card also assumes that TTS-generated inputs are a reasonable stand-in for real speech. The authors list the cases where this fails, such as math, code and heavy formatting §3.2. Most Preparedness evaluations cover text and vision only, depending on the risk §1.

Method

The card describes no training or inference algorithm. It describes four procedures:

1. Red teaming in four phases, with growing numbers of red teamers and checkpoints that accept more modalities §3.1.
2. Audio versions of existing text benchmarks. Text prompts are synthesized with a TTS system (Voice Engine), the model answers, and only the text of the reply is scored, except where the audio itself is checked, as in voice cloning §3.2.
3. Mitigations. Preset voices are used as ideal completions during post-training, a streaming classifier checks the output voice, refusals are post-trained for speaker identification and ungrounded inference, and the existing moderation classifier runs on transcripts of audio input and output §3.3.
4. Preparedness evaluations under four risk categories, with a Safety Advisory Group classification, followed by third-party assessments §3.4 §4.

The first procedure, the audio benchmark pipeline, is written out below. The card does not name the transcription system used to score replies §3.2.

# Audio version of a text benchmark  [§3.2]
# Input:  examples (text prompt, expected answer), tts_voice
# Output: one score per metric
def score_audio_benchmark(examples, tts_voice, model):
    # Synthesize each text prompt as speech with a TTS system  [§3.2]
    prompts_audio = [
        text_to_speech(example.prompt, tts_voice)
        for example in examples
    ]
    # Let the speech model answer with audio  [§3.2]
    replies_audio = [model.respond(audio) for audio in prompts_audio]
    # Score only the text of each reply, not its audio  [§3.2]
    replies_text = [transcribe(reply) for reply in replies_audio]
    return compute_metrics(replies_text, examples)

The voice-consistency guard works on the output stream and not on the finished reply. The card describes it in prose §3.3.1. The pseudocode below follows that description.

# Streaming guard on generated audio  [§3.3.1]
# Input:  audio_chunks (generated in order), preset_voice
# Output: audio chunks that passed the check
def guard_voice_stream(audio_chunks, preset_voice, classifier):
    # Check each chunk while audio is still being generated  [§3.3.1]
    for chunk in audio_chunks:
        # Block when the speaker is not the chosen preset voice  [§3.3.1]
        if not classifier.is_preset_voice(chunk, preset_voice):
            # Discontinue the conversation  [§3.3.1]
            raise VoiceMismatch()
        yield chunk

Key equations: none. The card contains no numbered equations or algorithms.

Hyperparameters

The card reports no model hyperparameters. The table lists the settings it does state, and marks the rest as not stated.

Results

Author-reported. The setup for each result is given in the Setting column. Baseline is the comparison the authors used; "none" means the card gives no baseline.

Speech safety

The speaker-identification text reports a 14-point and a 12-point improvement; the table gives 0.83 to 0.98 and 0.70 to 0.83. See Pitfalls.

Preparedness

Persuasion details. For text, AI articles and chatbots were not more persuasive than human-written articles in aggregate, but exceeded them in three of twelve instances. For voice, AI clips reached 78% and AI conversations 65% of the human effect size on opinion shift. One week later, the conversation effect was 0.8% and the clip effect was -0.72% §3.7.

Third-party and societal

Health, science and languages

Table 6 covers 22 health rows. The card reports more gains than the table shows (see Pitfalls).

Paper vs code

No official code. The paper links no code repository: its links point to the METR report, to translated datasets on Hugging Face, and to Kali Linux §3.5 §4.1 §5.4. No project page is linked. A GitHub repository search for gpt-4o system card restricted to the openai organisation returned no results. No code was cloned, and no code, code_commit or code_license is recorded.

Pitfalls

• Closed model, no recipe. The card gives no model size, architecture, learning rate, context length or data mixture ratios §2. An implementer cannot rebuild the model from it. The safety procedures are described in prose only.
• Audio evals are text evals through TTS. Scores depend on the TTS system as well as the model. The authors say TTS is lossy for equations, code and formatting-heavy text, and that mistakes may come from TTS §3.2. The card names Voice Engine but not the voice settings per test, nor the transcription system used for scoring §3.2.
• Coverage differs by risk. Most Preparedness work is text and vision only §1. Do not assume the biology, cybersecurity or autonomy results cover speech.
• Biology results are not all for the deployed model. The uplift study compared people using the internet, GPT-4o, or a specially trained research-only version that answers without refusals §3.6. Only some arms reflect the deployed model.
• Speaker identification numbers disagree. The text claims a 14-point improvement in refusing and a 12-point improvement in complying. The table gives 0.83 to 0.98 (15 points) and 0.70 to 0.83 (13 points) §3.3.2 tab. 2. Use the table values, since the text may be rounded.
• Health gains are overstated in the text. The text says performance improves on "21/22" evaluations. Table 6 shows 20 strict improvements and two ties at 0.95 (MMLU Medical Genetics 5-shot and MMLU College Biology 5-shot) §5.2 tab. 6.
• A language label is wrong in the text. The text gives "ARC-Easy-Hausa" as 6.1% to 71.4%. In the table, 6.1 and 71.4 are the Amharic column. Hausa is 26.1 to 75.4 §5.4 tab. 7.
• Internal test sets are undescribed. The speaker-identification, ungrounded inference and safe-behaviour results come from internal datasets. Their sizes are not stated §3.3.2 §3.3.4 §3.3.3.
• Subsets are not public benchmarks. The voice capability results use a fixed random subset and should not be compared with published scores §3.3.3 (footnote).
• Audio robustness has no numbers. Background noise, echo and interruptions reportedly weaken safety behaviour, but only anecdotally §3.3.7.
• Non-English moderation over-refuses. Non-English conversations can be cut off unnecessarily, which the authors call an active area of improvement §3.3.1 (footnote).
• Autonomy scaffold not stated. The SWE-Bench result uses "the best available post-training and public scaffolds at the time", but the scaffold is not named §3.8. The result is hard to reproduce.
• Persuasion is near a threshold. The text result is "marginal" medium. The Safety Advisory Group classified it as borderline medium before mitigations, and that decides the overall medium rating §3.4 §3.7.
• Card date and versions. The source is dated August 8, 2024. The model it describes will have changed since, so do not treat the figures as current.
• Counting method. Section, table and figure numbers were counted from the LaTeX source, not checked against the PDF. Several tables and figures have no caption and no number (the risk table, the Preparedness scorecards, the METR plot), so they are cited by section.

Lineage

• Builds on: the GPT-4 System Card and the GPT-4(V) System Card, named in §3.1 but not linked here, since the source gives no arXiv ids for them.
• Compared against: GPT-3.5 Turbo, GPT-4, GPT-4T (May 2024) and GPT-4o mini tab. 6 tab. 7 tab. 8 tab. 9. The text also quotes Med-Gemini and Med-PaLM 2 figures from other sources §5.2.
• Related: Voice Engine, the TTS system used for the audio evals §3.2, is not in this wiki.
• Superseded by: not known from content already in this wiki.