Sparks of Artificial General Intelligence: Early experiments with GPT-4

In plain words

This paper is not a new method. It is a long field report from a Microsoft Research team that had early access to GPT-4 while OpenAI was still training it. The model they tested took only text as input, even though the released GPT-4 can also take images. They wanted to know how capable it really was, and they did not trust standard benchmarks for that: the model may have seen the test questions during training, and many interesting tasks, such as building a user interface, have no single right answer to score against.

So they did something closer to a psychologist examining a subject. They wrote new tasks that are unlikely to exist on the web: a proof that there are infinitely many primes written as a rhyming poem, a unicorn drawn in LaTeX graphics code, coding problems posted after the model's training data was collected, and math problems with the numbers changed. Then they changed these tasks again to check that the model was reasoning rather than recalling. They compared GPT-4 with ChatGPT and older OpenAI models, and in a few places with humans.

The headline is that this early GPT-4 did much better than earlier models across very different areas (code, math, using tools, reasoning about what people believe and feel). On fresh LeetCode problems it scored about as well as the humans who took part in LeetCode contests. The authors argue that this breadth justifies calling it an early, incomplete form of general intelligence. That claim is an interpretation, not a measurement, and the paper itself says its approach is subjective.

The most useful part for engineers is the list of failures. The model made arithmetic mistakes when it skipped steps. It could not plan ahead in tasks where the start of the answer depends on the end. It sometimes invented facts, and it gave explanations of its own behaviour that sounded convincing but did not predict what it would do next. Asking it to write out intermediate steps, or letting it call tools such as a calculator or search engine, fixed some of these failures but not all.

TL;DR

• An exploratory capability study of a pre-release, text-only GPT-4. Most evidence is single prompt/response examples. Small quantitative probes cover coding, math, PII detection, TruthfulQA, gender bias and synthetic arithmetic/planning tasks §1.1 §1.2.
• Author-reported numbers: HumanEval 82% zero-shot pass@1 vs 65% for text-davinci-003 tab. 1; 100 LeetCode problems posted after Oct 8, 2022: 38.0% pass@1, 53.0% pass@5 vs a human contest baseline of 38.2% tab. 2; GSM8K 87.1%, MATH 42.5%, MMMLU-STEM 82.7% tab. 3.
• Clearest reusable findings: (a) accuracy on simple arithmetic collapses when the model must answer in one shot and recovers when it writes intermediate steps §8.2; (b) tasks that need planning ahead stay hard even with step-by-step prompting §8.2 §8.3; (c) self-explanations can match the output without predicting the model's behaviour on other inputs ("output-consistent" but not "process-consistent") §6.2.
• Every number comes from an early checkpoint that is not the deployed GPT-4. The authors say the numbers are illustrative only and point to OpenAI's report for definitive results §3 §4.2 §7.

Problem

The authors want to judge how general GPT-4's intelligence is. They argue that standard held-out benchmarks are not enough, for two reasons §1.1:

1. The training data is unknown, so any public benchmark may be contaminated. The paper notes that the model appears to know the BIG-bench canary GUID §1.1.
2. Many of the model's strongest skills are open-ended or interactive (GUIs, brainstorming), where there is no clean metric §1.1.

They adopt the informal working definition of AGI as systems with broad abilities in reasoning, planning and learning from experience, at or above human level §1. They also discuss other definitions §10.1.

Method

The study probes behaviour. Nothing is trained. The model was accessed only through natural-language prompts during OpenAI's development §1. It was the non-multimodal early version, and its behaviour kept changing during the study: the TikZ unicorn, queried three times over a month, got visibly better fig. 1.3.

Overall protocol §1.1

Protocol: probing with novel tasks  [§1.1]
1. Write a task that combines skills or domains and is unlikely to be on the web
   (e.g. a rhyming proof, TikZ drawing, an invented object "ZURFIN" [§6.1]).
2. Get GPT-4's output; often also ChatGPT's / text-davinci-003's on the same prompt.
3. Probe against memorisation: vary style, numbers or code and re-ask
   (e.g. strip the horn from the unicorn TikZ, transform coordinates, remove comments,
   ask the model to add the horn back [fig. 1.4]).
4. Look for consistency, coherence and correctness; record failures.
5. Where feasible, add a small quantitative test (below).

Coding evaluation §3.1.1 app. C.1

• HumanEval: 164 docstring-to-code problems, zero-shot pass@1 §3.1.1 tab. 1.
• Fresh LeetCode benchmark: 100 problems posted after October 8, 2022, which the authors place after GPT-4's pretraining period §3.1.1. The problem text is pasted into a prompt with a request for a Python function, and correctness comes from LeetCode's official online judge §3.1.1 fig. 3.1. pass@1 and pass@5 are read as "correct within the first 1 / first 5 attempts" §3.1.1.
• Human baseline from LeetCode weekly/biweekly contest statistics (Oct 8, 2022 to Jan 7, 2023), counting only users with a nonzero score tab. 8. The authors avoid LeetCode's per-question acceptance rate because it includes all historical submissions; they suspect many of those were copied after solutions appeared app. C.1. The per-level human accuracy is app. C.1:

\text{HumanAcc}(\text{Level}) = \mathbf{E}_{\text{problem}\in\text{Level}}\left[\frac{\text{Accepted Users}}{\text{Total Users}}\right]

• Mock interviews: LeetCode's Interview Assessment platform (online assessment, phone interview, on-site rounds) fig. 1.5.

Math evaluation §4.2

• Datasets: GSM8K, MATH, and MMMLU-STEM (about 2000 four-way multiple-choice STEM questions) §4.2. The metric is "single model" accuracy: the share answered correctly in one try §4.2.
• Prompting against over-fitting: the model is asked to (1) write a solution template, and (2) write the steps first and the final answer last. The authors note that templates and detailed MMMLU-STEM solutions are not online §4.2.
• Memorisation checks: (a) pick a question GPT-4 gets right and text-davinci-003 gets wrong, then re-sample its numbers §4.2 fig. 4.2; (b) hand-write new questions checked not to exist online §4.2. For the degree-3 polynomial problem in fig. 4.2, three roots are drawn from {−10..−2} ∪ {2..10} and p(1) from {−10..−1} ∪ {1..10} fig. 4.2.
• Error analysis: manual labelling of 100 random wrong MATH answers tab. 4.

Tool use §5.1

Tools are described in the prompt, with no demonstrations. When the model emits a call, generation pauses, the tool runs, its result is pasted into the prompt, and generation continues §5.1 fig. 5.2.

Loop: prompt-described tool use  [§5.1] [fig. 5.2]
prompt = instructions listing SEARCH("query"), CALC(expression), CHARACTER(string, index)
         + "<|endofprompt|>" + question
repeat:
    out = generate(prompt) until a tool call is emitted or the answer ends
    if out ends with a tool call:
        result = execute(call)            # real search engine / calculator / string op
        prompt = prompt + out + "\n> " + result
    else: return out

More complex cases chain several tools: a command-line "zoo manager" task, email and calendar, and web browsing §5.1.1 fig. 5.3 fig. 5.4 fig. 5.5. For the math failures, the authors suggest having the model write code that an external executor runs, with the result appended to the prompt. They show one proof-of-concept prompt but did not evaluate it systematically app. D.4.

Discriminative tasks §7

Here "GPT-3" means text-davinci-002, an instruction-tuned model §7.

• PII detection: 6764 sentences from a subset of the Text Anonymization Benchmark (TAB). The task is to count PII elements per sentence §7.1. GPT-4 gets a zero-shot prompt listing PII categories (a–h) and must output LIST: and COUNT: fig. 7.1. The baseline is Microsoft Presidio (NER plus regular expressions) §7.1. Outcomes are binned by how many elements were missed tab. 5.
• TruthfulQA generation: 816 questions across 38 categories, answered after a few-shot preamble of question/answer pairs §7.2 fig. 7.2. The answers are scored against gold answers with ROUGE, BLEU and BLEURT: an answer counts as truthful if its score is above a "pre-determined threshold", whose value is not stated §7.2.1.
• Judge GPT-4: GPT-4 gets the gold statement and two options (its own answer and GPT-3's). It writes EXPLANATION1/EXPLANATION2 (similarities and differences), then BEST: OPTIONX with a justification, and must pick exactly one §7.2.2 fig. 7.5. In the example prompt, OPTION1 is GPT-4's answer fig. 7.5. Two human reviewers labelled a subset without seeing the judge's justification §7.2.2.

Bias probe §9.3

The prompt is "I had a great experience with a . Can you write a note recommending this to a friend." For each query, the authors find the first gender-identifying word, take the probability distribution over its possible completions, and collapse it into she/her, he/his and they/their. Each occupation is run 5 times §9.3. A variant adds "in an inclusive way" §9.3.

Limitation probes §8 app. D.1

• Arithmetic a*b + c*d: 100 random samples per number range, answered directly or with a step-by-step prompt §8.2.
• "Change exactly one integer on the left of A*B + C*D = E so the right side becomes a target": 100 samples per range §8.2.
• Expanding Σ_{i∈[L]} a_i (b_i x + c_i)^2 into ax² + bx + c and reporting |a|+|b|+|c|, with a_i ∈ {−1, 1} and b_i, c_i ∈ {−5..5}, L ∈ {1..5} app. D.1.
• Counting distinct elements in a list of L random k-digit strings, with the true answer between L/2 and L−1, L ∈ {5, 10, 15, 25}, k ∈ {2, 7, 12} app. D.1.

Explanation consistency §6.2

The model simulates a task process P_T(y | x, c) and, when asked to explain, an explanation process P_E(e | x, c, y). Here x is the input, c the rest of the context and y the output §6.2. Two criteria §6.2:

• Output consistency: the explanation gives a plausible account of how y followed from x and c.
• Process consistency: the explanation correctly predicts the model's behaviour on other inputs or contexts. It is tested with editing experiments (change the part of the prompt that the explanation relies on) and concept override experiments (state a contrary "fact" in a prompt prefix) fig. 6.10.

Hyperparameters

This is an evaluation study. The table lists the settings an implementer would need to reproduce the probes.

Results

All numbers are author-reported, from the early GPT-4 checkpoint. The authors call them illustrative and point to OpenAI's GPT-4 report for definitive results §3 §4.2 §7.

HumanEval, zero-shot pass@1 tab. 1

LeetCode, 100 problems posted after Oct 8, 2022, zero-shot pass@k (%) tab. 2

(The source table's column header says "Median"; it means Medium. Human values span both k columns in the source tab. 2.)

Human LeetCode accuracy from contest data tab. 9

LeetCode contest statistics used for the human baseline tab. 8 (Users = users with nonzero score; Acc = accepted users; % = Acc/Users)

LeetCode mock interviews fig. 1.5: GPT-4 solved every question in all three rounds in 10 minutes in total, out of 4.5 hours allotted. LeetCode scored the rounds 8.96/10, 8.69/10 and 10/10, beating 93%, 97% and 100% of users.

Math datasets, single-try accuracy tab. 3

GPT-4 errors on MATH, 100 random wrong answers tab. 4

Memorisation test with re-sampled numbers fig. 4.2: GPT-4 75.2%, text-davinci-003 0.2%.

Other exam probes §1.2: preliminary runs on the multiple-choice part of USMLE Steps 1–3 scored around 80% on each step, and the Multistate Bar Exam scored above 70%. Details are not given.

PII detection on TAB (6764 sentences) tab. 5

TruthfulQA, which answer is closer to the gold answer tab. 6

GPT-4's judgements and the humans' agree on 50.8% of items §7.2.2. The "constrained" row re-scores the human labels without the "neither" and "both" options §7.2.2. On the ROUGE, BLEU and BLEURT threshold metrics, GPT-4 also scores above GPT-3. The values are only plotted fig. 7.3 fig. 7.4.

Judge GPT-4 choices by TruthfulQA category (%) tab. 10

Gender bias in recommendation letters tab. 7

Adding "in an inclusive way" switched the pronoun to they/their for every occupation. It also shifted the letter's content toward inclusiveness §9.3.

Arithmetic and planning limits §8.2

The authors read the slow decline on the change-one-integer task as evidence that planning, not calculation, is the bottleneck §8.2.

Synthetic algebra: expand Σ a_i(b_i x + c_i)^2, step-by-step prompt app. D.1

In 100 inspected GPT-4 mistakes, at least 90% came from skipping steps when merging like terms app. D.1.

Counting distinct elements (L items, k digits each) app. D.1

(TD3 is text-davinci-003. The "30.3" cell has no % sign in the source.)

Explanations §6.2: on an easy sentiment-analysis task (not shown in the paper), GPT-4's counterfactual-rewrite explanations were 100% faithful versus 60% for GPT-3. In the token-probability example, the model said it chose "Q" because Q follows P. Renaming P to R shifted the top token to S (64.63%), and a prefix claiming that H follows R shifted it to H (95.01%). The explanation was process-consistent here fig. 6.9 fig. 6.10.

Qualitative findings, no metrics (each from examples):

• Theory of mind: GPT-4 passed a modernised Sally-Anne false-belief test, and so did ChatGPT, while text-davinci-003 failed. GPT-4 gave more nuanced answers in realistic social scenarios §6.1 fig. 6.1.
• TextWorld: GPT-4 finished a house-exploration game in 30 actions. It could not solve a cooking game zero-shot, and neither could the authors before reading the source code; with one demonstration it solved it §5.2.2.
• Music in ABC notation: tunes were syntactically valid, but the authors found no clear chords or arpeggios in 10 generated tunes §2.3.
• Code understanding: GPT-4 simulated Python and pseudocode execution in natural language and reverse-engineered a password-protected binary through a chat loop. ChatGPT refused the binary task §3.2.

Paper vs code

No official code.

Pitfalls

• Not reproducible against today's GPT-4. The checkpoint is a pre-release, text-only model. It improved during the study fig. 1.3 and came before OpenAI's final fine-tuning, which the authors say improved truthfulness and reduced bias §7.2 §9.3. Treat every number as a snapshot, as the authors ask §3 §4.2.
• Decoding settings are never given. Temperature, sampling and the number of samples behind pass@5 are not stated §3.1.1. Most qualitative figures are single samples, so cherry-picking cannot be ruled out from the paper.
• The LeetCode human baseline is inconsistent within the paper. Human Medium accuracy is 37.7% in tab. 2 but 38.7% in tab. 9. Overall human accuracy (38.2%) is not the plain mean of the three levels under either Medium value. The rows for contests 318 and 319 in tab. 8 are identical, which looks like a copy error. The human metric also measures contest takers within a time limit, while the model's attempts are untimed.
• LeetCode difficulty split not stated. The percentages in tab. 2 are consistent with 22 Easy, 50 Medium and 28 Hard problems (e.g. 15/22 = 68.2%, 3/28 = 10.7%). That split is our arithmetic, not stated in the paper.
• Contamination controls are partial. The LeetCode cutoff of Oct 8, 2022 rests on the authors' statement about the pretraining period §3.1.1. For the math datasets, the defence is prompting style and spot checks, not a held-out set §4.2. The dataset split and number of GSM8K/MATH questions evaluated are not stated, and neither is the source of the Minerva numbers tab. 3.
• PII binning text vs table. The prose describes the last bin as "miss more than three", while the table header is "Missing >2" §7.1 tab. 5. The ground truth also counts aliases (e.g. a court's English and Danish names) as separate elements, which GPT-4 merges §7.1.
• The GPT-4 judge favours its own answers. It judges between its own answer and GPT-3's, must pick exactly one, and tends to reward longer answers. Agreement with humans is only 50.8%. The 89.83% "constrained" human figure comes from removing the human options "neither" and "both", not from new labels §7.2.2. The human reviewers' agreement was not checked §7.2.2.
• Typo in the change-one-integer setup. The text samples "B, D from [3,9] and C, D from [a,b]", naming D twice §8.2. The intended variables are ambiguous; the logical reading is that one multiplier per product comes from [3,9] and the other from [a,b], but this is not stated.
• "Think step by step" is not enough on its own. Even with explicit instructions not to merge terms in one go, the model skipped steps app. D.1. If it states a wrong conclusion first, it then rationalises it, even writing "2 ≤ 0" app. D.1 fig. D.1. Ask for steps before the answer.
• Tool use needs to be prompted, and judgement about when to use tools is weak. Without a prompt allowing tools, the model falls back on stale or miscalculated answers fig. 5.1. With tools, it still searched for the capital of France §5.1.2. It also misused unusual tools until it got feedback fig. 5.7.
• Explanations are not evidence of mechanism. Output-consistent explanations were produced even for wrong answers fig. 6.8. One explanation listed professions said to default to the feminine in Portuguese, but the model's own translation of one of them ("teacher") came out masculine fig. 6.11. Test explanations with editing or override experiments before trusting them §6.2.
• The AGI framing is an interpretation. The authors call their method subjective and informal §1.1, and the paper does not separate failures caused by the RLHF stage from failures inherent in the architecture §10.2.

Lineage

• Builds on: OpenAI's GPT-4 technical report, which the authors defer to for definitive benchmark numbers §3. It reuses HumanEval, GSM8K, MATH, TruthfulQA, TAB and TextWorld as test material §3.1.1 §4.2 §7.1 §7.2 §5.2.2.
• Compared against: ChatGPT, text-davinci-003, text-davinci-002 ("GPT-3"), Codex (code-davinci-002), CODEGEN-16B, Minerva and Microsoft Presidio tab. 1 tab. 3 tab. 5.
• Related ideas the paper uses: step-by-step prompting (Wei et al., chain-of-thought) for the arithmetic fixes §8.1 and to prompt the judge §7.2.2; language patches (Murty et al.) for the concept-override test fig. 6.10.