A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis and Few-Shot Learning at Human Level

In plain words

Language models trained only on ordinary text have had trouble with maths problems. This paper asks a model that was also trained on code, OpenAI's Codex, to answer each question by writing a short Python program instead of writing out an answer. The program is then run, and whatever it prints or returns is the answer. The model can also explain its own program line by line and write new practice questions.

Showing the model a few questions it already solved, chosen because they look most like the new one, lifts the rate further. Across 175 questions from seven MIT and Columbia courses, this reaches about 81% correct. A text-only GPT-3 model scored 18.8% on the same questions without examples, and 30.8% with step-by-step prompting. About a fifth of the questions needed a person to reword them or steer the output, so the pipeline is not fully hands-off. The results are the authors' own.

TL;DR

• Codex (text pre-trained, code fine-tuned) writes a Python program per question; running it gives the answer § Methods (Workflow).
• Zero-shot solves about 71% of 175 course questions; retrying the unsolved ones with up to five solved same-course examples gives 81% fig. 4.
• On 90 questions from six MATH topics, zero-shot gives 72.2% and few-shot 81.1%; GPT-3 zero-shot gives 25.5% fig. 4.
• Limits: image inputs, proofs and computationally intractable problems are out of reach § Results (Types of Problems the Model Cannot Solve).
• The release code shows the "automatic" labels are typed in by a person, and that the GPT-3 chain-of-thought baseline differs from the paper (see Paper vs code).

Problem

Text-only pre-training did not give transformers the ability to solve maths problems. GPT-3 with few-shot prompting and chain-of-thought (CoT) still failed on university-level course questions and on the MATH benchmark § Introduction (Related Work). The previous best automatic accuracy on the MATH benchmark was 8.8% § Methods (Dataset). The closest prior work, which predicts expression trees, was limited to numeric answers and one course, and the paper says it did not generalise to other courses § Introduction (Related Work).

Method

The approach needs no training. It prompts a fixed model and runs the code the model writes. The paper has no numbered equations, so the citations below use sections, tables, figures and the code.

1. Data. Seven courses (18.01, 18.02, 18.03, 18.05, 18.06, 6.042 and COMS3251), 25 questions each, plus six MATH topics, 15 questions each. The total is 265 questions § Methods (Dataset) § Results (Questions Solved). Example questions and solutions are in tab. 1.
2. Zero-shot prompt. The question is put inside a Python docstring with the instruction "write a program", followed by the question § Methods (Programming Language Context). The code adds "that answers the following question:" to the instruction zero-shot.py zero-shot.py.
3. Library context. For some questions the prompt also names a package, such as SymPy, or asks for a simulation. The paper describes this as automatic. The released zero-shot run uses only the "write a program" instruction § Methods (Library Context) zero-shot.py zero-shot.py.
4. Decode. Codex is called with temperature 0 and top-p 0, which the paper says is meant to make results reproducible § Implementation Details. The paper sets the frequency and presence penalties to 0 and passes no stop sequences § Implementation Details; the code does not set these, so the API defaults apply zero-shot.py.
5. Execute and mark. The program is run in a Python interpreter. An answer counts as correct if the printed output or the returned value is the reference answer § Implementation Details. In the code, the 1/0 "Zero-Shot Evaluation" column is filled in by a person after running each program README.md.
6. Neighbours. For each question, the other questions of the same course are ranked by cosine similarity of their text embeddings. The embedding model is text-similarity-babbage-001, which gives 2,048 dimensions § Methods (Automatic Zero-Shot and Few-Shot Learning) embedding.py embedding.py.
7. Few-shot retry. Only questions that failed zero-shot are retried. The prompt holds up to five solved neighbours, most similar first, each as its prompt followed by its program. The target's own prompt comes last § Methods (Automatic Zero-Shot and Few-Shot Learning) few-shot.py.
8. Manual remainder. About 19% of questions needed manual work: tidying the wording, or repeatedly prompting until a plot looked right § Methods (Manual Prompt Modification).
9. Explain. The question, the program and a fixed suffix are given to Codex. The suffix asks for a numbered, line-by-line explanation, with a 150-token limit § Methods (Automatic Explanation) zero-shot.py.
10. Generate questions. Codex gets a numbered list of 25 solved human questions, cut off at a random point, and writes the next one § Results (Automatically Generating New Questions) § Methods (Question Generation and their Human Evaluation). The generation code is not in the repository.
11. Survey. Students rate 60 questions: five human-written and five generated for each of six MIT courses. For each one they say whether it was human-written or machine-generated, whether it suits the course, and its difficulty from 1 to 5 § Methods (Question Generation and their Human Evaluation).

Few-shot selection in one block:

# Few-shot retry for one question  [§ Methods] (Automatic Zero-Shot and Few-Shot Learning)
# Input:  target (index), ranked_neighbours (same course, most similar first)
# Output: prompt string for the second Codex call

def build_few_shot_prompt(target, ranked_neighbours, solved, prompts, programs):
    # Zero-shot-correct questions are not retried  [code: code/few-shot.py#L51]
    examples = ''
    budget = 5  # up to five examples  [code: code/few-shot.py#L23]
    # Walk from the most similar neighbour down  [code: code/few-shot.py#L60]
    for other in ranked_neighbours:
        if solved[other] and budget > 0:
            # Prompt, then program  [code: code/few-shot.py#L63-L64]
            examples = examples + prompts[other] + programs[other] + '\n\n'
            budget = budget - 1
    # The target's own prompt goes last  [code: code/few-shot.py#L66]
    return examples + prompts[target]

Chain-of-thought baseline for GPT-3: the paper appends "Let's think step by step." after the question § Implementation Details. The released zero-shot code does this for GPT-3 zero-shot.py. The released few-shot code instead gives one example, the neighbour's question and reference solution, and then the target and the same phrase few-shot.py. See Paper vs code.

Hyperparameters

Results

Author-reported. Setup: 175 questions from seven courses (25 each), 90 questions from six MATH topics (15 each), the GPT-3 and Codex engines named in the Hyperparameters table, deterministic decoding § Results (Questions Solved) § Implementation Details. Baselines are GPT-3 with zero-shot and with chain-of-thought fig. 4.

Per-course rates from the release data (25 questions each, so steps of 4%). The CSV rounds the totals to 19%, 31%, 71% and 81% figure4-mit-math-solve-rates.csv fig. 4.

Other results:

• Solved by which route. The paper says 213 of the 265 questions were solved automatically § Results (Questions Solved). The release labels 216 as automatic (190 zero-shot, 26 few-shot) and 49 as manual README.md.
• Embeddings. The 175 course questions cluster by course in a two-dimensional UMAP projection. Linear algebra questions from 18.06 and COMS3251 sit together; calculus courses form another group; probability and discrete maths form a third § Results (Visualization of Embedded Questions) fig. 8.
• Generated questions. The authors report 130 generated questions in the supplement, ten per course and per MATH topic § Results (Automatically Generating New Questions). tab. 2 gives one example per course or topic, with its most similar human question and a similarity score from 0.12 to 0.99.
• Student survey. Fifteen students rated all 60 questions. The paper reports that generated and human questions were rated similarly hard within the confidence intervals. Human questions were rated slightly more appropriate for their course. Human questions were more often identified as human-written; generated questions were identified about equally often as either. The plotted values are in fig. 7 and not given in the text § Results (Student Survey Results).
• Human-level claim. The paper describes this as human level. Its comparison is with typical student performance on the same problem sets, which the paper does not measure directly § Results (Human Level).

Paper vs code

Official code found at the paper's footnote, github.com/idrori/mathQ § Implementation Details. The owner is the first author. The MIT licence (copyright Iddo Drori, 2022) covers the repository, and the commit below is the one read. Discrepancies:

• Automatic versus manual labels. The paper calls the solve rates automatic. The code expects a person to run each program and enter 1 or 0 in the "Zero-Shot Evaluation" column README.md README.md. Few-shot selection depends on those labels few-shot.py.
• Zero-shot prefix. The paper says the instruction is "write a program" § Methods (Programming Language Context). The code adds "that answers the following question:" zero-shot.py zero-shot.py.
• Library context not applied. The paper adds package hints such as SymPy and simulation automatically § Methods (Library Context) fig. 3. The zero-shot code defines these hints but uses only the first one zero-shot.py zero-shot.py.
• GPT-3 chain-of-thought baseline. The paper describes few-shot examples before the chain-of-thought phrase § Implementation Details. The released few-shot code uses one example, the most similar question, and its reference solution, for every question, solved or not few-shot.py. The reported 30.8% and 42.2% should be read against this code.
• Explanation wording. The paper's suffix reads "Here is what the above code is doing" § Methods (Automatic Explanation). The code reads "Here's what the above code is doing" zero-shot.py.
• Sampling. The paper says penalties are 0 and no stop sequences are used § Implementation Details. The code sets neither zero-shot.py.
• Question sampling. The paper says questions were sampled at random § Methods (Dataset). The release has files 01 to 25 for each course, and the code reads those; the sampling step is not in the code zero-shot.py.
• Few-shot count in the data. The paper and the code cap examples at five. Two released records list "Automatic Few-shot (6)" README.md. Count taken with grep over the "Solution type" fields.
• Generation. The paper gives temperature and top-p 0.1 for new questions § Implementation Details. The generation script is not in the repository; only zero-shot and few-shot scripts are.
• Plot answers. The paper says plot answers were judged manually § Methods (Manual Prompt Modification). The code's comparison covers printed and returned values only § Implementation Details.

The embedding model, engine names, temperature and top-p, and the few-shot prompt layout match the paper.

Pitfalls

• Labels are human. "Automatic" in the paper means the program ran and a person compared its output with the reference. Reproduce by running programs and recording 1 or 0; do not expect a script to do it.
• Same-pool retrieval. Few-shot examples come from the same 25 (or 15) questions being tested. Each target excludes itself, but its neighbours' solutions are visible to it. This is transductive, not held-out.
• Few examples may be none. If no neighbour is solved, the few-shot prompt is just the target's prompt few-shot.py.
• Ties. The neighbour code stores scores in a dictionary keyed by score embedding.py. Two questions with identical scores overwrite each other and lose an index.
• Ordering. The examples are placed most similar first, so the nearest one sits furthest from the target, which the paper does not state few-shot.py.
• Engine access. The results depend on access to the named engines (code-davinci-002 and text-davinci-002) at temperature 0. The paper does not state how long they were available.
• Generated code runs on your machine. The method executes model-written Python. Run it in a sandbox.
• Contamination. The authors say COMS3251 was not online and was unseen when Codex was trained § Methods (Dataset). The paper gives no overlap check for the other courses or for the MATH benchmark.
• Hidden manual work. About 19% of questions needed manual tidying or prompting § Methods (Manual Prompt Modification). The rate of manual intervention is not broken down by course in the text.
• Limits. Image inputs, proofs and intractable problems are out of scope § Results (Types of Problems the Model Cannot Solve). Figure-based questions were excluded from the sample fig. 2.
• Course question text. The repository is MIT-licensed. The paper does not say whether course question texts carry their own rights. This card does not copy them.

Lineage

• Builds on: Codex, a model first trained on text and then on code, cited as chen2021evaluating § Introduction. Chain-of-thought prompting, cited as kojima2022stepbystep § Methods (Implementation Details).
• Compared against: GPT-3 text-davinci-002, zero-shot and with chain-of-thought fig. 4. The expression-tree and graph-network approach for machine-learning course questions, cited as tran2021solving, which reports up to 95% on that course § Introduction (Related Work). The MATH benchmark, cited as MATH, where the previous best was 8.8% § Methods (Dataset).
• Superseded by: not stated in the paper or the repository.