Solving Quantitative Reasoning Problems with Language Models

In plain words

Language models are good at writing text but often fail at school-level maths and science, where the answer depends on getting every calculation right. The authors take a general language model and train it further on a collection of technical text: scientific papers and web pages with the formulas kept intact, so the model sees how equations are written. They then ask it to write out its working step by step, not just give a number.

The model is not allowed to use a calculator or a programming tool. To improve its answers, the authors let it try the same question many times and keep the answer it gives most often. Each attempt is judged only by its final answer. The model's working is not checked. The result is a model that solves many written problems correctly, but still misses many of them.

TL;DR

• Minerva is PaLM (8B, 62B, 540B) trained further on a maths and science corpus that keeps LaTeX notation §2.1 §2.2.
• It answers with no tools, using few-shot chain-of-thought prompts §2.3.
• Majority voting over sampled solutions (maj1@k) gives the headline numbers, not single greedy answers §2.5.
• Author-reported: 50.3% on MATH and 78.5% on GSM8k at 540B with maj1@k tab. 3.
• Final answers are checked by string normalization plus SymPy equivalence; the reasoning is never checked automatically §2.3 app. D.1.

Problem

Language models struggled with quantitative reasoning: maths, science and engineering problems at college level §1. These tasks need the model to read the question, recall facts, carry out a sequence of calculations, and write exact symbols. Earlier work got good maths results from training on domain data or from writing code, which calls external libraries. The authors instead ask whether a model can reach the answer by its own reasoning, with no external tools §1 §1.1.

Method

The method has three parts: a continued-training corpus, further training of PaLM checkpoints on it, and an inference procedure that votes over samples.

1. Training corpus. The technical dataset has 38.5B tokens §2.1 tab. 1:

• Math web pages: 47.5% of the data, 17.5B tokens. Pages were kept if their raw HTML contained <math or MathJax-Element- app. B.2. Pages were cleaned so that TeX and AsciiMath formulas survive, while MathML-only pages (about 20% of documents) were discarded app. B.2.
• arXiv papers: 47.5% of the data, 21.0B tokens. The source is 2M papers up to February 2021 in LaTeX. Comments are removed, and papers are dropped if they exceed 75k tokens, average more than 0.6 tokens per character, have no section headers, or end up empty app. B.1.
• General natural language data: 5% of the data, more than 100B tokens, which is the same set used to pretrain PaLM §2.1 tab. 1.

2. Continued training. Each starting checkpoint is a decoder-only PaLM model §2.2. Training continues with an autoregressive objective on the technical dataset §2.2. Architecture and token counts are in tab. 2. The 540B model sees only 26B tokens and is described by the authors as highly undertrained compared with the smaller models §2.2.

3. Learning-rate schedule. The schedule is reciprocal square-root decay, continued from the PaLM pretraining schedule. The 8B model had 1M pretraining steps and 600k further steps. The 62B model had 520k and 400k. The 540B model had 257k and 383k app. C. At the end, the learning rate is cut 10x and each model trains for 4% more steps app. C. Training used the t5x framework on TPU v4 app. C.

4. Single-sample inference. Greedy decoding is used when one answer is sampled per problem §2.3.

5. Majority voting (maj1@k). Sample k solutions and return the most common final answer §2.5. Each solution is sampled with nucleus sampling at temperature 0.6 and p = 0.95 §2.3. The paper's pseudocode below is a summary of the procedure described in §2.5 and app. D.1; it is not a listing from the paper.

# maj1@k decoding  [§2.5] [§2.3]
# Input:  model; problem (text); k (number of samples)
# Output: one final answer (text)

def maj1_at_k(model, problem, k):
    # Draw k independent solutions at T=0.6, nucleus p=0.95  [§2.3]
    responses = [
        sample(model, problem, temperature=0.6, top_p=0.95)
        for _ in range(k)
    ]
    # Take the text after "Final Answer: The final answer is"  [app. D.1]
    answers = [extract_final_answer(response) for response in responses]
    # Normalize each answer with the listing in [app. D.1]
    answers = [normalize_final_answer(answer) for answer in answers]
    # Return the most common answer; tie-breaking is not stated  [§2.5]
    return most_common(answers)

Majority voting saturates faster than pass@k. The authors report that 97% of the large-k accuracy is reached at k = 64 for MATH and k = 16 for GSM8k §2.5. Their rough estimate of the sample count needed to recover the majority answer with 95% confidence is

k > 4\left(\frac{1}{p_1} - 1\right) app. E.4

where p_1 is the share of the most common answer. They also give a rough resolution limit on the gap between the top two answers, \frac{4}{(p_1 - p_2)^2} app. E.4. For k = 64 they estimate this resolution at about 0.25 app. E.4.

6. Answer scoring. The prompt asks for the sentence "Final Answer: The final answer is ANSWER. I hope it is correct." app. D.1. The answer is extracted from that sentence and passed through a string normalizer app. D.1 [lst. in app. D.1]. The two normalized answers are compared with SymPy's parse_latex, then simplify on their difference. A 5 s timeout counts as not equal app. D.1. A solution is correct only if its final answer matches; the reasoning is not checked §2.3.

7. Datasets and prompts.

• MATH (12K problems): a fixed 4-shot prompt with four training examples whose targets are not too long, chosen to keep most inputs under 1024 tokens §2.3 app. D.2.
• GSM8k: chain-of-thought prompt from Wei et al., no calculator §2.3.
• MMLU-STEM: 18 subtopics app. G. Standard 5-shot dev-set prompt for pass@1. For maj@k, a chain-of-thought prompt with k = 16 samples §2.3 app. G.
• OCWCourses (272 problems, 191 numeric, 81 symbolic): MIT OpenCourseWare problems rewritten by contractors to be self-contained with a clear final answer §2.4 app. F. Problems asking for proofs or open-ended answers were excluded §2.4.

Inputs are truncated from the left to 1024 tokens and at most 512 tokens are generated §2.3.

Key equations. The equations used in the appendix analysis of majority voting, reproduced with the authors' notation app. E.4:

p_1 - 2\sqrt{\frac{p_1(1 - p_1)}{k}} > 0 \;\Rightarrow\; k > 4\left(\frac{1}{p_1} - 1\right) app. E.4

Hyperparameters

Results

Author-reported. Setup: MATH (4-shot), GSM8k (chain-of-thought), MMLU-STEM and OCWCourses. Baselines are the un-tuned PaLM checkpoint, OpenAI davinci-002 at T = 0.2 §3, and the published best for each dataset, which the authors take from different papers and settings tab. 3.

Other reported results, all author-reported:

• MATH with majority voting at more samples, maj5@k: 47.6 (8B) and 64.9 (62B). GSM8k maj5@k: 56.8 (8B) and 89.0 (62B) tab. 7.
• Polish National Math Exam: 57% for 62B, and 65% for 540B, against a 2021 national average of 57% §3.
• Basic arithmetic at 540B: above 80% on 10-digit addition and above 20% on 18-digit addition §3.1 app. H.
• MATH with SymPy vs. exact match: SymPy adds about 1 point at each setting tab. 6.
• Log-likelihood reranking is much worse than majority voting on MATH with 62B: 23.8 vs. 42.0 tab. 8.
• Finetuning on MATH does not help Minerva 8B (12.9 to 13.4 across three finetuning prompts, vs. 14.1 few-shot). It does help PaLM 8B, from 1.5 few-shot to between 4.4 and 5.6 after finetuning tab. 9.
• Estimated false-positive rate of 8% on MATH for 62B greedy answers, rising with difficulty to 30% at level 5 §4.2 tab. 5.
• Estimated pass@256 for 62B: 84.5%, and above 68% after false positives are removed §4.2.
• Mistake types for 8B, against 201 problems solved by 62B only: incorrect reasoning 82, incorrect calculation 70 §4.1 tab. 4.

Paper vs code

No official code. The paper and arXiv metadata link no code repository. The only linked material is a supplementary zip of prompts app. G; it is not a code repository and was not used. Searches: GitHub repository search for "minerva" within the google-research organisation returned nothing. A search for "minerva math" and for the paper's title returned only third-party reimplementations (for example kyegomez/Minerva), which are not official and were not used. No model weights or training code are available from the paper.

Pitfalls

• No official code, weights or data release. The corpus is described only in outline: sources, date ranges and filters app. B. The filter rules are concrete, but the crawl and processing tools are not. The OCWCourses problems are said to be planned for release in a later manuscript §2.4, so this paper alone does not supply them.
• Unstated training details. The peak learning rate, warmup, optimizer, weight decay, gradient clipping and tokenizer are not stated. The exact step at which the learning rate is cut 10x is not stated.
• Step counts disagree. tab. 2 lists 624k, 416k and 399k steps. The appendix lists 600k, 400k and 383k. The table includes the final 4% of steps, and the appendix does not. Use the appendix counts plus 4% to match the table.
• Two different GSM8k numbers for 62B majority voting. tab. 3 gives 68.5. The appendix table with k = 100 gives 67.5 tab. 7. The paper does not explain the gap.
• Two different GSM8k numbers for the published best. tab. 3 gives 74.4 and tab. 7 gives 74.5.
• Sample counts differ by model size. MATH uses k = 256 for 8B and 62B but 64 for 540B. GSM8k uses 100 but 40 for 540B. Compare maj1@k rows only at the same k tab. 3.
• Greedy vs. sampled single answers. The 62B pass@1 is 26.5 with greedy decoding and 21.8 at T = 0.6 tab. 8. The headline pass@1 numbers are greedy. Majority voting uses T = 0.6.
• Answer normalization is lossy and string-based. The normalizer removes every occurrence of substrings such as a and spaces, and removes a fixed list of unit words anywhere in the string. It keeps only the part after the last =. Results depend on it. Port the listing exactly if you want comparable numbers app. D.1. The authors note that SymPy equivalence changes overall MATH accuracy by only about 1 point tab. 6.
• Majority grouping rule is underspecified. The paper says predictions are grouped by final answer §2.5, but does not say whether grouping uses the normalized string or SymPy equivalence. Tie-breaking is not stated. Your implementation must choose one.
• Timeouts count as wrong. A SymPy simplify call that exceeds 5 s is treated as not equal app. D.1. This can lower scores on slow expressions.
• Correctness checks only the final answer. Reasoning is not verified §2.3 §6.1. The false-positive estimates come from manual checks of 100 MATH questions, 20 per level §4.2. Treat them as rough.
• Fine-tuning ablation picks its checkpoint on the test set. The checkpoint with the best test accuracy after 50 steps is reported app. E.3. The numbers are optimistic.
• Memorization analysis is narrow. It covers MATH only §5. Its checks are a BLEU search over 500-character chunks, manual inspection of 250 documents, and 20 hand-modified problems §5.1 §5.2. It concludes "little evidence" of memorization §5.
• Baselines are not like-for-like. The davinci-002 baseline uses OpenAI's API at T = 0.2, and the published-best column mixes settings from three different papers tab. 3 §3. The authors say direct comparison is difficult §1.2.
• MathML-only pages are dropped. About 20% of math web pages were discarded app. B.2. Maths written only as MathML is underrepresented in training.
• 540B is undertrained. It saw only 26B tokens of continued training §2.2. Its gains over smaller models are therefore not from long training.
• Prompt length limits. Inputs are cut from the left to 1024 tokens §2.3. The 4-shot prompt was chosen so most problems fit that limit app. D.2. Longer problems are truncated, which hurts accuracy in ways the paper does not measure.

Lineage

• Builds on: PaLM, whose decoder-only checkpoints are the starting point §2.2.
• Builds on: majority voting over sampled chains (the paper's maj1@k, following §2.5). The paper cites it as Wang et al.
• Builds on: chain-of-thought prompting, Wei et al., arXiv:2201.11903 §2.3.
• Evaluated on: MATH, arXiv:2103.03874; GSM8k, arXiv:2110.14168; MMLU, the STEM subset §2.3.
• Compared against: OpenAI davinci-002 at T = 0.2, and the published-best values for each dataset tab. 3.
• Superseded: not stated in the wiki's content.