A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers

In plain words

Math word problems ("Tim bought 2 sandwiches and 4 cups of pudding...") are a common way to test whether a program understands language and arithmetic. The authors noticed that many existing collections of these problems repeat the same wording over and over, so a program can score well by matching a familiar pattern without really reading the problem.

They built a new collection of 2,305 English elementary-school problems. Each one is labelled with the arithmetic idea it needs, the school grade it fits, and its answer. They also wrote a score that measures how similar the problems in a collection are to each other, so a corpus can be checked for variety before anyone trusts results on it.

When older solvers were run on a less varied subset of another collection, they did well. On the new corpus they did much worse, which suggests the earlier scores overstated how well these programs solve real word problems.

TL;DR

ASDiv is a 2,305-problem English math word problem corpus. Each problem has a problem type, a grade level (G1 to G6), an equation and an answer. A BLEU-based lexicon usage diversity score (LD, averaged over the corpus to CLD) shows ASDiv is more varied than the corpora it is compared with. Three pre-LLM solvers reach about 36% accuracy on ASDiv, against 86% for one solver on the less diverse MathQA-C subset tab. 3.

Problem

Word-problem benchmarks are used to measure language understanding in math, but a corpus can be dominated by near-identical problems, so an answer can be found from an existing template without understanding the text §1. The authors also observe that large crowd-built corpora can contain inconsistent equations and answers §1 §3.2. They argue that low-diversity corpora tend to give over-optimistic results, and that variety matters more than volume §1. Figure 1 shows the share of near-duplicate problems per corpus fig. 1.

Table 2 compares the corpora by category, annotation and size tab. 2. The authors argue that a corpus of this kind needs consistent annotations, a measure of lexical variety, and problem types and difficulty labels §1 §3.

Method

The contribution has three parts: a corpus, a diversity metric and an evaluation. The corpus is built as follows §3.3.

1. Collection. Problems were gathered from 28 websites. Any problem that was highly similar to an existing one under the LD metric was pruned or rewritten. The similarity threshold is not stated §3.3. This produced 2,305 problems.
2. Annotation. One master's-degree research assistant labelled each problem with its problem type, equation, answer and grade level §3.3. If the source already gave equations, answers or grade levels, those were reused: 22.6% of source problems had equations and answers, 52% had answers only, and 63.5% had a grade level §3.3.
3. Consistency check. Each annotated equation is solved with SymPy and the result is compared with the labelled answer. Inconsistent problems were annotated anew and re-checked. A random sample of 480 problems (20 per problem type) was then checked for annotation correctness, and all were correct §3.3. For the MathQA arithmetic subset this check found 27% of the annotated equations did not match their labelled answers §3.2.
4. Problem types. The corpus uses 24 problem types in three groups: basic arithmetic operations (14 types), aggregative operations (6 types) and problems needing extra domain knowledge (4 types) §2 tab. 5. Examples are in tab. 5.
5. Grade level. Each problem is labelled with one of six grade levels, G1 to G6, based on a grade classification reference §3.3 fig. 2.

The diversity metric is the lexicon usage diversity (LD) of each problem, averaged over a corpus to give the corpus lexicon diversity (CLD) §3.1. Per problem P_i in a corpus P, with M problems, the authors define:

LD_i = 1 - \max_{j \neq i} \frac{BLEU(P_i, P_j) + BLEU(P_j, P_i)}{2} §3.1

The text is normalised before scoring: tokenisation and POS tagging with Stanford CoreNLP, lemmatisation with NLTK, removal of stop words, and replacement of person names and quantity values with meta symbols §3.1. The same problem-level values are used to compare corpora in fig. 1.

# Lexicon usage diversity of each problem and the corpus CLD  [§3.1]
# Input:  problems: list of M normalised token sequences
# Output: corpus_diversity: scalar in [0, 1]

def corpus_lexicon_diversity(problems):
    # Score each problem against its most similar other problem  [§3.1]
    diversities = []
    for i in range(len(problems)):
        most_similar = max(
            (bleu(problems[i], problems[j])
             + bleu(problems[j], problems[i])) / 2
            for j in range(len(problems)) if j != i
        )
        diversities.append(1 - most_similar)
    # Corpus score is the mean of the per-problem scores  [§3.1]
    return mean(diversities)

The evaluation trains no new model. It runs three existing solvers on three data sets §4:

1. Subsets. From MathQA, 4,117 problems were collected. Those with only arithmetic operations, no external constants and no reused operands were kept, and inconsistent ones removed, leaving 3,000 problems, called MathQA-C §4. From ASDiv, 1,218 arithmetic problems were taken, called ASDiv-A §4.
2. Folds. MathQA-C uses 5-fold cross-validation. For ASDiv-A and ASDiv, the problems within each problem type are assigned at random to five folds of near-equal size §4.
3. Systems. Two statistical solvers (LCA++ and UnitDep) and one neural solver (GTS) are evaluated §4. Only GTS is described as repeated: five runs, averaged §4. The paper does not restate the training settings of any solver.

Hyperparameters

Results

Author-reported. Setup: English math word problems; systems are LCA++ (L), UnitDep (U) and GTS (G); accuracy is the 5-fold cross-validated answer accuracy §4.

Corpus diversity fig. 1 (mean LD and standard deviation per corpus):

Share of problems with LD = 0 (identical under the metric): about 85% for MathQA, 28% for IL, 22% for AI2, 20% for ALGES, none for ASDiv §3.1.

Accuracy by corpus and system tab. 3. "-" means the system failed on that corpus. "#" means the result is significantly lower than MathQA-C (p < 0.01):

Accuracy by grade level on ASDiv tab. 4:

Corpus comparison tab. 2: ASDiv has 2,305 problems, with arithmetic and algebraic problems plus domain knowledge, and is annotated with equation, answer and grade level and problem type. MathQA has 37K problems with decomposed-linear-formula annotation and multiple-choice options. AQuA has 100K problems with rationale annotations tab. 2.

Paper vs code

No official code. The paper gives a dataset URL in footnote 1 on page 2: https://github.com/chao-chun/nlu-asdiv-dataset p. 2 fn. 1. That URL returned "repository not found" on clone and HTTP 404 on fetch. A GitHub repository search found chaochun/nlu-asdiv-dataset, but its owner name is not shown and nothing verifies that it belongs to the authors, so it is not used. No license or commit is recorded.

Within the paper, the consistency check is described for MathQA in §3.2 and for the corpus in §3.3. The text does not say whether the SymPy check covered all 2,305 problems or only the arithmetic subset.

Pitfalls

• The metric is not fully specified. The stop-word list, the meta-symbol rules, the lemmatiser setting, and BLEU smoothing and weights are not given §3.1. CLD values such as 0.49 and 0.05 may not be reproducible from the text alone.
• Max over j. LD takes the single most similar other problem, so one near-duplicate sets that problem's LD to near 0. CLD is a mean, so a few duplicates in a large corpus may move it little. The paper does not discuss how BLEU behaves when a short problem has no matching n-grams.
• Quadratic cost. Each problem is compared with every other, which is O(M²) BLEU computations. The paper gives no runtime.
• Histogram edges. fig. 1 bins LD into 20 intervals on [0, 1] §3.4. How the exact value 0 is assigned to a bin is not stated, and the "85%" in §3.2 and "82%" in §4 refer to different subsets (MathQA and MathQA-C).
• Subsets are not the same corpus. ASDiv-A (1,218) is a subset of ASDiv, so those two rows measure overlapping problems. MathQA-C (3,000) comes from a different corpus with its own filters, so its row is not directly comparable with the ASDiv rows.
• Single annotator. One research assistant labelled every problem §3.3. No inter-annotator agreement is reported. The authors argue that automatic equation checking gives more objective labels §3.2, but this covers only the equation and answer, not the problem type or grade level.
• Grade levels are partly inferred. Only 63.5% of source problems had a grade level, so the rest were assigned by the authors using a grade reference in footnote 5 §3.3. The grade-level results tab. 4 depend on that labelling.
• Significance is not explained. "#" in tab. 3 reports p < 0.01, but the test is not named. Grade-level differences in §4 (grades 5 and 6 are "significantly lower") come with no test described.
• Cross-validation folds. ASDiv folds are random per problem type. The seed and fold assignment are not stated, so the splits cannot be reproduced from the text.
• Solvers are older. Two of the three solvers are statistical and one is neural §4. The paper does not test language models on ASDiv, so the results say nothing about them.
• Appendix. Problem-type examples tab. 5, inconsistent MathQA examples tab. 6 and the MathQA sentence-pattern figure fig. 3 are in Appendices A and B. They are in the arXiv PDF on pages 7 and 8.

Lineage

• Builds on: the MathQA corpus (Amini et al., 2019), from which the MathQA-C subset is drawn §3.2 §4. The other corpora in tab. 2 are compared, not built on.
• Compared against: Dolphin1878, AI2, IL, AllArith, KAZB, ALGES, DRAW, Dolphin18K, AQuA and MathQA tab. 2 fig. 1.
• Solvers evaluated: LCA++ (Roy and Roth, 2015), UnitDep (Roy and Roth, 2017), and GTS (Xie and Sun, 2019) §4.
• Superseded by: not known from content in this wiki.