Evaluating the Performance of Large Language Models on GAOKAO Benchmark

In plain words

Chinese students take a national exam at the end of high school, the gaokao. Its questions cover maths, physics, chemistry, biology, Chinese, English, history, geography and politics, and include multiple choice as well as essays and short written answers. This paper takes nearly 2,800 real questions from the exams given between 2010 and 2022 and uses them as a test for large language models.

The models answer each question without any worked examples in the prompt. The multiple-choice answers are checked automatically. The written answers are marked with the official teacher marking points: first by people, and then by a strong model acting as the marker. The result is a score out of the same total as a real student's, so a model can be read as "this would pass" or "this would not". The authors found that the best model reaches a competitive score, that scores vary a lot from subject to subject, and that a model marker agrees with human teachers well enough to be useful, but not perfectly.

TL;DR

• A benchmark of 2,811 gaokao questions (1,781 objective, 1,030 subjective) across 10 subjects, 2010-2022 §2.2 tab. 3.
• Zero-shot prompting, temperature 0.3 for the tested models §3.1; objective answers are matched by rule, and subjective answers are scored by human teachers or by GPT-4-turbo as judge §3.2-§3.4.
• GPT-4 exceeds 400 converted points out of 750 §3.5 fig. 1; the judge's question-level correlation with teachers is 0.825-0.864 Spearman tab. 4.
• All results are author-reported; the benchmark questions are the exam's, so contamination is possible §5.

Problem

Existing Chinese and general benchmarks (for example C-Eval and AGIEval) mostly use multiple-choice items, and the authors argue that they lack subjective questions that test generation §1. Human grading of generated answers is expensive, so the paper also asks whether an LLM can grade subjective answers reliably §1 §3.4. The authors state that a zero-shot protocol is used to match how humans take exams §3.1. They note that benchmark leakage is plausible because the questions may appear in training corpora §5.

Method

Dataset. Questions from all national gaokao papers for nine subjects from 2010 to 2022 were converted from PDF to JSON with scripts and manual checking; maths formulas were converted to LaTeX §2.2. The paper says 2,811 questions in total: 1,781 objective and 1,030 subjective §1 §2.2 tab. 3. The question types are listed in tab. 3 and, per question, in tab. 8.

Converted total score. Each subject carries a fixed total in the gaokao (sciences and humanities each total 750 points) §3.1 tab. 7. Per subject, the model's objective scoring rate and subjective scoring rate are each multiplied by that part's points, and the parts are summed over a stream:

S_{total,\mathcal{S}} = \sum_{i \in \mathcal{S}} \left( R_{i,obj} \cdot M_{i,obj} + R_{i,subj} \cdot M_{i,subj} \right)

S_{total,\mathcal{H}} = \sum_{i \in \mathcal{H}} \left( R_{i,obj} \cdot M_{i,obj} + R_{i,subj} \cdot M_{i,subj} \right)

Here S is the set of science subjects, H the set of humanities subjects, R the scoring rate and M the points available for that part §3.1 (the equations are unnumbered in the paper). Per-subject points are in tab. 7.

Prompts. Each question type has a prompt that asks for the reasoning and then the answer in marked fields. The prompt both asks the model to do the task and specifies the output format §3.1. The example prompts are in app. A.1 to app. A.3. The prompts shipped with the code are per subject and type Obj_Prompt.json Sub_Prompt.json.

Objective questions. The input is the question. The output is reasoning plus an outcome. A point is awarded only when the outcome matches the standard answer §3.2. The authors state they follow the regular-matching scoring of the GPT-4 technical report §3.2.

Subjective questions. Human scoring: each subjective question is marked by two teachers and the average is the final score §3.3.

LLM as a judge. Teachers wrote detailed marking criteria, which break each answer into scoring points §3.4. GPT-4-turbo (GPT-4-1106-preview) is prompted zero-shot with the question, the analysis, the standard answer, the points, the marking criteria and the model answer; it outputs a grading process and a final score §3.4. Temperature is 0 §3.4. The judge's converted totals and its question-level Spearman and Kendall-Tau correlations with human scores are reported §3.4 tab. 4.

# Objective scoring  [§3.2] [tab. 1]
# Input:  per question a standard answer and points per blank
# Output: scoring rate per subject

def objective_rate(answers, standard, points):
    # Extract the chosen letters from the model output  [§3.2]
    chosen = extract_letters(model_output)
    # Award points only where the answer matches  [§3.2]
    earned = points * count_matches(chosen, standard)
    return earned / (points * len(standard))

# LLM-as-judge grading  [§3.4] [tab. 4]
# Input:  question, analysis, standard answer, points, marking criteria
# Output: judge score for one subjective answer

def judge_score(question, criteria, model_output, points):
    # Ask the judge for a reasoned grade with a final total  [§3.4]
    grade_text = judge_gpt4_turbo(prompt, inputs, temperature=0)
    # The final total must be a single number within the points  [code]
    return parse_total(grade_text), points

Hyperparameters

Tested models §3.1 tab. 1: GPT-4-0613, GPT-4-0314, GPT-3.5-turbo-0301, ERNIE-Bot-0615, ERNIE-Bot-turbo-0725 (closed); LLaMA-7b, Vicuna-7b, Baichuan2-7b-Base, Baichuan2-7b-Chat, Baichuan2-13b-Chat, ChatGLM-6b and ChatGLM2-6b (open) tab. 1.

Results

Author-reported. Setup: zero-shot, temperature 0.3, 2010-2022 gaokao questions, human teachers for subjective questions except where noted §3. Full tables follow.

Headline results:

• GPT-4 scores above 400 converted points §3.5. Per fig. 1 (values as printed on its bars), GPT-4-0314 reaches 485 humanities and 447 sciences out of 750; GPT-4-0613 reaches 480 and 434 fig. 1.
• Objective scoring rate, GPT-4-0314: 72.2% overall tab. 1.
• Subjective scoring rate, GPT-4-0314: 51.9% overall, human-scored tab. 2.
• Judge vs human, question-level: Spearman 0.825-0.864, Kendall-Tau 0.685-0.740 across five models tab. 4.
• Marking criteria matter for the judge: with criteria, GPT-4-0613 reaches Spearman 0.854 vs 0.820 without tab. 6.
• Scores are stable across years 2013-2022 §4.3 fig. 3.
• Judge deviation from human converted totals: under 2% for sciences and about 5% for humanities §4.4.

Table 1: Scoring rate of objective questions tab. 1. Percent.

Open-source models sit above the line in the paper; closed-source models below it tab. 1.

Table 2: Scoring rate of subjective questions, human-scored tab. 2. Percent.

Table 3: Distribution of question types tab. 3.

Table 4: Converted total score and judge correlation, human vs GPT-4-turbo tab. 4.

Table 5: Objective scoring rate on GAOKAO-Bench vs GAOKAO-Bench-2023 tab. 5. Percent.

The paper says the 2023 variation is within normal difficulty fluctuation §5.

Table 6: Effect of marking criteria on the judge tab. 6.

Table 7: Converted total score per subject tab. 7. Points.

Table 8: Statistics of question types tab. 8. Upper block objective, lower block subjective. "Avg. Length" as printed.

Figures, described and cited, not reproduced fig. 1 fig. 2 fig. 3:

• fig. 1: bar chart of converted totals for five models, humanities and sciences, with the values on the bars (listed in Results).
• fig. 2: radar chart of objective and subjective scoring rates across subjects.
• fig. 3: line chart of converted totals per year, 2013-2022, showing the trend is flat §4.3.

Paper vs code

Official repository: github.com/OpenLMLab/GAOKAO-Bench, linked from the paper's abstract footnote §abstract. Commit 6dbb24f8d8439041e5431c4c184a582182a6ce9c, Apache-2.0 LICENSE. The repository ships benchmark data, scripts and the model-graded results for GPT-4-0314 and 2010-2022 objective scores.

• Physics partial credit. The paper says a point is awarded only when the outcome matches §3.2. The code gives partial credit on multi-answer physics items: 6 points per fully correct blank, 3 for a partly correct one, 0 otherwise OBJ_score_evaluation.py.
• Judge retries and a dead assertion. The paper does not describe retries. The code retries the judge up to three times until exactly one total no greater than the item's points is found bench_function.py. The check assert count < 3 at bench_function.py can never fail, because count is 2 when the loop ends. After three failures the last output is kept and used, even with no total or a total above the maximum.
• Judge score denominators. A subjective item with no valid judge score is left out of the score total but still counted in the question count SUB_score_evaluation.py. Scores above the item's points are not capped there.
• Human scores not shipped. The paper's subjective figures rely on two teachers per question §3.3. The repository's subjective results contain only the model-graded folder (gpt-4-1106-preview_correction_wo_marking_criterion). The human scoring path reads a correction_score field that the shipped data does not include SUB_score_evaluation.py.
• Marking criteria not shipped. The paper's judge with marking criteria §3.4 needs a criteria directory. The shipped driver sets w_marking_criterion = False, and the data folders contain no criteria file subjective_grade.py. The with-criteria numbers in tab. 6 cannot be reproduced from the repository alone.
• Single-choice extraction. The code takes the last capital letter A-D in the output, scanning from the end bench_function.py. The paper says only "regular matching" §3.2. Any A-D letter after the answer changes the result.
• Multi-question choice. If the number of 【答案】 markers matches the number of blanks, each marker gives one answer; otherwise the code takes the first letters in the output bench_function.py.
• Five out of seven. The first five letters A-G in the output are taken, in order bench_function.py.
• Temperature and model names. Temperature 0.3 for tested models openai_gpt4.py and 0 for the judge subjective_grade.py match the paper.
• Paper vs shipped totals. The README reports humanities and sciences totals of 485 and 447 for GPT-4 README_EN.md. The shipped Results/merge_score.json for gpt-4-0314 with model grading gives 548.3 and 466.1 merge_score.json. These files come from different subjective-scoring runs; I did not reconcile them.
• Subject weights. The per-subject objective and subjective weights in the code match tab. 7 (for example Chinese 0.3/0.7 of 150, Physics 0.4/0.6 of 110) merge_OBJ_SUB_score.py.

Pitfalls

• Typo in the paper's data section. §2.2 says "1030 objective questions and 1781 objective questions". §1 and tab. 3 give 1,781 objective and 1,030 subjective. Use the table.
• Equations are unnumbered. The converted-total equations have no number in the paper, so cite them as §3.1.
• Paper vs repo on partial credit. Physics multi-answer items get partial credit in the code and none in the text. Reimplement the code's rule if you want to match the released numbers, and say so.
• Subject and stream totals. Sciences and humanities both total 750 only if all six subjects in each stream are counted with the tab. 7 points. Chinese and English count in both streams.
• Judge tolerance. The paper reports science deviations under 2% and humanities deviations around 5% of total §4.4. Humanities answers depend on matching semantics to marking points, which the paper calls a significant challenge §4.4.
• Question-level correlation is not total-score agreement. The judge's ρ and τ are question-level tab. 4; the converted totals are close but not identical to human totals.
• Costly API loop. The client retries failed requests forever with a 2-second pause openai_gpt4.py. A quota error will block a run indefinitely.
• Benchmark leakage. The paper accepts that the gaokao questions may be in pre-training data §5. The 2023 follow-up set reduces the score for some models, but not uniformly tab. 5.
• Snapshot-bound. Results reflect the model snapshots named in §3.1 (for example GPT-4-0613, GPT-4-0314); later snapshots are not covered.
• Multi-choice answers. Physics multi-choice items accept several letters; the code joins every A-D letter in the trailing text, so stray capital letters can add answers bench_function.py.

Lineage

• Related benchmarks named by the authors, not compared on the same items: MMLU, BIG-Bench, AGIEval, C-Eval, CMMLU, HELM §1 §7.
• The judge correlation metrics follow two cited works §3.4; the grading protocol itself is the authors' own.
• Successor data: GAOKAO-Bench-2023 (objective questions only) §5 tab. 5; the README points to a GAOKAO-Bench-Updates repository for 2023-and-later multiple-choice items README_EN.md.