Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora

In plain words

Children learn to speak from far less text than language models see. By age 13 a child has heard fewer than 100 million words, while today's large models train on trillions. The BabyLM Challenge asked researchers to see how well a language model can learn when the training text is capped at roughly the amount a child hears.

Teams were free to change the model, the training recipe, the order of the text, or the way the text is prepared. Every entry was then tested on grammar, on a set of ordinary language-understanding tasks, and on a few tests that were kept secret until the end. This paper is the organisers' report on that contest.

The clearest winner was a small encoder model built on an earlier design called LTG-BERT. Many entries tried to make training easier by feeding the model examples in a sensible order, and most of those did not help. The paper does not propose a new model. Its value is the shared rules, the test suite, and a summary of what worked.

TL;DR

• The first BabyLM Challenge (2023) capped pretraining text at 100M English words, or 10M in its smallest track. Models were scored on grammar, GLUE, the Mixed Signals Generalization Set (MSGS), and a hidden grammar supplement §1 §3 §5.
• 31 papers and 162 models were submitted. The top Strict and Strict-Small entry is ELC-BERT (aggregate 0.74 and 0.66), ahead of a 0.60 and 0.50 OPT-125M baseline tab. 2.
• The most popular approach, curriculum learning, mostly did not help. Architecture changes, training-objective changes, data preprocessing and teacher–student distillation produced the strongest gains §7.4.
• This is a protocol and meta-analysis, not a method. All scores are author-reported and come from the organisers' pipeline.

Problem

Children are exposed to fewer than 100 million word tokens by age 13, while modern language models are typically trained on three or four orders of magnitude more §2. Models such as Llama 2 are trained on trillions of words §4. Scaling up data and parameters answers a different question, and it excludes researchers without large compute budgets §2. Work on training pipelines for a fixed, small data budget was relatively thin, so the paper sets up a shared budget, a shared test suite and a shared leaderboard to compare methods directly §1 §2.

Method

The paper has no model of its own. It defines the challenge (tracks, corpus, evaluation), reports baselines and skylines, and then hand-codes each of the 31 submissions by approach, comparing scores across approaches §6 §7.4. The steps below are the protocol an implementer would reproduce.

1. Tracks and budget. Three tracks. Strict and Loose allow about 100M words; Strict-Small allows 10M. Loose additionally allows non-linguistic data without a limit, and expert-annotated data counts against the word budget. Repeating the same text over several epochs does not count as new text. Every piece of software in the pipeline counts toward the budget §3.
2. Corpus. Ten English sources. About 56% is transcribed or scripted speech, and about 40% is intended for, or suitable for, children §4.1. Strict contains 98.04M words and Strict-Small 9.96M, sampled at random from the Strict training split tab. 1 §4.2. Splits are about 83.3% / 8.3% / 8.3% §4.2. Newlines are preserved and do not mark document boundaries §4.2.
3. Evaluation tasks.
   • BLiMP, zero-shot: 12 grammatical phenomena, minimal pairs §5.1 app. B.
   • BLiMP supplement, zero-shot: five hidden suites (hypernyms, question–answer congruence in easy and tricky forms, subject–auxiliary inversion, turn-taking) §5.1.1 app. B app. C.
   • (Super)GLUE subset, finetuned: 11 tasks, from CoLA to WSC §5.1 app. B.
   • MSGS, finetuned: six ambiguous subtasks, scored with the Matthews correlation coefficient (MCC). A value of 1 means a systematic linguistic generalisation and −1 a systematic surface one §5.1.1 app. D.
   • Age-of-acquisition prediction, optional: mean absolute deviation in months §5.1.1 app. E.
4. Scoring.
   • Zero-shot pairs: the model is correct when it assigns the acceptable sentence the higher probability §5.1. Masked LMs are scored as described in Salazar et al. (2020), via the minicons implementation §5.2.1 huggingface.py.
   • Finetuned tasks: one finetuning run per subtask, adapted from HuggingFace's text-classification script to support encoder–decoder models §5.2.1.
   • Lexical filter: an example is dropped if any of its words appears fewer than twice in the Strict-Small training corpus §5.2. Evaluation counts are in app. B.
5. Aggregate score. Each task score is the mean of its subtasks. The leaderboard aggregate weights BLiMP and its supplement together at 50%, with all subtasks equally weighted, GLUE at 30% and MSGS at 20% §5.3 tab. 2.

# Zero-shot minimal-pair accuracy (BLiMP, BLiMP supplement)  [§5.1] [§5.2.1]
# Input:  model; pairs of (acceptable, unacceptable) sentences
# Output: accuracy in [0, 1] for one subtask

def minimal_pair_accuracy(model, pairs):
    # Start with no correct pairs
    correct = 0
    for acceptable, unacceptable in pairs:
        # Whole-sentence probability; masked LMs use Salazar et al.'s scoring [§5.2.1]
        p_good = sentence_probability(model, acceptable)
        p_bad = sentence_probability(model, unacceptable)
        # A pair counts only if the acceptable sentence is more probable [§5.1]
        if p_good > p_bad:
            correct += 1
    return correct / len(pairs)

# Leaderboard aggregate  [§5.3] [tab. 2]
# Input:  mean score per task group (grammar pools BLiMP and supplement)
# Output: one number in roughly [-1, 1]

def aggregate_score(grammar_mean, glue_mean, msgs_mean):
    # BLiMP and supplement subtasks pooled, each weighted equally [§5.3]
    # Weights are heuristic; winners were stable across reasonable
    # alternatives [§5.3]
    return 0.5 * grammar_mean + 0.3 * glue_mean + 0.2 * msgs_mean

6. Baselines and skylines. Three baselines (OPT-125M, RoBERTa-base, T5-base) trained on the Strict and Strict-Small corpora with the same pretraining settings §5.4. Two skylines: Llama 2 70B, evaluated in-context on GLUE and finetuned on MSGS, and a fully trained RoBERTa-base run through the same pipeline §5.4.
7. Meta-analysis. Each submission was hand-coded with one or more of nine approaches: curriculum learning, teacher–student or auxiliary model, data preprocessing, hyperparameter tuning and model scaling, multimodal learning, architecture modification, training objectives, linguistic bias, and data augmentation §7.4. Figure 3 counts submissions per approach fig. 3. Figures 5 and 6 plot score against approach and backbone fig. 5 fig. 6.

Hyperparameters

Results

Author-reported, on the Dynabench leaderboard and the organisers' pipeline §5.3. Setup: Strict and Strict-Small use a 100M and 10M word budget; Loose is 100M words plus non-linguistic data. Baselines are OPT-125M, RoBERTa-base and T5-base §5.4. The skylines are Llama 2 70B (2T-token pretraining, far beyond the budget) and a fully trained RoBERTa-base §5.4. The four task columns are task means; Aggregate is the weighted score tab. 2.

• Winning LTG-BERT-based models beat both skylines on the aggregate and on every suite except GLUE §7.1 tab. 2.
• Strict and Strict-Small submissions are close to the Llama 2 skyline on BLiMP and GLUE §7.1.
• Submission counts: 162 models from 31 participants; Strict 24 models, Strict-Small 118, Loose 20 tab. 3.
• Age-of-acquisition (optional task): no submission beat its track's OPT-125M baseline. Best strict baseline MAD is 2.04 months against 2.05 for BabyStories; for Strict-Small it is 2.03 for the baseline and 2.05–2.07 for submissions app. E.
• Subtask breakdowns (MSGS, BLiMP, supplement, GLUE) are in app. D and figures 7 and 8 in app. G.

Paper vs code

The only official code linked by the paper is the organisers' evaluation pipeline, which the paper names as the implementation of the evaluation setup §5.2. It does not contain the baseline pretraining code or the submissions. Checked at commit 17b3806376fa90c24653efe0c6c571adfb6a72c7.

• Baseline pretraining. Paper: constant learning rate 1e-4, batch 128, context 128, 5000 warmup steps, 20 epochs §5.4. Code README: the baselines reuse the original models' pretraining hyperparameters, with context length and sometimes batch size reduced README.md. These two statements do not describe the same procedure, and the baseline training code is not in this repository, so the difference cannot be resolved here.
• Finetuning defaults. Paper: "a default set of hyperparameters" that worked well, values not given §5.2.1. Code: learning rate 5e-5, batch 64, max 10 epochs, eval every 200 steps, patience 10, seed 12, max sequence length 128 finetune_model.sh finetune_model.sh. These are the only finetuning values we found, and the README repeats them README.md.
• Checkpoint selection. Paper: not stated. Code: with patience set, it early-stops and reloads the best checkpoint finetune_classification.py. The metric is F1 for binary tasks and accuracy otherwise finetune_classification.py. MCC is logged for binary tasks finetune_classification.py.
• Masked-LM scoring. Paper: pseudo-likelihood via minicons, after Salazar et al. §5.2.1. Code: a masked-LM class adapted from minicons and Salazar et al. huggingface.py. Consistent.
• Lexical filter. Paper: examples with any word appearing fewer than twice in the Strict-Small train split are removed §5.2. Code: the filtered data ships as filter_data.zip at the repository root; no filtering script was found among the .py and .sh files, so the rule is not verifiable here.
• Aggregate weights. Paper: 50/30/20 §5.3. Repository: no aggregation code found; the leaderboard computes it on Dynabench.
• Licence. LICENSE.md is MIT but names EleutherAI (2020) as copyright holder, reflecting the lm-eval-harness origin. transformers_modified/ has no licence file of its own. We record MIT as the repository licence.

No verbatim code is quoted on this card.

Pitfalls

• MSGS column mismatch. tab. 2 reports MSGS as positive numbers (Llama 2 0.26, ELC-BERT strict 0.47, Contextualizer 0.58). The appendix reports MCC macro averages that are mostly negative (Llama 2 −0.249, ELC-BERT strict −0.104, Contextualizer −0.241) app. D. The text says MSGS is an MCC and that the models mostly prefer surface cues §5.1.1 app. D. The paper does not explain the conversion. The aggregate in tab. 2 uses the positive column, so reproduce the aggregate from that column and state the choice.
• Aggregate reconstruction. Pooling the 12 BLiMP and 5 supplement subtasks equally and then applying 0.5 / 0.3 / 0.2 reproduces the aggregate column of tab. 2 for the rows we checked (ELC-BERT strict 0.74, Llama 2 0.71). This is our calculation from the table, not a formula the paper states.
• Baseline recipe. The paper's uniform baseline recipe §5.4 and the README's "original hyperparameters" README.md disagree. Pin the one you use.
• Epochs are free but expensive. Epochs over the same text do not count against the budget §3, so the winning Strict entry trained for hundreds of epochs (about 450 Strict, over 2000 Strict-Small) §7.2. The paper calls this costly and not cognitively plausible §8.
• Data augmentation can exceed the budget. The Loose winner (Contextualizer) builds extra samples by repeating chunks 40 times, which gives a dataset as large as 4B words from 100M words §7.2.
• Rules on external data. Text generated by a model trained only on BabyLM data does not count, nor does data bootstrapped by such a model §3 footnote. Baby's CoThought used a far larger LLM and does not qualify under any track app. F. Expert-annotated data counts toward the Loose word budget §3.
• GLUE is finetuned, Llama 2 is not. Finetuned models are compared with Llama 2 evaluated in-context on GLUE. The paper notes that finetuning adds labelled data beyond the budget §5.2.1 footnote §5.4.
• Filtered counts are not comparable. Evaluation sizes are after the lexical filter, so they cannot be compared with published full datasets app. B.
• MLSM citation. The MSGS table in the appendix attributes the MLSM row to the CogMemLM citation key app. D, while tab. 2 and the text cite Better Together for MLSM §7.2 §6.
• Placeholder TODOs in the source. The LaTeX source has \todo{Replace with final data} on the submission-count figure and table fig. 2 tab. 3. The numbers are the paper's, but the authors marked them for replacement. Treat them as unverified.
• Loose track baseline. tab. 2 lists no baseline row for the Loose track, so a Loose-versus-baseline comparison cannot be read from the table.

Lineage

• Builds on: LTG-BERT, which ELC-BERT and BootBERT adapt. Not linked, as no card for it exists in this wiki §7.2.
• Compared against: BLiMP and its supplement §5.1 app. B; MSGS §5.1.1; GLUE and SuperGLUE §5.1; OPT-125M, RoBERTa-base, T5-base baselines §5.4; Llama 2 skyline §5.4.
• Superseded by: not known from this wiki. The paper plans future challenges with harder evaluations §8.