Efficient Continual Pre-training for Building Domain Specific Large Language Models

In plain words

Building a model that is good at one field, such as finance, usually means training a new model from scratch on field text. That is expensive. This paper takes an existing general model and keeps training it on finance text for a while instead. The authors call this continual pre-training.

They build two models this way, one smaller and one larger, and report better scores on finance tasks while the general-purpose scores stay roughly the same. They also ask whether you need all of the finance text. Their answer is that a carefully chosen tenth of it can do better than the whole pile. They pick the text in two ways: by how close it looks to a small set of unlabelled task examples, or by measuring how varied its word types are. The second way needs no task examples at all.

The results come from one paper, one base model family and four finance tasks, so they show a promising direction rather than a settled recipe.

TL;DR

• Continual pre-training of Pythia-6.9B and Pythia-1B on 23.9B tokens of financial text raises the financial 5-shot F1 average (+8.27% and +2.82% relative) tab. 2.
• Training on a 10% subset chosen by similarity to unlabelled task text (or by token-type entropy, which needs no task text) beats training on all of the corpus on Pythia-1B tab. 3.
• Open-domain scores (ARC, MMLU, TruthfulQA, HellaSwag) show no significant change for the 10% variants tab. 4 §4.2.
• Numbers are author-reported and come from one domain and two model sizes.

Problem

Domain LLMs are usually trained from scratch on domain data, which is very costly §1 §2.2. Continual pre-training of an open model is cheaper, but the authors ask three things: whether it helps domain tasks, whether selecting data helps, and whether the model loses general ability §1.

Selecting data is hard for large models because the usual data-selection work was done on small BERT-scale models §5. Task labels are often scarce, so a selection method cannot always rely on labelled or even unlabelled task data §4.2.

Method

Corpus

The corpus has two parts, totalling 23.9 billion tokens (16.5 billion words) §2.1 app. G:

1. Financial news filtered from News CommonCrawl dumps from 2016 to 2022, using a domain filter (a list of news outlets) and a URL keyword filter app. G. The keywords are: economy, market, finance, money, wealth, invest, business, industry app. G.
2. Ten-K SEC filings from 1993 to 2022, parsed with the EDGAR-CORPUS package. Report sections with fewer than 20 words are dropped app. G.

Duplicates are removed with the procedure of the Pythia suite §2.1.

Background: DACP and TACP

• DACP (domain-adaptive continual pre-training): continue training on the whole domain corpus §2.2.
• TACP (task-adaptive continual pre-training): continue training only on unlabelled task text, with no task labels §2.2.

Efficient DACP: train on a selected subset

Both methods pick a subset of the domain corpus with a token budget (10% in the main experiments) §4.2. Each document gets a score from one metric. The scores are converted to 0 to 100 quantiles, and the quantile value becomes the document's selection weight §2.3.1.

ETS-DACP (task-similar; needs unlabelled task text). Score each document by cosine similarity between its embedding and the task-text embedding, using a spaCy model §2.3. The embedding model variant and how task documents are aggregated are not stated §2.3.

ETA-DACP (task-agnostic; no task text). Two metrics are available:

• Perplexity as a novelty signal. Perplexity is computed with Pythia-70m as a surrogate for Pythia-1B, because computing it with the target model costs about 25% of training compute. The paper reports a correlation of 0.97 between the two on a sample dataset §2.3.
• Entropy of part-of-speech tags as a diversity signal, following the curriculum-learning literature §2.3.

ETS-DACP-com combines all three scores. In the main text it averages the three quantile values for ranking §4.2; appendix F describes weighted sampling without replacement with these averages as weights, at 10% and 20% of the corpus app. F.

Sampling modes

• Hard sampling. Rank all documents by the metric and keep the top-k that fill the token budget §2.3.1.
• Soft sampling. Instead of cutting off the rest, draw each document with probability proportional to its normalised quantile score, so documents outside the top set can still be seen §2.3.1. The exact normalisation is not given in the text.

Hard sampling outperforms soft sampling in the reported experiments §4.2.

Pseudocode

# Hard sampling for efficient DACP  [§2.3.1]
# Input:  documents (list), scores (one metric per document, floats)
#         token_budget (int, tokens to keep)
# Output: selected (list of documents)

def select_hard(documents, scores, token_budget):
    # Rank documents by the chosen metric, highest first  [§2.3.1]
    ranked = sort_descending(documents, key=scores)
    selected = []
    used_tokens = 0
    # Keep documents until the token budget is reached  [§2.3.1]
    for document in ranked:
        if used_tokens >= token_budget:
            break
        selected.append(document)
        used_tokens += num_tokens(document)
    return selected

# Quantile conversion of raw metric values  [§2.3.1]
# Input:  raw_scores (N,) floats for one metric
# Output: quantiles (N,) in [0, 100]

def to_quantiles(raw_scores):
    # Each document takes the percentile it falls on  [§2.3.1]
    return percentile_rank(raw_scores)

# Combined selection score for ETS-DACP-com  [§4.2] [app. F]
# Input:  similarity, perplexity, entropy (each (N,) in [0, 100])
# Output: combined (N,)

def combined_score(similarity, perplexity, entropy):
    # Average the three quantile values per document  [§4.2] [app. F]
    return (similarity + perplexity + entropy) / 3

Hyperparameters

Values for continual pre-training. Anything not listed is not stated in the paper.

Results

Author-reported. Evaluation is FLARE (an extension of lm-evaluation-harness) with 5-shot prompts §3.1 §4.1.

Finance tasks, 5-shot, average F1 over FPB, FiQA SA, Headline and NER tab. 2:

Efficient DACP on Pythia-1B, 5-shot, average F1 over the four finance tasks, mean of 10 seeds tab. 3:

Win rates (percentage of pairwise comparisons won) are also given in tab. 3.

Open-domain tasks, Pythia-1B baseline tab. 4: ARC, MMLU, TruthfulQA and HellaSwag. Averages:

The text describes these as "no significant change" §4.2.

Ablation, data amount vs F1 fig. 2: ETS-DACP and ETA-DACP-ent reach an average F1 near 59% at 5% of the corpus and decline after 10%. DACP rises steadily with more data. The perplexity-based selection peaks early and drops sharply between about 1% and 5% §4.3. Values are read from the text, not from the figure.

Losses: the 6.9B run shows a fast fall in financial test loss and a mild rise in Pile loss app. D fig. 3. The Pile loss is highest for ETS-DACP app. D.

Qualitative examples for Pythia-6.9B vs FinPythia-6.9B are in tab. 6; the paper says FinPythia defines "SPAC" correctly where Pythia hallucinates app. C.

Benchmark against BloombergGPT tab. 5: the authors argue that BloombergGPT's reported numbers are not comparable with FLARE numbers, because the splits and metric details differ app. A.

Paper vs code

No official code. Searched: the paper text and bibliography for links (only lm-evaluation-harness, the CommonCrawl bucket and cited sources appear); no project page is linked; GitHub repository search for "FinPythia", "efficient continual pre-training domain LLM" and "DACP continual pretraining" returned no results; a search of the amazon-science organisation for "continual" returned only an unrelated repository. No discrepancies can be checked.

Pitfalls

• Table 3 labels. The DACP row is labelled "2.39B (10%)", but the caption and text describe the 100% row as the one matching FinPythia-1B tab. 3 §4.2. The 10% DACP baseline's selection rule (random, or something else) is not stated.
• Averages do not match their rows. For FinPythia-6.9B the four F1 values in tab. 2 average to 55.01, but the table reports 54.83. The +8.27% change matches the printed 54.83. Use the per-task numbers with that in mind.
• Word and percentage split. The corpus appendix gives 13.2B words at 83.5% and 3.3B words at 16.5% app. G. Those word counts are 80% and 20% of 16.5B. The percentages are inconsistent with the counts.
• Entropy definition conflicts. The method uses POS-tag entropy §2.3. The results discussion says the highest-entropy samples are chosen by "named entity distribution" §4.3. Which feature the reported runs used is not resolved.
• ETS-DACP-com has three descriptions. Averaged quantiles for ranking §4.2; weighted sampling without replacement at 10% and 20% app. F; and hard-sampling rows in tab. 3. Table 3 also lists one soft ETS-DACP-com row with "2.29B" tokens where other rows say 2.39B. Treat the exact procedure as unclear.
• Soft sampling is under-specified. The text says the normalised score is the selection probability §2.3.1, but does not state the normalisation or how many documents are drawn.
• Simulated task data. The "unlabelled task data" are made by setting aside a 50-sample shot pool per task and stripping labels from the rest of each task's training split, which gives 0.24M tokens §4.2. This simulates scarce task data; it is not a real task corpus, and the paper calls it small.
• Similarity aggregation. How document–task similarity is aggregated across the task set is not stated; the spaCy model variant is not named §2.3.
• Surrogate perplexity. The 0.97 correlation is on an unspecified sample dataset §2.3. The other correlations quoted (perplexity with similarity 0.21, with entropy 0.14) appear in the text of §4.3; the correlation heatmap is fig. 6.
• Baselines section is empty. The section heading "Baselines" §3.2 has no text, so the baseline list must come from the results tables.
• Standard-task averages. The Acc average appears to use TruthfulQA MC1, while Acc Norm uses MC2 tab. 4. The paper does not state this.
• ARC description. The text describes ARC as predicting an output grid after one demonstration §3.1. That description fits the grid-puzzle task, yet the cited reference is the 2018 Boratko et al. paper. The page does not resolve this.
• Model naming. Text says "Pythia-6.9B"; tables say "Pythia 7B" tab. 2. Both refer to the same model in the source.
• Per-run cost, derived. The paper gives 800–1000 USD per day for the instance §7. Eighteen days of the 6.9B run is therefore about 14,400 to 18,000 USD by the paper's own rate. This is arithmetic on stated numbers, not a stated total.
• Scale limits. The data-selection results are tested only on the 1B model and one domain. The authors say the 7B selection experiments were not run §7.
• Learning-rate schedule. Constant LR, no warmup stated, and the reduced batch size of 512 differ from the original Pythia run §3.3. A reimplementation must reproduce these choices to match.
• Loss is not a proxy. The paper says validation or test loss on unlabelled data is a poor proxy for task performance in this domain app. D.

Lineage

• Builds on: Pythia as the base model family and the source of the training configuration §3.3.
• Compared against: DSIR and RepSet on the same 10% budget tab. 3; DAS, which does importance sampling on the fly and is described as about 3× as costly as vanilla pre-training §5; TACP and DACP §2.2.
• The paper says it is the first to study data selection for continual pre-training of LLMs §5. That is the authors' claim.
• Superseded by: not known from the wiki.