Galactica: A Large Language Model for Science

In plain words

Galactica is a language model built for science rather than for general web text. Its authors gathered scientific writing (research papers, textbooks, encyclopedia-style reference pages, code, and databases of molecules and proteins) and trained one model on it. The aim is a model that can store, combine and reason about what science has written, so a researcher can ask for it directly instead of searching through lists of results.

The authors added small markers to help with science. One pair of markers wraps a citation, so the model can learn to produce a reference where a sentence needs one. Another marker wraps step-by-step working, so the model can write out intermediate steps. Chemical and biological sequences also get their own markers, so a question can mix ordinary words with a molecule string or a protein sequence.

The largest model has 120 billion parameters. The authors report that it beats larger general-purpose models on several science tests and on some general tests. It still makes mistakes: it can invent facts and references, and it favours popular papers when it suggests citations. The released weights carry a non-commercial licence, which matters for any use beyond research.

TL;DR

• Galactica is a family of decoder-only language models from 125M to 120B parameters, trained on a curated, mostly academic corpus of 106B tokens §4.1 tab. 2.
• Special tokens mark citations ([START_REF]/[END_REF]), step-by-step working (<work>) and SMILES, amino-acid and DNA sequences §3.1.
• Prompt-formatted datasets are mixed into pre-training instead of being added later by fine-tuning §3.2. The 120B model is stopped at the start of the fifth epoch fig. 6.
• Author-reported gains are largest on equation recall, MATH, PubMedQA and citation prediction; gains on TruthfulQA are small in absolute terms tab. 6 tab. 9 tab. 10 tab. 11 tab. 21. Weights are CC BY-NC 4.0 (see Pitfalls).

Problem

Large language models are usually trained on large, uncurated web crawls. The authors argue that this is a poor fit for science: a general corpus can carry misinformation and off-topic text, scientific terms and mathematics tokenise poorly, and much of the training budget goes to tokens with little value for scientific tasks §2. The paper asks whether a small, curated and normative corpus can yield a useful LLM at lower cost §3.

The paper also argues that search engines point to secondary layers (Wikipedia, UniProt, PubChem) that need human curation, and that a model storing knowledge in its weights could organise literature directly §1. Retrieval-augmented models (RAG, RETRO, Atlas) are discussed as the alternative, and the paper deliberately studies weight memory alone §2. The corpus is restricted to open-access sources, so much closed scientific text is excluded §7.1.

Method

Galactica is a decoder-only Transformer trained with a next-token objective on a curated corpus. The main design choices are the data format and the special tokens, plus training on prompt-formatted tasks alongside the corpus.

1. Corpus. The corpus has 106 billion tokens: papers 83.0% (88B tokens, 48M documents), code 6.9%, reference material 6.5%, knowledge bases 2.0%, filtered CommonCrawl 1.0%, prompts 0.3%, other 0.2% tab. 2. Paper sources include arXiv, PMC, Semantic Scholar and bioRxiv; PDFs are converted with GROBID and combined with LaTeX where available app. Papers. Journals in some fields and journals with an approximate impact factor below 1 are excluded, and non-English papers are dropped app. S2.
2. Deduplication. Identical spans of at least 100 bytes are found across the corpus, removed from all but the first file in a fixed source order, and cut back to paragraph boundaries app. Dataset Deduplication.
3. Markdown format. All documents are stored as markdown rather than LaTeX, so that sources blend. Mathematical LaTeX is kept where it is available app. Papers §3.
4. Tokeniser. Byte-pair encoding with a 50k vocabulary, built from a random 2% of the training data §4.1. Digits are split into single tokens; ASCII operators and parentheses are split into single characters; the wrapped sequence types are split character by character §3.1.
5. Citation tokens. Citations are wrapped with [START_REF] and [END_REF]. The identifier is the title, followed by a comma and the author's surname, in the style shown in the example §3.1.2 fig. 4. Title identifiers gave higher citation accuracy than alphanumeric IDs, but hallucinate more at small scale §3.1.2 app. Citation Identifier Ablations.
6. Working memory token. Step-by-step working is wrapped in <work> and </work> §3.1.1. Four prompt sources of this form are included in pre-training tab. 3. Where a calculation is too long for a forward pass, the example writes a Python script inside <work>; the authors did not turn on running that script in their experiments §3.1.1 fig. 3.
7. Prompt pre-training. Task prompts are included in pre-training alongside the corpus, with prompt-style variation, rather than used only in a later instruction-tuning step §3.2. Prompt types and sources are in tab. 4 and app. Prompt pre-training datasets.
8. Architecture. Decoder-only Transformer with GeLU activations, a 2048-token context, no biases in dense kernels or layer norms (following PaLM), learned positional embeddings (ALiBi was tried at small scale and not used) §4.1.
9. Training. AdamW, clipping and learning-rate decay as in the pseudocode below §4.2 tab. 5. The models are trained for 450B tokens, about 4.25 epochs §5.1.
10. Data augmentation. UniProt entries are randomly ordered (sequence first with probability 0.5) and some descriptions become question-and-answer pairs (probability 1/3) app. Knowledge Bases. Papers with Code and IUPAC Goldbook entries are rewritten as questions and answers in part app. Reference Material.

The pseudocode below follows the paper's data and training descriptions. Symbols are illustrative.

# Optimiser settings  [§4.2]
optimizer = AdamW(params, beta1=0.9, beta2=0.95, weight_decay=0.1)
# Clip the global gradient norm  [§4.2]
clip_grad_norm(params, max_norm=1.0)
# Learning rate: warmup, then linear decay to 10% of the maximum  [§4.2]
# Warmup shape and the decay's end step are not stated; max_lr and
# warmup length per model size are in [tab. 5]
lr = learning_rate_at(step, max_lr, warmup_steps)

# Wrap a citation as title, comma, surname  [§3.1.2] [fig. 4]
def cite(context, title, surname):
    return f"{context}[START_REF]{title}, {surname}[END_REF]"

# Wrap step-by-step working in <work> ... </work>  [§3.1.1] [fig. 3]
# Long calculations may be written as a script inside <work>  [§3.1.1]
def format_work_example(question, working, answer):
    return f"{question} <work>\n{working}\n</work>\nAnswer: {answer}"

# Citation prediction: the prompt ends where the reference goes  [§5.4.1]
# Append [START_REF] when none is open  [code: galai/model.py#L384-L397]
def predict_citation(context):
    if "[START_REF]" not in context:
        context = context + "[START_REF]"
    # Generation stops once [END_REF] appears  [code: galai/model.py#L401-L405]
    return generate_until(context, stop_token="[END_REF]")

Hyperparameters

Model sizes and per-size training settings tab. 5. The batch size and warmup units are not stated in the paper.

Other settings, with the units the paper uses:

Results

All numbers are author-reported. Galactica results are zero-shot unless the table says otherwise. Other models are often 5-shot; check each table's caption before comparing.

Further results in the paper:

• Repeated tokens. Validation loss keeps falling over four epochs for all sizes. The 120B model starts to overfit at the beginning of the fifth epoch, and the 30B and 120B models show a repeated plateau-then-decline pattern per epoch §5.1 fig. 6 fig. 7.
• Knowledge probes. Equation recall scales smoothly with size tab. 6. Domain probes scale steadily except BioLAMA, which the authors attribute to zero-shot prompt difficulty. The 120B model scores 43.1% on chemical reactions and 29.4% on mineral groups tab. 7.
• Chain-of-thought vs <work>. On MMLU dev (261 examples), <work> gives 42.4% overall for 120B, against 19.1% for PaLM 540B with 5-shot CoT [app. Chain-of-Thought vs ].
• Citation distribution. Kolmogorov–Smirnov distance between predicted and true citation counts falls with scale, so the bias towards popular papers shrinks §5.4.2 fig. 11. The 120B model still shows some bias §7.1.
• Protein and molecule tasks. Perplexity on four holdout sets falls with size tab. 16. Protein function scores rise with scale tab. 18. On DockSTRING regression, the models solve three targets from sequence alone and fail on ESR2 and PGR app. Docking Regression.
• Decontamination. Removing test items that share 13-gram matches with the corpus moves MMLU subject scores by small amounts in both directions app. Deduplication Results.

Paper vs code

The official code is the galai repository at github.com/paperswithcode/galai, owned by the paperswithcode organisation. The paper does not link it. We treat it as official because the repository's model card credits the Papers with Code group at Meta AI as the developers and reproduces the paper's author list, and the checkpoints are on the Hugging Face facebook/galactica-* pages model_card.md. The repository holds inference code only; training code is not in it.

Checks against the code at 3a724f5:

• Model sizes. The layer, head and head-size settings of the five released models match the paper's table: 12/12/64, 24/32/64, 32/32/128, 48/56/128 and 96/80/128 utils.py tab. 5. The vocabulary is 50,000 and the position limit is 2048, matching §4.1.
• Architecture. The code loads the checkpoints as OPTForCausalLM from Hugging Face model.py. The "no biases" setting is read from a config flag, enable_bias model.py. The paper's GeLU, no-bias and learned-position choices are therefore inferred, and the checkpoint configs are not in the repository.
• Precision. The paper does not state precision. The code uses float32 for all sizes except 120B, which defaults to float16 [code: galai/init.py#L9-L15] [code: galai/init.py#L70-L71].
• Input length. The paper uses a 2048-token context §4.1. The code truncates inputs from the left at 2020 tokens model.py.
• Special token IDs. The code sets pad 1, BOS 0, EOS 2 and UNK 3 model.py. A new document is prefixed with padding, which is then replaced with EOS model.py. The paper does not state these IDs.
• Wrapped sequences. The code splits each character inside [START_DNA], [START_SMILES], [START_I_SMILES] and [START_AMINO] blocks utils.py. The paper also says digits and ASCII operators are split §3.1. That step is in the tokeniser, which is loaded from Hugging Face and is not in this repository, so it is not verified here.
• Citation generation. The code appends [START_REF] when it is missing and stops when every sequence contains [END_REF] model.py model.py. This matches §5.4.1.
• 120B hardware. The paper says 120B runs "on a single A100 node" §4.3. The code's own size estimate is about 2 bytes per parameter utils.py, or about 240 GB for 120B. A single 80GB GPU cannot hold that, so the node must hold several GPUs.
• Generation defaults. The code generates 60 new tokens when no length is given, greedily unless top_p or top_k is set model.py model.py. The paper does not state decoding settings for most results.
• Training. The paper's training library is metaseq, linked from its footnote §4.3. The galai repository has no training code, and I did not check metaseq for Galactica training configs.
• Weights licence. The paper says the model is open-sourced and does not state a licence for it. The repository's LICENSE-MODEL.md and model card state CC BY-NC 4.0 for the weights, while the code is Apache-2.0 LICENSE-MODEL.md model_card.md LICENSE.

No discrepancy in the reported numbers was found, because the repository has no evaluation scripts.

Pitfalls

• 120B learning rate. The table prints 0.7×10⁻⁵ for 120B, against 1×10⁻⁴ for 30B tab. 5. The trend suggests a typo, but the paper gives no correction. Treat it as unknown and check before reproducing.
• Units not stated. Batch size and warmup are listed without units tab. 5. Reading them as tokens is an assumption.
• Prompt-count total. The prompt rows of tab. 4 sum to 1,313,132 prompts, but the total row reads 783,599. The corpus table gives 1.3 million prompt documents tab. 2, which matches the rows. The token total of 358M matches the rows.
• Early stopping. The 120B model is stopped at the start of epoch five fig. 6, so its real token count is not stated. Repeat-epoch results for 120B cannot be reproduced exactly from the paper.
• Zero-shot vs 5-shot. Galactica numbers in the MMLU and QA tables are zero-shot, while several baselines are 5-shot tab. 8 tab. 10. The MATH headline (20.4%) uses 5-shot mCoT, while the <work> MATH row (16.6%) is a different setting tab. 9.
• In-domain results. PubMedQA, MedMCQA and BioASQ training prompts were included in pre-training, so those results measure in-domain fit §3.2 app. Prompt pre-training datasets. tab. 10 labels each row in-domain or out-of-domain.
• Corpus is curated. Several sources are subsets: 2M of 110M PubChem compounds, 0.5M of 227M UniProt proteins, and a filtered set of journals §5.6 §5.7 app. S2. The paper says these limits can be relaxed, so results for other sizes are unknown.
• Section numbers. The paper's own cross-references ("Section 2.2", "Section 2.4") do not match its section numbering. This card uses numbers counted from the LaTeX source. Check against the PDF before citing a section.
• Hallucination. The paper and the model card both warn about hallucination §7.1 model_card.md. The paper advises against using the model for general-knowledge tasks §7.1 and calls verification an open need §7.2.
• Popularity bias. Predicted citations still favour popular papers at 120B §7.1 fig. 11. Do not treat predicted references as verified.
• Weights are non-commercial. The repository's model card and LICENSE-MODEL.md give CC BY-NC 4.0 for the weights model_card.md LICENSE-MODEL.md. Do not serve the weights commercially; this card's code licence (Apache-2.0) does not cover them.
• Memory. The paper notes that 120B fits in a single A100 node for inference §4.3. The code's own size estimate is about 240 GB for 120B utils.py. Plan for multi-GPU hardware.
• Text only. The model has no notion of 3D geometry, so it cannot know atom coordinates §7.1. The DockSTRING failures on ESR2 and PGR show the limit app. Docking Regression.
• Chemistry. IUPAC naming reached 39.2% at 120B, with 9.2% invalid names tab. 13. The paper says that, with only 2M molecules in the corpus, better results are likely with more molecule data tab. 13.
• <work> script running. The authors did not turn on running the Python script within <work> in their experiments §3.1.1. The example in the paper shows that the model can write the script, not that it runs it.
• Tokeniser not in repository. The tokeniser that splits digits and operators is not in the galai repository; it comes from Hugging Face. Reproducing tokenisation exactly needs that tokeniser.

Lineage

• Builds on: OPT, whose training foundation the authors build on [Acknowledgments] (/papers/2205.01068); PaLM, for the no-bias setting §4.1 (/papers/2204.02311); scaling-law work, including Chinchilla's token-budget argument §2; prompt-style multitask training (FLAN, T0, UnifiedQA) §3.2.
• Compared against: GPT-3 (text-davinci-002), OPT, BLOOM, Gopher, Chinchilla, PaLM 540B, Minerva 540B (/papers/2206.14858) tab. 6 tab. 8 tab. 9 tab. 12; ElasticSearch, Contriever and FAISS retrieval baselines for citation prediction §5.4.1; Uni-Mol for MoleculeNet tab. 15.
• Alternative to: chain-of-thought prompting (/papers/2201.11903); the <work> token is offered as an in-weights route to step-by-step working §3.1.1.
• Superseded by: not known from content already in this wiki.