A Comprehensive Overview of Large Language Models

In plain words

Large language models are programs trained on huge amounts of text to predict the next word. Trained that way at a large enough size, they can follow instructions, answer questions, write code and reason through simple problems.

This paper is a long guide to that field rather than a new model. It explains the parts a language model is built from, such as how text is split into pieces, how word order is encoded, and how attention works. It then walks through the families of models that were built this way, how they are trained and tuned to follow instructions, how they are made smaller and cheaper to run, and how they are tested. It ends with where they are used and what still goes wrong.

Read it as a map. It is useful for finding which design choices other teams made and which ones remain unsettled. It does not tell you which model is best, because it does not run experiments.

TL;DR

• A survey, not a method. It covers LLMs with 10B or more parameters and their building blocks, training, tuning, efficiency, evaluation, applications and challenges §1.
• It summarises architecture and optimizer settings of many LLMs in tables §4 tab. 5 tab. 6 tab. 7, and notes that the literature does not settle several design choices, such as positional encoding §3.8.1.
• Its results table tab. 13 collects top scores from cited papers. The paper ran no experiments, and it does not re-check those numbers.
• Some tables are internally inconsistent and use undefined symbols. Check the cited source before you copy any setting §4.

Problem

LLMs can approximate human-level performance on many language tasks, but training and inference are slow, need large hardware and cost a lot to run. The paper names these costs as the reason for work on better architectures, training strategies, parameter-efficient tuning, pruning, quantization and longer context windows §1.

The survey's stated goal is a concise but self-contained overview of the direction of LLM research, with fine-grained architecture and training details for pre-trained models, and with the design lessons of the popular models §1.

Method

This is a literature survey. It has no algorithm of its own. Its structure is given in the introduction as seven branches: pre-training, fine-tuning, efficiency, inference, evaluation, applications and challenges fig. 3 §1. Section by section:

1. Scope §1. The survey covers pre-trained LLMs with 10B parameters or more, following an earlier survey. Smaller models are referred to other surveys §1.

2. Background §2. The building blocks, each with its own subsection.

1. Tokenization: WordPiece, byte pair encoding and UnigramLM are named as the common schemes §2.1.
2. Positional encoding: absolute, relative and learned encodings are named. ALiBi and RoPE are described as the two relative encodings used in LLMs §2.2.
3. Attention: self-attention, cross-attention, sparse attention and FlashAttention. The paper gives the quadratic cost of self-attention and names tiling as the FlashAttention idea §2.3.
4. Activations: ReLU eq. 1, GeLU, and the GLU family eq. 2 with the ReGLU, GEGLU and SwiGLU variants §2.4.
5. Layer normalization §2.5; distributed training §2.6; libraries §2.7; data pre-processing §2.8.
6. Architectures: encoder-decoder, causal decoder, prefix (non-causal) decoder and mixture-of-experts §2.9.
7. Pre-training objectives: full (causal) language modeling, prefix language modeling, masked language modeling and unified language modeling §2.10.
8. Scaling laws: loss follows a power law in model size, data and compute, and model size and training tokens should grow together §2.11.
9. Adaptation stages: pre-training, fine-tuning (transfer learning and instruction tuning) and prompting or utilization. Alignment with human preferences is covered through RLHF, which fits a reward model and then optimises with reinforcement learning §2.12.

3. Large language models §3. Individual models and design summaries.

1. Pre-trained LLMs §3.1. Each model is described with its architecture and training choices. Examples: a PanGu-α equation eq. 3 and a GPT-NeoX block equation, x + Attn(LN_1(x)) + FF(LN_2(x)) eq. 4.
2. Fine-tuned LLMs §3.2: instruction tuning on manual and generated data, alignment with human preferences, continued pre-training and sample efficiency.
3. Increasing context window §3.3.
4. Augmented LLMs §3.4: retrieval-augmented and tool-augmented models.
5. LLM-powered agents §3.5.
6. Efficient LLMs §3.6: parameter-efficient fine-tuning, quantization and pruning.
7. Multimodal LLMs §3.7.
8. Summary and discussion §3.8: layer normalization, positional encoding, parallel attention, multi-query attention, mixture of experts, mixed precision, training instability, weight initialization, learning rate, training parallelism, mode switching, controllable generation, and the supervised-versus-generalized, zero-shot-versus-few-shot and encoder-versus-decoder questions §3.8.1 §3.8.2 §3.8.3 §3.8.4 §3.8.5.

4. Model configurations §4. Tables of architecture, pre-training optimizer and instruction-tuning optimizer settings tab. 5 tab. 6 tab. 7.

5. Datasets and evaluation §5. Training datasets §5.1 and evaluation datasets and tasks §5.2, with a table of training and evaluation datasets per model tab. 8 to tab. 12, and a table of top performers tab. 13.

6. Applications §6 (general purpose, medicine, education, science, mathematics, law, finance and robotics). 7. Challenges §7 (computational cost, bias, hallucination, security, interpretability, real-time processing and others). 8. Conclusion §8.

The survey also records the findings of popular models in tables tab. 1 tab. 2 tab. 3 tab. 4, which are in section 3.

Pseudocode for the GLU family, as the survey defines it. The Swish function is named in the paper but not defined there §2.4:

# SwiGLU feed-forward layer  [§2.4] [eq. 2]
# Input:  x (batch, d_in); W, V (d_in, d_hidden); b, c (d_hidden,)
# Output: (batch, d_hidden)

def swiglu(x, W, V, b, c, beta):
    # Gate branch: linear map, then Swish with factor beta  [§2.4]
    gate = swish(x @ W + b, beta)
    # Value branch: second linear map, with no nonlinearity  [§2.4]
    value = x @ V + c
    # Element-wise product of the two branches  [§2.4] [eq. 2]
    return gate * value

Hyperparameters

The survey gives no hyperparameters for a method of its own. The settings below are copied from the tables in the paper, which list them per model. The tables do not define the symbols -, × and ✓. Here - is read as not stated, which is consistent with the text, which says that some details are not provided by the papers §4.

Architecture, a subset of tab. 5:

nL is the number of layers, nH the number of attention heads and HS the hidden size tab. 5. This is a subset; the full table lists every model the survey covers tab. 5 §4.

Pre-training optimizer settings, a subset of tab. 6:

The caption of tab. 6 reports a common value for most models: 0.1 for the weight-decay coefficient, 1.0 for the gradient-clipping threshold and 0.1 for dropout. The rows list these per model, and several entries are × or - tab. 6.

Instruction tuning settings: tab. 7 lists batch size, sequence length, LR, warmup, decay and optimizer for each instruction-tuned model. Examples: Alpaca (13B) LR 1e-5, cosine decay; LIMA (65B) LR 1e-5, linear decay; Flan-U-PaLM (540B) LR 1e-3, constant tab. 7. The caption states that no model uses weight decay for instruction tuning tab. 7.

Other settings in the text: mixed precision keeps master weights and optimizer states in FP32 §3.8.2; the usual learning rate range is stated as 1e-4 to 8e-4 §3.8.2; MT-NLG (530B) and GPT-NeoX-20B interpolate their learning rates from the GPT-3 models of 13B to 175B §3.8.2.

Results

The survey reports no experiments of its own. Its only table of results is a compilation of top-performing models per benchmark, taken from the cited papers tab. 13 §5. The survey does not state which of these numbers were rerun. The table is author-reported, and the shot settings differ between rows.

Top-1 entries of tab. 13, as listed:

Source for all rows: tab. 13 §5. The table also lists the top-2 and top-3 entries per task. Model sizes are shown as the survey writes them, and - means the survey gives no size tab. 13.

Paper vs code

No official code. The paper has no code link, and it is a survey with no implementation of its own. The only GitHub mentions in the source are for datasets and models in its tables, not for this paper. No repository search was run for the paper itself, since a survey has no implementation to find.

Pitfalls

• Labels that conflict within the paper. ERNIE 3.0 is listed as (10B) in the architecture table tab. 5 and as (12B) in the optimizer table tab. 6. DeepSeek-v2 is listed as (67B), the same size label as DeepSeek (67B) tab. 5 tab. 6. Check the size against the original model report before you use it.
• Undefined symbols. The tables use -, × and ✓ without a legend. - is read here as not stated. × and ✓ are not explained tab. 5 tab. 6 tab. 7.
• Stated rule vs the table. The text says the usual learning rate is within 1e-4 to 8e-4 §3.8.2. tab. 6 also lists values of 0.01 (T5, PaLM), 7e-6 (Galactica) and 0.97e-5 (GPT-NeoX-20B). Treat the range as a rough guide only.
• Incomplete rows. LLaMA-2 has no layer, head or hidden-size values, and PaLM-2 has almost no architecture data tab. 5. Look up the source before you rely on a row.
• Positional encoding and normalization are open. The survey reports that one study found ALiBi better than learned and rotary encodings, while another found rotary better than ALiBi. For normalization, one study found pre-norm degrades fine-tuned performance compared with post-norm, and GLM-130B uses a post-norm variant. The survey draws no conclusion §3.8.1.
• Performance table is not comparable across rows. Shot counts, model sizes and whether a score is fine-tuned vary within tab. 13. GPT-4 is listed without a size tab. 13.
• Version. This card follows the LaTeX source fetched for arXiv version v10, as recorded in the metadata. Later versions may differ.
• Figures are not reproduced. The survey's figures, including the seven-branch overview fig. 3, are described here only by their captions.
• Scope. The survey covers models of 10B parameters or more §1. Smaller models are outside its tables.

Lineage

• Compared against: other LLM surveys and topic surveys, which the paper cites in its introduction without comparing their content §1.
• Superseded by: not stated.