Instruction tuning

Overview

Instruction tuning continues training a pretrained model on pairs of a request and a desired response, so that the model follows requests in the prompt rather than only continuing text. The core loss is the usual next-token loss, applied to the response 2305.11206 §2. What varies across the carded papers is the data (a few curated pairs, large prompted dataset mixtures, or model-written dialogue), what is trained (all weights, an adapter, or only a projection between frozen encoders and a frozen language model), and whether a preference stage such as RLHF or DPO follows 2310.06825 §4 2401.04088 §4.

Forty-six carded papers cover this topic. Almost every number below is author-reported, from the authors' own benchmarks, and often from one run.

Variants

Small curated sets

Tuning on about 1,000 hand-picked pairs can reach strong open-ended quality. LIMA tunes LLaMA 65B on 1,000 pairs with a plain supervised loss and no RLHF, and reports it equal or preferred to GPT-4 in 43% of comparisons 2305.11206 §1. The checkpoint was chosen by hand on a 50-example dev set, and the ablations use 7B models 2305.11206 §5. Card: LIMA.

OpenAssistant collects volunteer-written conversation trees, with 161,443 messages in 35 languages, and tunes Pythia, Falcon and LLaMA models with supervised tuning and PPO-based RLHF 2304.07327 tab. 1. The data's licence is called permissive but no licence is named 2304.07327 app. L. Card: OpenAssistant Conversations.

Instruction tuning is not the only lever. A benchmarking study over ten LLMs finds that instruction tuning, more than size, decides zero-shot news-summary quality 2301.13848 §3.2. Card: Benchmarking LLMs for News Summarization.

Prompted multitask mixtures

T0 rewrites 62 datasets with several prompts each and tunes T5+LM 11B (3B also trained). It matches or exceeds GPT-3 on 9 of 11 held-out datasets 2110.08207 §6.1. Its medians span all prompts, while GPT-3 was given one prompt per dataset 2110.08207 §6.1. Card: T0.

Flan-PaLM widens the mixture to 1,836 tasks plus nine human-written chain-of-thought sets. At 540B it reaches 75.2% five-shot MMLU with CoT and self-consistency 2210.11416 tab. 4. Its checkpoint was chosen on the held-out tasks, so those numbers are slightly optimistic 2210.11416 §2.2. Card: Flan-PaLM.

Tuning also changes how a model weighs in-context examples against its prior knowledge. In a flipped-label test, PaLM drops as low as 31% when every exemplar label is flipped, while Flan-PaLM does not fall below chance 2303.03846 §5 2303.03846 fig. 7. The same study reports that instruction tuning improves learning from exemplars but strengthens priors more 2303.03846 §5. Card: Larger language models do in-context learning differently.

The Flan Collection study varies the design choices on T5-XL. Mixing zero- and few-shot templates is part of the recipe, but removing few-shot templates can raise zero-shot CoT 2301.13688 tab. 1. Card: The Flan Collection.

Distillation can also produce reasoning-tuned small models. Chain-of-thought solutions from a large teacher are distilled into T5 and FlanT5 models of 250M to 11B parameters; FlanT5-XXL reaches 27.1% on GSM8K against 16.1% before, but every specialised model scores 0.0 on BBH-CoT 2301.12726 tab. 1. Card: Specializing Smaller LMs to Multi-Step Reasoning.

Frontier-scale and open general models use the same recipe with more stages. PaLM 2 reports Flan-style tuning, with BIG-Bench Hard at 78.1 against 65.2 for PaLM with CoT 2305.10403 tab. 6, but its model sizes and training settings are not published 2305.10403. Card: PaLM 2. Mistral 7B, Gemma, Qwen, Code Llama and ChatGLM also ship chat variants tuned with SFT and sometimes RLHF 2310.06825 2403.08295 2309.16609 2308.12950 2406.12793. A survey of LLMs tabulates tuning settings for many of these models, copied from their papers 2307.06435 §4. Cards: Mistral 7B, Gemma, Qwen, Code Llama, ChatGLM. Mixtral is instruction-tuned by SFT and then DPO 2401.04088 §4; its instruct model reaches MT-Bench 8.30 2401.04088 tab. 3. Card: Mixtral of Experts.

Synthetic and model-written data

Self-Instruct avoids human writers. Starting from 175 hand-written seed tasks, GPT-3 is prompted to write new instructions, then inputs and outputs. Its filters use ROUGE-L and heuristics. The result is about 52K instructions and 82K instances 2212.10560 §2 2212.10560 tab. 1. Fine-tuning GPT-3 on this data raises zero-shot SuperNI ROUGE-L from 6.8 to 39.9, against 40.8 for InstructGPT-001 2212.10560 tab. 3. One author judged only 54% of a 200-instruction sample fully valid 2212.10560 tab. 2. Card: Self-Instruct.

CAMEL generates 25,000 role-play conversations with gpt-3.5-turbo agents, then tunes LLaMA-7B. Human votes prefer CAMEL over gpt-3.5-turbo in 76.3% of cases on AI Society 2303.17760 tab. 1. Card: CAMEL.

Gorilla tunes LLaMA-7B on about 1,645 hub APIs, each with ten GPT-4-written requests 2305.15334 §3.1. It beats zero-shot GPT-4 on all three hub benchmarks, and its authors report fewer hallucinated calls 2305.15334 tab. 1. Card: Gorilla.

Tuning on generated data can also erode general knowledge; see the forgetting row in When to use which.

Domain adaptation

Medical models include ChatDoctor, which tunes LLaMA-7B on about 100k dialogues 2303.14070 tab. 1; DoctorGLM, which uses LoRA on ChatGLM-6B 2304.01097 §3.3; Med-PaLM 2, which tunes PaLM 2 on MultiMedQA and reports 86.5 on MedQA 2305.09617 §4.1; and Med-PaLM M, which tunes PaLM-E on 14 medical tasks 2307.14334 tab. 2. Med-PaLM (the Flan-PaLM variant) only trains a 100-token soft prompt on 40 exemplars 2212.13138 app. A.1. Cards: ChatDoctor, DoctorGLM, Med-PaLM 2, Med-PaLM M, MultiMedQA.

GeoChat applies the LLaVA-1.5 recipe to remote sensing. It trains LoRA on Vicuna-v1.5 7B with about 318k image-instruction pairs made from detection, scene and VQA datasets 2311.15826 tab. 1. Vicuna writes the questions and answers from text descriptions, not from the images 2311.15826 §4. Zero-shot scene classification reaches 84.43% on UCMerced and 72.03% on AID 2311.15826 tab. 5. Card: GeoChat.

A benchmark study compares LoRA-tuned LLaMA 2 13B and PMC LLaMA 13B with zero- and few-shot GPT-3.5 and GPT-4. The tuned baseline wins on 10 of 12 datasets, but its literature baselines were not re-run 2305.16326 p. 6. Card: Biomedical NLP benchmark.

ChipNeMo applies domain-adaptive pretraining, with about 24B tokens, and then supervised tuning on 1.4K domain samples. Its 70B model scores 76.6 on an internal design-knowledge set against 58.4 for GPT-4 2311.00176 app. Auto Evaluation Results. Continued pretraining on a chat-aligned model weakens its alignment 2311.00176 §2.3. Card: ChipNeMo.

Parameter-efficient tuning

LoRA is used in several cards. DoctorGLM applies rank 8 with alpha 16 to the query and value projections, on one A100 for about 8 hours 2304.01097 §3.3. PandaGPT trains only a projection and LoRA on attention, on 160k examples 2305.16355 §3. SpeechGPT's third stage uses LoRA 2305.11000 §5.1. Card: PandaGPT, SpeechGPT. Med-PaLM is the extreme case, training one 100-token prompt 2212.13138 app. A.1.

Multimodal instruction tuning

Most multimodal cards keep the vision encoder and the language model frozen and train a connector first. LLaVA trains a linear projection in stage one and the projection with the language model in stage two, on 158K GPT-4-written visual instructions 2304.08485 app. C. Its successor, LLaVA-1.5, uses a two-layer MLP and reports author-run state-of-the-art results on 11 benchmarks 2310.03744. MiniGPT-4 has only a linear layer in its first stage; its second stage on about 3,500 curated descriptions cuts the failure rate on detailed descriptions from 35% to 2% 2304.10592 tab. 3. Cards: LLaVA, LLaVA-1.5, MiniGPT-4.

mPLUG-Owl also keeps the language model frozen in its first stage. It trains the vision encoder and an abstractor on image-caption pairs, then trains LoRA adapters on the language model with text-only and image-text instructions 2304.14178 §3.2. On the authors' 82-question OwlEval set, 66 answers are rated A or B against 54 for MiniGPT-4, by manual rating 2304.14178 §4.2. Card: mPLUG-Owl.

Other designs change what is trained or what is in the prompt. InstructBLIP trains only the Q-Former, which also reads the instruction, and reports a 15.0% average relative gain over BLIP-2 with FlanT5-XL 2305.06500 §3.1. Shikra writes coordinates as text, and reaches 87.01 REC on RefCOCO val for the 7B model 2306.15195 tab. 3. Visual prompts such as boxes and masks are blended into the image in ViP-LLaVA, which reports 87.91% on Visual7W at 13B 2312.00784 §5.2. LLaVA-Plus trains on tool-use dialogues and scores 69.5 against 57.1 for LLaVA on In-the-Wild 2311.05437 tab. 4. Cards: InstructBLIP, Shikra, ViP-LLaVA, LLaVA-Plus.

Some cards train on several images or on interleaved sequences. Otter adds other image-instruction-response examples to each training item, drawn from MIMIC-IT, about 3M pairs 2305.03726 §4 2305.03726 tab. 1. Only the perceiver resampler and the gated cross-attention layers train, about 1.4B parameters 2305.03726 §3.1. Card: Otter. Emu2 is a 37B model that places images and text in one sequence, with 64 embeddings per image 2312.13286 §2.1. Emu2-Chat is tuned on academic-task and multimodal chat data, with 16x16 pooling per image, 8,000 steps and a global batch of 768 2312.13286 §2.3.1 app. C.2. It scores 84.9 on VQAv2 and 48.5 on MM-Vet, author-reported 2312.13286 tab. 2. Card: Emu2. The card records that the released weights are listed as non-commercial, so check that before any commercial use.

A survey of multimodal LLMs covers these training stages and evaluation choices, but runs no experiments of its own 2306.13549. The OFA model tunes one encoder-decoder on vision and language tasks, with VQA test-std 82.0 2202.03052 tab. 2. Cards: OFA, survey of multimodal LLMs.

Video, speech and driving

Video-ChatGPT tunes Vicuna-7B on 100K video-instruction pairs and scores 2.40 on a GPT-judged benchmark 2306.05424 tab. 1. MVBench introduces a 20-task temporal benchmark, and its VideoChat2 model averages 51.1 against 35.5 for VideoChat 2311.17005 tab. 2. Cards: Video-ChatGPT, MVBench.

SpeechGPT converts speech to discrete units and tunes a LLaMA backbone on about 9M speech-text pairs. Its evaluation is qualitative, with no baselines 2305.11000 tab. 1. Card: SpeechGPT.

DriveMLM tunes LLaMA-7B to output driving decisions with an explanation, and reaches driving score 76.1 against 71.4 for Apollo on CARLA Town05 Long 2312.09245 tab. 4. Card: DriveMLM.

Motion and other domains

MotionGPT adds motion codes to the vocabulary of Flan-T5 and instruction-tunes on text-to-motion, motion-to-text, prediction and in-between prompts 2306.14795 §3.3. Card: MotionGPT.

When to use which

Constraints the cards give: data licence, teacher-model terms (several pipelines rely on GPT-4 outputs), and compute. Cards that name hardware range from 4 A100s to 128 A100s, and TPU runs use up to 512 chips. The Flan-PaLM card puts 540B finetuning at 0.2% of pretraining compute 2210.11416 §2.2.

Implementation sketch

Common core, in Python-flavoured pseudocode. The loss is computed on the response tokens only, and each variant changes which parameters receive gradients and where the data comes from.

# Instruction-tuning loop, common core  [arxiv:2305.11206 §2]
# Input:  pairs (instruction, response); model; trainable_params
# Output: tuned model

def instruction_tune(model, pairs, trainable_params, epochs=3):
    # Only the chosen parameters get gradients: all weights, LoRA
    # adapters or a connector [arxiv:2304.01097 §3.3] [arxiv:2304.08485 app. C]
    optimizer = make_optimizer(trainable_params)
    for _ in range(epochs):
        for instruction, response in pairs:
            # Concatenate prompt and answer into one token sequence
            tokens = tokenize(instruction + response)
            # Mask the prompt so only the response is scored
            labels = mask_prompt(tokens, prompt_length(instruction))
            loss = model(tokens, labels=labels).loss
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return model

For multimodal models, freeze the encoder and the language model in stage one, train only a connector, then unfreeze the language model in stage two 2304.08485 app. C.

Contradictions

• Size of the tuning set. LIMA reports strong results from 1,000 pairs 2305.11206 §1, while Flan-PaLM reports gains from a far larger mixture 2210.11416 tab. 4. The settings differ (chat quality versus benchmark accuracy), so the two results are not directly comparable.
• Mistral 7B. The abstract says the model beats Llama 2 13B on all benchmarks, but the body's table shows one exception, NQ, at 29.0% against 28.8% 2310.06825 tab. 2.
• Med-PaLM 2. The 86.5 MedQA score comes from a MedQA-only variant; the unified model reports 85.4 2305.09617 §3.2 2305.09617 §4.1.
• Judge dependence. Video-ChatGPT scores its own model at 2.40 with a GPT-3.5 judge and 2.49 with a Vicuna judge 2306.05424 tab. 4.
• Paper and code. ChatDoctor's README and paper give different learning rates, epochs and batch sizes 2303.14070. CAMEL's repository prompts and termination rules differ from the paper 2303.17760. Code Llama's paper and its repository disagree on which sizes support infilling 2308.12950.
• mPLUG-Owl's first stage. The text gives learning rate 1e-4 and 2,000 warm-up steps 2304.14178 §4.1, while the appendix table gives 2e-4 and 375 steps 2304.14178 app. A. The abstract says LoRA also trains the abstractor, but §3.2 freezes it 2304.14178 §1 2304.14178 §3.2.
• GeoChat's LoRA targets. The paper applies LoRA to W_q and W_v only 2311.15826 §5.1. The released code applies it to more layers; see the GeoChat card.

Papers

• LIMA: 1,000 curated pairs for LLaMA 65B.
• OpenAssistant Conversations: volunteer-written dialogue with RLHF.
• Benchmarking LLMs for News Summarization: evaluation of instruction tuning for summarization.
• T0: prompted multitask training for zero-shot transfer.
• Flan-PaLM: scaling instruction tuning to 1,836 tasks.
• The Flan Collection: design choices for Flan-style mixtures.
• Specializing Smaller LMs to Multi-Step Reasoning: distilled reasoning for small models.
• ChatDoctor: medical dialogue tuning of LLaMA-7B.
• DoctorGLM: LoRA medical dialogue tuning of ChatGLM-6B.
• Med-PaLM 2: instruction-tuned PaLM 2 for medicine.
• Med-PaLM M: one multimodal model for medical tasks.
• MultiMedQA: soft-prompt tuning of Flan-PaLM for medicine.
• Biomedical NLP benchmark: LoRA tuning against zero- and few-shot LLMs.
• ChipNeMo: domain adaptation for chip design.
• OFA: one sequence-to-sequence model for vision and language.
• LLaVA: visual instruction tuning with GPT-4-written data.
• MiniGPT-4: a projection between a frozen vision encoder and Vicuna.
• InstructBLIP: instruction-aware Q-Former tuning.
• LLaVA-1.5: baselines for visual instruction tuning.
• Shikra: spatial referring through text coordinates.
• SpeechGPT: speech units with a LLaMA backbone.
• PandaGPT: multimodal tuning through a shared embedding space.
• Video-ChatGPT: video instruction tuning.
• survey of multimodal LLMs: survey of multimodal instruction tuning.
• ViP-LLaVA: visual prompts for region-level instructions.
• LLaVA-Plus: tool-use dialogue for multimodal models.
• MVBench: temporal video benchmark and VideoChat2.
• DriveMLM: instruction-tuned driving decisions.
• CAMEL: synthetic role-play conversations.
• Gorilla: tuning for API calls.
• MotionGPT: motion as tokens for instruction tuning.
• A Comprehensive Overview of Large Language Models: survey of LLM tuning and architectures.
• Catastrophic forgetting under continual tuning: forgetting of general knowledge.
• Mixtral of Experts: SFT and DPO tuning of a sparse mixture model.
• Gemma: instruction tuning with SFT and RLHF.
• ChatGLM: alignment and tool use in the GLM-4 family.
• PaLM 2: technical report with instruction tuning.
• Code Llama: instruction tuning for code.
• Qwen: base and chat models with SFT and RLHF.
• Mistral 7B: instruction tuning of a 7B model.
• Self-Instruct: model-written instructions from 175 seed tasks.
• mPLUG-Owl: modular image input to LLaMA-7B with LoRA tuning.
• Otter: multi-image in-context instruction tuning.
• Emu2: 37B interleaved model with chat and generation tuning.
• GeoChat: instruction tuning for remote-sensing dialogue.
• Larger language models do in-context learning differently: flipped and unrelated labels, with PaLM vs Flan-PaLM.