Low-rank adaptation (LoRA and variants)

Overview

Low-rank adaptation keeps the pre-trained weight \mathbf{W} frozen and adds a trainable low-rank path next to it. With input \mathbf{X}, adapter factors \mathbf{L}_1 \in \mathbb{R}^{h\times r}, \mathbf{L}_2 \in \mathbb{R}^{r\times o} and a scalar s, the layer computes 2305.14314 §2 / eq. 3:

\mathbf{Y} = \mathbf{X}\mathbf{W} + s\,\mathbf{X}\mathbf{L}_1\mathbf{L}_2

Only \mathbf{L}_1 and \mathbf{L}_2 receive gradients and optimizer state; gradients still flow through the frozen \mathbf{W} to reach adapters in earlier layers 2305.14314 §2 / §3.

Where the memory goes: the adapters themselves are tiny. For LLaMA 7B on FLAN v2 at batch size 1 (adapters about 0.2% of the weights), the LoRA parameters take 26 MB, LoRA input gradients 567 MB, and even a 4-bit base 5,048 MB 2305.14314 §2. Shrinking the adapter further buys little; shrinking the frozen base is what makes large models fit, which is QLoRA's contribution 2305.14314 §2 / §3.

This page currently covers only papers published on wikillm under a reuse license. The original LoRA paper (arXiv:2106.09685) and DoRA (arXiv:2402.09353) are not summarized here because their licenses do not allow it.

Variants

LoRA with a 16-bit base

The frozen base stays in 16-bit (BF16) and adapters are added to selected linear layers 2305.14314 §2 / eq. 3. In QLoRA's comparisons this is the reference point: BF16 LoRA on LLaMA 7B-65B averages 53.0 5-shot MMLU after Alpaca or FLAN v2 fine-tuning 2305.14314 tab. 4, and on T5-3B Super-NaturalInstructions 55.4 RougeL vs 54.3 for BF16 full fine-tuning 2305.14314 tab. 3. Regular 16-bit fine-tuning of LLaMA 65B needs more than 780 GB of GPU memory, which is the motivation for going further 2305.14314 §1.

QLoRA (4-bit frozen base)

The same adapters, but the frozen base is stored in 4-bit NormalFloat (NF4, block size 64), and the per-block absmax constants are quantized again to FP8 (block size 256, "double quantization"). Each matmul dequantizes the weight to BF16; the adapters are BF16 2305.14314 §3 / eq. 5 / eq. 6. Optimizer states live in NVIDIA unified memory ("paged optimizers"), so rare long-sequence memory spikes spill to CPU RAM instead of failing 2305.14314 §3. The quantizer is covered in weight-quantization.

Adapter recipe: adapters on all linear layers, r=64, \alpha=16 2305.14314 app. B.2. The authors find that the number of adapted layers is what matters, that r is unrelated to final performance once all layers are adapted, and that q/v-only adapters do not reach 16-bit full fine-tuning quality 2305.14314 §4 / fig. 2 / app. A.1 / fig. 4.

Reported results: NF4 with double quantization averages 53.1 5-shot MMLU vs 53.0 for BF16 LoRA over LLaMA 7B-65B 2305.14314 tab. 4; 55.3 RougeL on T5-3B Super-NaturalInstructions 2305.14314 tab. 3. Guanaco 65B (4-bit, 41 GB) reaches 99.3% of ChatGPT on the Vicuna benchmark under GPT-4 judging 2305.14314 tab. 6. Guanaco 33B trains in under 12 h on a 24 GB GPU, 65B in 24 h on one professional GPU 2305.14314 §1 / §5.3.

Card: QLoRA: Efficient Finetuning of Quantized LLMs.

LoRA against other adapter families on LLaMA, BLOOMz and GPT-J (LLM-Adapters)

LLM-Adapters trains four families with one training recipe: prefix-tuning, series and parallel bottleneck adapters, and LoRA 2304.01933 §3.3. The LoRA setting is rank 32 on attention and MLP layers 2304.01933 §4.1; the paper sweeps rank over 4, 8, 16 and 32 on LLaMA-7B math 2304.01933 app. A.4. Learning rate is 3e-4 for everything except prefix-tuning (3e-2), with three epochs 2304.01933 §3.3. Alpha and dropout are not stated in the paper; the card records the released defaults. On LLaMA-13B the author-reported averages are 65.4 for LoRA on six math sets and 80.5 for LoRA on eight commonsense sets, against 64.4 and 81.5 for the parallel adapter 2304.01933 tab. 3 tab. 4. Those are single runs with no variance, and the paper's math sets overlap its training data, so the math gap to GPT-3.5 is not usable evidence (see its card). Card: LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models.

Small-rank QLoRA recipe (BioMistral)

BioMistral 7B, a Mistral 7B model further pre-trained on medical text, is instruction-tuned with QLoRA and 8-bit quantization 2402.10373 §4.4. Its adapter settings are rank r=16, \alpha=16, dropout 0.05, and adapters on seven projections: q, k, v, o, gate, up and down 2402.10373 tab. 6. Those seven cover the attention and MLP blocks, so the choice of layers matches QLoRA's all-linear recipe, while the rank is four times smaller. With \alpha=r the scalar s=\alpha/r is 1; with QLoRA's r=64, \alpha=16 it is 0.25. The paper reports no comparison of ranks and no LoRA-versus-full fine-tuning run, so it supports neither a rank choice nor an equivalence claim 2402.10373 §4.4, tab. 6. Card: BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains.

QLoRA for clinical summarisation (in-context examples as the comparison)

The study adapts eight models to four clinical summarisation tasks in two ways: in-context examples chosen by embedding similarity, or QLoRA, which is used for the open-source models only 2309.07430 §3.2. The paper does not state the rank, alpha or dropout. The released code sets rank 8, alpha 32 and dropout 0.1 constants.py. The same code validates on the training split and evaluates the last saved epoch, which differs from the paper's separate validation set train.py run.py 2309.07430 §4.1.2. QLoRA was not the top configuration. The best setup was GPT-4 with the most in-context examples 2309.07430 §5.1.4, and with many examples most models passed the best QLoRA model, FLAN-T5 2.7B 2309.07430 §5.1.2. Card: Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization.

Adapters for a new input modality (PandaGPT)

PandaGPT gives Vicuna-13B image, video, audio, depth, thermal and IMU input. A frozen ImageBind encoder produces one vector per input, a linear projection maps it into the language model's embedding space, and only that projection plus LoRA weights on the attention modules are trained, on 160k image-instruction examples 2305.16355 §3 2305.16355 eq. 1. The paper gives no rank, alpha or dropout value 2305.16355 §3; the card records the released settings. Its evaluation consists of examples of cross-modal replies, not benchmark scores 2305.16355 §5. Card: PandaGPT: One Model To Instruction-Follow Them All.

Adapters for a robot policy (OpenVLA)

OpenVLA, a 7B vision-language-action model, is fine-tuned on Franka-Tabletop tasks with LoRA of rank 32 on all linear layers. The authors report 68.2 ± 7.5% success, against 69.7 ± 7.2% for full fine-tuning, with about 97.6 M trainable parameters instead of about 7,188 M 2406.09246 §5.3 2406.09246 app. B.3.2. Rank 64 gives the same success with twice the trainable parameters 2406.09246 app. B.3.2. The paper puts a new LoRA task at 10 to 15 hours on one A100 2406.09246 §5.3. The comparison covers two tasks with 33 rollouts per approach, so it does not establish the result for other robots 2406.09246 app. B.3.2. Card: OpenVLA: An Open-Source Vision-Language-Action Model.

LoRA inside a parallel-decoding robot recipe (OpenVLA-OFT)

OpenVLA-OFT keeps LoRA on the OpenVLA-7B backbone and changes what surrounds it: parallel decoding, action chunks and an L1 action head 2502.19645 §4.2 tab. 4. The LIBERO run uses LoRA rank 32. The paper does not state alpha, and the released code uses 16, which gives a scale of 0.5 2502.19645 tab. 4 finetune.py. Of 279 M trainable parameters, 111 M are LoRA, and the rest are the action head and proprio projector 2502.19645 tab. 4. The headline is 97.1% average LIBERO success with wrist image and proprioception, against 95.3% with the third-person image alone 2502.19645 tab. 1. The card records no comparison of this LoRA against full fine-tuning, so the table tests the recipe as a whole, not the adapter 2502.19645 §5.1. Card: Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success.

Adapters for predicting group answer distributions (SubPOP)

The model is trained so that its probabilities over answer letters match the human answer distribution of a demographic group, with a forward KL divergence over the option tokens 2502.16761 §3.1. The adapter is rank 8, alpha 32, dropout 0.05, on the query and value projections 2502.16761 app. B. The authors report a Wasserstein distance on OpinionQA of 0.094 to 0.106 for the four fine-tuned models, against 0.138 to 0.173 for the best zero-shot baseline 2502.16761 tab. 1. Training epochs are not stated in the paper 2502.16761 app. B. Card: SubPOP survey fine-tuning.

Adapters in a diffusion model (One-2-3-45++)

Here LoRA is only the first of two fine-tuning stages. Stage A adds LoRA to the self-attention and cross-attention key and value projections of Stable Diffusion 2, with the rank not stated. Stage B then trains the full UNet at a learning rate the paper calls conservative, without giving the value 2311.07885 §3.1. A LoRA stage followed by full fine-tuning is therefore a different recipe from the adapter-only setups above. Card: One-2-3-45++.

Adapters trained on preference-selected images (Human Preference Score)

Here the adapter is trained on Stable Diffusion v1.4 images that a learned preference scorer ranked, rather than on a labelled task. The LoRA sits on the key, query, value and output projections of the UNet, with the VAE and text encoder frozen. The paper sets rank 32, 10,000 steps, batch size 40 and a constant learning rate of 1e-5 2303.14420 §6 tab. 4. Non-preferred captions are prefixed with "Weird image.", and that phrase is then used as the negative prompt at inference 2303.14420 §6. Regularisation images from LAION-5B are mixed in 2303.14420 §6. In a 100-prompt study with 20 participants, 74% of the adapted model's images got more than 10 positive votes, against 22% for the original 2303.14420 §7.2 fig. 8. The quantitative gains are small: FID 19.72 to 19.35, aesthetic score 5.90 to 6.06, and HPS 0.1898 to 0.1916, with no variance reported 2303.14420 tab. 4 §7.2. The released training script defaults to rank 4, not 32 train_text_to_image_lora.py, so reproduce with the paper's rank stated explicitly. Card: Human Preference Score: Better Aligning Text-to-Image Models with Human Preference.

Adapters in a segmenting multimodal model (PixelLM)

PixelLM uses LoRA on LLaVA-7B and LLaVA-Llama2-13B. The paper does not state rank, alpha, dropout or the target modules 2312.02228 §5.1. The released code's defaults are recorded on the card, not in the paper. Results are the authors' own and are reported against LISA, with about half the TFLOPs 2312.02228 tab. 1. Card: PixelLM: Pixel Reasoning with Large Multimodal Model.

Adapters in a grounding multimodal model (GLaMM)

GLaMM gives its 7B Vicuna LLM LoRA of rank 8, while the CLIP and SAM image encoders stay frozen 2311.03356 app. A.2. The region encoder, the two projections and the pixel decoder are trained in full 2311.03356 app. A.2. The paper writes alpha as 8, but the released training code uses 16 at rank 8. The code's target modules are q_proj and v_proj, and the paper does not state them 2311.03356 app. A.2 train.py. The gains are not broken down by component, so they cannot be attributed to the LoRA step. On the GCG test split it reports mIoU 65.6 and AP50 29.2, against 61.7 and 24.8 for LISA 2311.03356 tab. 3. On refCOCO val it reports 79.5 against 74.9 for LISA-7B, with the metric not named 2311.03356 tab. 4. Card: GLaMM: Pixel Grounding Large Multimodal Model.

Adapters in a social-navigation vision-language model (Social-LLaVA)

Social-LLaVA fine-tunes LLaVA-v1.5-7B with LoRA on over 40K human-annotated question-answer pairs about images from robots moving among people 2501.09024 §I §III-B §IV-A. The paper states no rank, alpha, dropout or target layers, and no learning rate or optimizer. It reports 15 epochs at batch size 4 on one A100 2501.09024 §IV-A. The baselines, GPT-4V and Gemini 1.5 Pro, are only prompted and not fine-tuned 2501.09024 §IV-B. Fifteen human judges scored 50 items on a 1-to-5 scale, and the fine-tuned model scored highest on all five task types, from 3.95 to 4.19 against 2.77 to 3.87 for the baselines 2501.09024 tab. 1. These are author-reported figures, and the paper gives no significance test or variance 2501.09024 §IV-B. Card: Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces.

Adapters in a two-stage image-input LLM (mPLUG-Owl)

mPLUG-Owl trains only LoRA adapters on the LLM, in its second stage. The LLM is frozen in the first stage, which aligns images with captions, and the whole model is frozen in the second, which trains on text-only and image-text instructions 2304.14178 §3.2. The paper does not state rank, alpha, dropout or target modules. The released code uses rank 8, alpha 32 and dropout 0.05, and attaches LoRA to the query and value projections train.py train.py. Its headline result is author-reported: 66 of 82 OwlEval answers rated A or B, against 54 for MiniGPT-4 2304.14178 §4.2. Card: mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality.

Adapters in a vision-centric language decoder (VisionLLM)

VisionLLM writes detection, segmentation, grounding and captioning outputs as text tokens. An Alpaca-7B decoder is mostly frozen, and LoRA is the only trainable part of the LLM 2305.11175 §3.4 §4.1. The paper names LoRA but gives no rank, alpha, dropout or target layers 2305.11175 §3.4 §4.1. Its LoRA ablation varies task randomness: with random task wording and no LoRA, COCO box AP falls to 1.2, against 44.8 with LoRA 2305.11175 tab. 4b. The paper reads LoRA as a bridge between language and visual tokens, but does not measure that app. D. Card: VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks.

Adapters in a geospatial foundation model (Prithvi-EO-2.0)

Prithvi-EO-2.0 uses LoRA in several downstream fine-tunings, for example for segmentation on PASTIS, where the encoder and decoder get different learning rates 2412.02732 §4.3.3. On the BioMassters biomass regression, the best Prithvi configuration uses LoRA on Sentinel-2 bands only with 12 timestamps and reaches RMSE 33.40. The challenge-winner baseline, at 27.49, is better than every Prithvi configuration on that task 2412.02732 §4.4.1. Card: Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications.

Adapters on a frozen 3D foundation backbone for driving scenes (ReconDrive)

ReconDrive keeps the VGGT 3D foundation model frozen and adds LoRA of rank 8 and scale 32 to its backbone. Two new DPT heads, one for Gaussian centres and one for the other Gaussian attributes, are trained 2603.07552 §3.2. The paper does not state which layers get LoRA. The released code adds adapters to the qkv, projection and MLP linear layers, with dropout 0.05 2603.07552 §3.2 recondrive_model.py. The paper's only ablation compares one input frame with two, so the card records no test of the LoRA step against full fine-tuning 2603.07552 tab. 3. The reported gains are author-reported on 14 nuScenes validation scenes: reconstruction PSNR 32.66 against 29.58 for the best per-scene baseline, at about 15 s per scene 2603.07552 tab. 1. Card: ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction.

Analysing a collection of LoRAs (not a training method)

The Universal Weight Subspace paper stacks the A and B factors of many LoRAs layer by layer and runs PCA on them. Projecting a new adapter onto the shared directions keeps most of its behaviour 2512.05117 §4 2512.05117 app. B.5. For 500 Mistral-7B LoRAs the authors report about 19 times less storage 2512.05117 §4. Out-of-distribution reconstruction on Mistral LoRAs reaches ROUGE-L 72.3 with 450 adapters in the basis, against 73.7 for the normal model 2512.05117 app. B.7. Because LoRA factors are not unique, the analysis describes the stored factors rather than the product BA 2512.05117 app. B.5. The paper does not prove that the subspace is universal 2512.05117 §3. Card: The Universal Weight Subspace Hypothesis.

LoRA written by a hypernetwork from a document (Doc-to-LoRA)

Doc-to-LoRA does not train the adapter by gradient descent on each document. A Perceiver-style hypernetwork with 309M trainable parameters reads the document's activations from a frozen LLM and outputs a rank-8 LoRA on the down-projection of every MLP block 2602.15902 §5 §3.2. The hypernetwork is meta-trained so the adapted model without the context matches the same model with it, using self-generated questions and teacher answers 2602.15902 §3.1 app. B.1. Long documents are split into chunks, one adapter per chunk, and the adapters are concatenated along the rank dimension 2602.15902 §3.2. The code adds a bias adapter, so the rank is (K+1)·r rather than the paper's r·K lora_merger.py. On 2WikiMultihopQA, the author-reported normalized score is 0.857 at 0.209 s per update in batched mode, against 0.901 at 40.171 s for oracle context distillation 2602.15902 tab. 1. Batched mode peaks at 11.522 GB of update memory, against 7.820 GB for the oracle; iterative mode peaks at 3.791 GB 2602.15902 tab. 1. On MultiFieldQA the batched method trails context distillation with 25 generated queries (0.481 against 0.528) 2602.15902 tab. 9, so the speed gain does not come with a quality win everywhere. Card: Doc-to-LoRA: Learning to Instantly Internalize Contexts.

Tensor-factorised adapters for a vision transformer (FacT, with LoRA as baseline)

FacT stacks the weight matrices of a Vision Transformer into one tensor and factorises the weight change with Tensor-Train (FacT-TT) or Tucker (FacT-TK) factors, instead of adding one low-rank pair per matrix 2212.03145 §3.2. The paper shows that LoRA applied to each matrix is a special case of this framework, which removes redundancy within a matrix but not across layers 2212.03145 §3.2.1. On VTAB-1K, the rank-8 LoRA baseline averages 74.5% with 0.295 M backbone parameters. NOAH averages 75.5% with 0.361 M, FacT-TK with rank searched per task up to 32 averages 75.6% with 0.069 M, and FacT-TT at rank 4 averages 73.5% with 0.008 M 2212.03145 tab. 1 §4.1. The numbers are author-reported on the test splits and come without standard deviations 2212.03145 §4.1. The paper does not describe a selection rule, but the released code keeps the best test-set accuracy over training epochs fact_tt.py, so the reported figures may be optimistic. Card: FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer.

LoRA on a 6B chat model for Chinese medical dialogue (DoctorGLM)

DoctorGLM adapts ChatGLM-6B with LoRA of rank 8 and alpha 16 on the query and value projections. It trains for one epoch with the Lion optimizer at learning rate 2e-5 2304.01097 §3.3. The paper's evaluation is a set of doctor-marked example answers, with no quantitative benchmark 2304.01097 tab. 2 tab. 3 tab. 4. The stated count of about 7 million trainable parameters does not follow from rank 8 on those two projections. The released model's width and depth give about 3.7 million 2304.01097 §3.3 config.json. The released repository also reports that its LoRA weights were later dropped in favour of P-tuning v2 because the LoRA model lost dialogue ability README.md. Card: DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task.

LoRA for an LLM-based dense retriever (Llama2Vec)

Llama2Vec first adapts LLaMA-2-7B without labels. It trains a joint prompt that rebuilds a Wikipedia passage and predicts the next one, read from two special-token embeddings 2312.15503 §3.2. It then fine-tunes contrastively on MS MARCO with LoRA 2312.15503 §4.1. The paper does not say whether the unlabelled adaptation step used LoRA or full training 2312.15503 §4.1. The LoRA rank is not stated either. The result is 43.1 MRR@10 on MS MARCO passages and 56.4 average NDCG@10 on zero-shot BEIR, against 41.2 and 53.9 for RepLLaMA, both author-reported 2312.15503 tab. 1 tab. 3. Card: Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval.

Two adapters in sequence for temporal reasoning (TG-LLM)

TG-LLM runs two LoRA adapters on Llama-2 13B, one after the other. The first turns a story into a temporal graph, and the second reasons over that graph with chain-of-thought 2401.06853 §4.2. The paper names LoRA without giving a rank 2401.06853 §3.1. The released code uses rank 8, alpha 8 and dropout 0.1 on the q, k, v and o projections Models.py. On the TGQA test set, the fine-tuned Llama2-13B reaches EM 0.797, against 0.821 for GPT-4 with CoT 2401.06853 tab. 4. Card: Large Language Models Can Learn Temporal Reasoning.

Stage-wise LoRA for a multimodal embedder (UniME)

UniME turns a multimodal LLM into an image-text embedder in two stages. Both use QLoRA on the language part: rank 32 in stage 1 and rank 16 in stage 2 2504.17432 §3.4 app. B.1. In the rank study on Phi-3.5-V, stage-1 overall scores stay between 39.9 and 40.3 for ranks 4 to 64. Stage-2 overall scores run from 62.6 to 64.2, with the best at rank 16 2504.17432 tab. 8. Only Phi-3.5-V has these ablations 2504.17432 app. B.1. Card: Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs.

Reweighting training samples by modality information (PID-guided LoRA)

The paper uses its information decomposition to weight vision-language training samples, then trains LoRA on the weighted data. The LoRA settings are rank 16, alpha 32 and dropout 0.05, at learning rate 2e-5 on the last 20% of layers 2606.00959 app. H. On Qwen2.5-VL-7B, uniform-weight LoRA scores 62.0 on MMStar and the PID-weighted version 64.3, a gain of 2.3 points with a 95% paired-bootstrap interval of 1.4 to 3.2 2606.00959 tab. 2 §5.1. The same comparison on Gemma3-12B lowers MMStar from 56.2 to 54.3 2606.00959 tab. 3. Card: Modality interaction via Partial Information Decomposition.

LoRA as the repair step after structured pruning (LLM-Pruner)

LLM-Pruner removes groups of coupled structures from LLaMA and Vicuna models, then repairs the pruned model with LoRA of rank 8 on all linear layers, trained for two epochs on about 50k Alpaca samples 2305.11627 §3.3 app. B.2. The adapter is merged back into the weights, so the final model keeps the pruned shape 2305.11627 §3.3. At 20% of parameters removed, LLaMA-7B averages 60.07 with this tuning and 56.82 without it, against a re-run dense baseline of 63.25 2305.11627 tab. 1. At 50% the paper reports a large degradation, and recovery adds about 7.39% 2305.11627 app. C.5. The calibration samples only score the structures; recovery uses the Alpaca data 2305.11627 §3.2 §4.1. Card: LLM-Pruner: On the Structural Pruning of Large Language Models.

LoRA in the last training stage of a video model (VideoChat2)

VideoChat2 adds LoRA of rank 16, alpha 32 and dropout 0.1 to the frozen LLM in its third stage, while the visual encoder and Q-Former are trained in the same stage 2311.17005 app. A §4.2. Its MVBench ablation compares the same encoder and LLM with and without LoRA: 48.6 without and 51.1 with on Vicuna-7B v0, and 48.1 against 51.2 on Vicuna-7B v1.5 2311.17005 tab. 6. The LoRA setting for stage 2 is not stated 2311.17005 app. A. The benchmark is multiple choice and author-reported, and its rows differ in frame count and LLM, so compare within the ablation rather than across rows 2311.17005 tab. 2. Card: MVBench: A Comprehensive Multi-modal Video Understanding Benchmark.

Random subspace instead of a low-rank product (intrinsic dimension)

Intrinsic-dimension training keeps every pretrained weight frozen and trains only d numbers. They reach the weights through a fixed Fastfood projection, not a learned low-rank product 2012.13255 §3 eq. 2. On MRPC with RoBERTa-Large, the SAID variant reaches 90% of full fine-tuning with 207 trainable values, and DID with 322 2012.13255 tab. 1. The paper compares only against full fine-tuning and its own DID variant 2012.13255 §4.1. The learning rates are not stated, and the d90 values are upper bounds, so reproduce them before relying on them 2012.13255 §4.1 §4.2. Card: Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.

LoRA as the last stage of a speech-language model (SpeechGPT)

SpeechGPT extends a LLaMA model with discrete speech units and trains it in three stages. Stages 1 and 2 are full training, and LoRA appears only in stage 3, which tunes the model on chain-of-modality data 2305.11000 §4.2. The stage-3 adapter has rank 8 and alpha 16, and the paper places it on the attention layers, with about 6M trainable parameters 2305.11000 app. D §4.2. The released script adapts only the query and value projections, with dropout 0.05 com_sft.sh. The paper gives a peak learning rate of 2e-4 for stage 3, while the script uses 2e-5 2305.11000 app. D com_sft.sh. The evaluation is qualitative: human-judged examples, with no baseline and no LoRA ablation 2305.11000 §5.2 tab. 1 tab. 2. Card: SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities.

Mixture of LoRA experts for continual instruction tuning (SAME)

SAME places several LoRA experts and a router in the feed-forward layers of the language model of LLaVA-v1.5-7B, then trains one task at a time without stored old data 2602.01990 §3 §5.1. Its three additions are a router update limited to a covariance subspace, a curvature-scaled update for the expert weights, and a per-task freeze of experts that the current task barely uses 2602.01990 §4. The paper's setting is rank 8, one epoch per task and learning rate 2e-4 2602.01990 §5.1. Author-reported average accuracy is 66.82 on CoIN against 50.58 for MoELoRA, 67.12 on UCIT against 52.06, and 46.53 on TriGap against 44.45 2602.01990 tab. 2 tab. 3 tab. 1. On CoIN, the router step accounts for 10.7 of the 16.2 points over MoELoRA, curvature scaling for 4.6 more, and freezing for 0.9 2602.01990 tab. 4. The CoIN baselines are copied from their original papers rather than rerun, and the released code departs from the paper's equations, so measure before relying on these numbers 2602.01990 tab. 2 §5.1. Card: SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning.

LoRA on a remote-sensing vision-language model (GeoChat)

GeoChat is a LLaVA-1.5-style model for remote sensing. Its CLIP encoder and MLP adaptor stay frozen, and LoRA trains only the Vicuna-v1.5 7B language model, with adapters merged into the weights for inference 2311.15826 §3.1 §3.2. The paper sets rank 64 and names W_q and W_v as the targets 2311.15826 §5.1. It states no alpha or dropout; the released code uses 16 and 0.05 train.py. Training runs in two stages, with 2,400 steps on all data and then 1,600 grounding-only steps, while the released LoRA script runs a single epoch 2311.15826 §5.1 finetune_lora.sh. The reported zero-shot scene classification accuracy is 84.43% on UCMerced and 72.03% on AID, against 68.00% and 51.00% for LLaVA-1.5; these are author-reported 2311.15826 §5.2 tab. 5. Card: GeoChat: Grounded Large Vision-Language Model for Remote Sensing.

Two LoRA heads on one shared LLaMA 2 (graph-fusion VLA)

This vision-language-action model runs a 7B LLaMA 2 with two separate LoRA adapters in the shared blocks, one per head. The LLM head is trained by next-token prediction on plan steps and reasoning traces, and the action head by regression on Cartesian poses and gripper states. The two heads are trained on alternating batches 2508.05342 §4.4. The LLM head uses 125 annotated videos and the action head 120 robot trials 2508.05342 §4.4 §5.3. The rank, alpha and target modules are not stated, and neither are the optimiser and learning rate 2508.05342 §4.4. The text says that the LLaMA backbone is updated together with the adapters 2508.05342 §4.4. The paper reports 40 h of training on one RTX 4090 2508.05342 §4.4, without splitting that time between the two heads. Results are author-reported on four dual-arm block tasks 2508.05342 §5.3 tab. 1. Card: Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control.

LoRA under closed-loop GRPO for an on-board controller (RobotxR1)

RobotxR1 trains a Qwen2.5 model to write the parameter dictionary of a model predictive controller for a racing car. It runs SFT first, then GRPO, where each sampled completion is scored by a simulated lap 2505.03238 §3.3 §4.1. The GRPO stage uses LoRA adapters, but the paper does not state the rank, alpha or target layers 2505.03238 §4.1 §3. The released config sets rank 64, alpha 64, adapters on the q, k, v, o, gate, up and down projections, and 4-bit weight loading rl_mpc_train.yaml rl_mpc_train.yaml rl_mpc_train.yaml rl_mpc_train.yaml. The 3B GRPO run needed about 11 GB of GPU memory, and the authors say models above 3B could not be trained with GRPO on their consumer GPUs 2505.03238 §4.1. Author-reported control adaptability on the Grand Tour track rises from 50.4 to 63.3 at 3B when GRPO follows SFT, and from 11.3 to 41.5 at 1.5B 2505.03238 tab. 2. The 3B SFT row is marked as computed without its one extraction failure. The paper's text gives the 1.5B gain as 20.2 points, but the table gives 30.2 2505.03238 tab. 2 §4.3. Table 2 varies SFT and GRPO but never the adapter, so these gains cannot be credited to LoRA alone. In the released code each completion waits at least 7 s before simulation time rl_mpc_train.py rl_mpc_train.py. Card: RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning.

LoRA on a simulated tutoring student (SFT, then DPO)

The simulated student is Llama 3.1 8B or Llama 3.2 3B. It is first trained with SFT on real student turns, then with DPO on preference pairs ranked by the paper's metrics 2601.04025 §3.2.1 §2.1 §2.3. The LoRA settings are rank 32, alpha 64 and dropout 0.05 2601.04025 app. B.1. The adapter is rank-stabilised 2601.04025 app. B.1 and covers all seven projections model.py. The results table has no rank sweep and no full fine-tuning baseline, so the adapter is not isolated from the training recipe 2601.04025 tab. 1. Author-reported on the Eedi test set, DPO on the 8B model reaches 0.6840 on acts against 0.6671 for SFT, and 0.7390 against 0.7383 on cosine similarity. Prompted GPT-4.1 scores 0.4998 on acts 2601.04025 tab. 1. Card: Simulated Students in Tutoring Dialogues: Substance or Illusion?.

When to use which

Implementation sketch

adapted linear layer                                  [arxiv:2305.14314 eq. 3 / eq. 5]
params: W (h x o, frozen; NF4 + double-quantized constants in QLoRA), bias (frozen)
        L1 (h x r), L2 (r x o) trainable, BF16; s = alpha / r
init:   one of L1, L2 is zero so the adapter starts as a no-op (peft default)

forward(X):
  Wd = W                          # QLoRA: Wd = doubleDequant(W) to BF16, every call
  Xd = dropout(X)                 # dropout on the adapter branch input
  return X Wd + s * (Xd L1) L2 + bias

backward: gradients for L1, L2 only; dX flows through Wd (no gradient stored for W)
optimizer: AdamW over {L1, L2}; QLoRA: paged AdamW 32-bit, grad-norm clip 0.3
code: norm layers in fp32, adapters in bf16, lm_head and embeddings unquantized
      and without adapters; gradient checkpointing on

Implementation notes from the QLoRA card's code check:

• Alpha scaling. QLoRA's eq. 5 omits the scalar, while the code applies peft's \alpha/r (16/64) 2305.14314 eq. 5. Copy r and \alpha together from one source.
• Dtypes (code only). The paper specifies BF16 compute 2305.14314 §3; the released script uses it only when --bf16 is passed and otherwise computes in fp32.
• Merging. Merging an adapter trained against a 4-bit base back into a weight is not covered by the paper. BioMistral merges full models, not adapters, so it does not answer this question either 2402.10373 §3.3.
• Expert rank and routing (SAME). The paper gives each expert rank r and a per-input gate 2602.01990 eq. 3. The released code splits r across the experts, sends all tokens of a batch one shared routing vector, and mixes the A and B experts in a way the paper's equation does not same.py same.py same.py. Pick one form and state which.
• Scale per recipe. The scalar s=\alpha/r ranges from 0.25 (QLoRA, r=64, \alpha=16) to 5.66 (Doc-to-LoRA's code, see below) across the papers above. It is 1 for BioMistral, TG-LLM and RobotxR1's code (r=64, α=64), 2 for DoctorGLM and PID-guided LoRA, 2 for VideoChat2's third stage (r=16, \alpha=32), 2 for LLM-Pruner's code default (r=8, \alpha=16), 2 for SpeechGPT's stage 3 (r=8, \alpha=16), 2 for SAME's released code (\alpha=2r), 2 for LLM-Adapters' commands (r=32, \alpha=64) and 2 for GLaMM's code (r=8, \alpha=16), 0.5 for OpenVLA-OFT's code (r=32, \alpha=16), 4 for SubPOP (r=8, \alpha=32), 4 for the clinical summarisation code (r=8, \alpha=32) and 4 for ReconDrive's backbone (r=8, \alpha=32) 2603.07552 §3.2. Doc-to-LoRA's code sets \alpha = 2r^{3/2} = 45.25 at r=8 model_loading.py, which gives 5.66 only if its scale is \alpha/r as in PEFT; the paper gives no scale. The tutoring simulator is rank-stabilised 2601.04025 app. B.1, so its scalar follows a different rule from \alpha/r. An \alpha copied from one paper changes the effective learning rate of the adapter 2305.14314 app. B.2 2402.10373 tab. 6 2304.01097 §3.3 2606.00959 app. H 2311.17005 app. A 2305.11000 app. D 2311.03356 app. A.2 2602.15902 §3.2 2502.19645 tab. 4 finetune.py post_training.py constants.py same.py rl_mpc_train.yaml rl_mpc_train.yaml.

Contradictions

• Dropout. QLoRA's dropout advice disagrees with itself: 0.05 is said to help small models only, while the chatbot setup uses 0.1 for models up to 13B and 0.05 for 33B/65B; the released 13B script uses 0.05 and the code default is 0.0 2305.14314 app. A.1 / app. B.2.

• Equivalence to full fine-tuning at scale. On T5-11B Super-NaturalInstructions, both BF16 LoRA (60.7) and QLoRA (60.9) are below BF16 full fine-tuning (62.0) 2305.14314 tab. 3, although the paper's headline claims matching 16-bit performance 2305.14314 §4. OpenVLA reports rank-32 LoRA within about one point of full fine-tuning on its robot tasks (68.2% vs 69.7%) 2406.09246 app. B.3.2. These are different models and tasks, and the papers do not test each other's settings, so the two results are not resolved here.

• Target layers. QLoRA finds that adapting only q and v falls short on large models 2305.14314 §4. The survey-distribution paper adapts only query and value and reports gains over zero-shot baselines 2502.16761 app. B. DoctorGLM also adapts only query and value 2304.01097 §3.3, while TG-LLM adapts the attention projections without the MLP Models.py. PixelLM's released code defaults to q and v projections, a setting its paper does not state 2312.02228 §5.1 (see its card). SpeechGPT's paper puts its stage-3 LoRA on the attention layers, but its released script adapts only query and value 2305.11000 §4.2 com_sft.sh. SAME puts its experts only in the feed-forward layers 2602.01990 §3. RobotxR1's code adapts all seven projections, while its paper does not state the targets rl_mpc_train.yaml 2505.03238 §3. None of these papers compares its target layers with all-linear adaptation on the same task, so the results do not settle the question.

• Rank sensitivity. QLoRA reports that r is unrelated to final performance once all layers are adapted 2305.14314 app. A.1. UniME's stage-2 sweep is best at rank 16 and varies by 1.6 points across ranks 4 to 64 2504.17432 tab. 8. LLM-Adapters' LoRA sweep on LLaMA-7B math rises from 57.9 at rank 4 to 61.9 at rank 32, the largest rank tried, and it is single-run 2304.01933 app. A.4. These are different models, stages and metrics, so the findings are not reconciled here.

• Modality reweighting across models. PID-guided LoRA gains 2.3 points on MMStar for Qwen2.5-VL-7B 2606.00959 tab. 2 and loses 1.9 points for Gemma3-12B 2606.00959 tab. 3. The authors say the method cannot create fusion capacity that the baseline model lacks 2606.00959 §5.3.

• Adapted module in mPLUG-Owl. The abstract and §1 say LoRA covers both the LLM and the visual abstractor. §3.2 and the released code train LoRA on the LLM only and freeze the abstractor 2304.14178 §1 §3.2 train.py. Follow the code when reproducing its checkpoints.

• GeoChat target modules. The paper names W_q and W_v as LoRA targets 2311.15826 §5.1. The released code applies LoRA to every linear layer outside the multimodal modules train.py, so the paper's setting is not what the code runs.

• Backbone under LoRA. The graph-fusion VLA paper says the LLaMA backbone and the LoRA adapters are both updated 2508.05342 §4.4. The usual LoRA setup freezes the base weights 2305.14314 §2, so this paper's training is not that setup, and its rank and schedule are not stated.

Papers

• QLoRA: Efficient Finetuning of Quantized LLMs: adapters on all linear layers over a 4-bit NF4 double-quantized base with paged optimizers; 65B fine-tuning on one 48GB GPU.
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models: prefix, series, parallel and LoRA adapters on LLaMA, BLOOMz and GPT-J; LoRA at rank 32 on attention and MLP layers, with a math benchmark overlap caveat.
• BioMistral: A Collection of Open-Source Pretrained Large Language Models for Medical Domains: QLoRA instruction tuning with rank 16 on seven projections of a medical 7B model; the adapter is one step in a wider pipeline of pre-training, merging and quantization.
• PandaGPT: One Model To Instruction-Follow Them All: LoRA on the attention modules of Vicuna-13B plus a projection from frozen ImageBind embeddings, for six input types; no benchmark scores.
• One-2-3-45++: LoRA as the first of two stages in fine-tuning Stable Diffusion 2 for multi-view generation, followed by full UNet training.
• Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success: rank-32 LoRA on OpenVLA-7B inside the OFT recipe of parallel decoding, action chunks and an L1 head; LIBERO and ALOHA results.
• PixelLM: Pixel Reasoning with Large Multimodal Model: LoRA-tuned LLaVA models that emit learned tokens for segmentation masks; LoRA settings not stated in the paper.
• GLaMM: Pixel Grounding Large Multimodal Model: rank-8 LoRA on a Vicuna LLM inside a grounding model that also trains a region encoder and mask decoder; author-reported gains over LISA on grounded captioning.
• OpenVLA: An Open-Source Vision-Language-Action Model: rank-32 LoRA on all linear layers of a 7B robot policy, compared with full fine-tuning on two tasks.
• Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications: LoRA in several downstream fine-tunes of an Earth-observation foundation model, including a biomass regression task.
• SubPOP survey fine-tuning: rank-8 LoRA on query and value, trained with forward KL to group answer distributions.
• The Universal Weight Subspace Hypothesis: shared principal directions across many LoRAs and other model families, used for storage savings and coefficient-only tuning.
• FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformer: tensor-factorised weight changes for a ViT, with per-matrix LoRA as a special case and the baseline.
• DoctorGLM: Fine-tuning your Chinese Doctor is not a Herculean Task: rank-8 LoRA on query and value of ChatGLM-6B for Chinese medical dialogue; no quantitative benchmark.
• Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval: unlabelled adaptation of LLaMA-2-7B followed by LoRA contrastive fine-tuning for passage retrieval.
• Large Language Models Can Learn Temporal Reasoning: two sequential LoRA adapters on Llama-2 13B, one for a temporal graph and one for reasoning over it.
• Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs: UniME, a two-stage QLoRA embedder built from a multimodal LLM, with a rank sweep on one model.
• Modality interaction via Partial Information Decomposition: information-based sample weights for LoRA training of vision-language models, with mixed results across models.
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning: trains d numbers through a fixed Fastfood projection of a frozen model; a random-subspace alternative to a low-rank product, with no LoRA comparison.
• LLM-Pruner: On the Structural Pruning of Large Language Models: rank-8 LoRA on all linear layers as the repair step after structured pruning of LLaMA and Vicuna.
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization: QLoRA (code settings: rank 8, alpha 32) for open-source models, compared with in-context examples, for clinical summarisation.
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark: VideoChat2's stage-3 LoRA of rank 16 on the LLM, with a LoRA ablation on the benchmark.
• SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities: speech units added to a LLaMA vocabulary, three training stages, and a rank-8 LoRA only in the last stage; human-judged examples, no benchmark scores.
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning: mixture of LoRA experts in the feed-forward layers of LLaVA, with constrained router and expert updates for rehearsal-free task sequences; the released code departs from the equations.
• Doc-to-LoRA: Learning to Instantly Internalize Contexts: a Perceiver hypernetwork writes a rank-8 LoRA for a document in one pass, trained to match context distillation.
• ReconDrive: Fast Feed-Forward 4D Gaussian Splatting for Autonomous Driving Scene Reconstruction: rank-8 LoRA on a frozen VGGT backbone with two Gaussian heads, for six-camera driving scenes in one pass.
• Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces: LoRA on LLaVA-v1.5-7B fine-tuned on a crowd-navigation VQA dataset; rank and targets not stated, human-rated with no benchmark.
• mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality: LoRA-only second stage for an image-input LLaMA-7B; the paper leaves LoRA settings open and conflicts with its own abstract on the abstractor.
• VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks: LoRA-tuned Alpaca-7B decodes detection and segmentation as text; the LoRA ablation shows it is needed under randomised task wording.
• GeoChat: Grounded Large Vision-Language Model for Remote Sensing: LoRA rank 64 on the Vicuna-7B language model of a remote-sensing LLaVA-style model, with W_q and W_v as the paper's targets.
• Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control: two LoRA adapters on a shared LLaMA 2 7B, one for a planning head and one for an action head, trained on alternating batches.
• RobotxR1: Enabling Embodied Robotic Intelligence on Large Language Models through Closed-Loop Reinforcement Learning: SFT then GRPO for a Qwen2.5 model that writes controller parameters, with rank-64 LoRA in the released code and simulator-based rewards.
• Simulated Students in Tutoring Dialogues: Substance or Illusion?: rank-32 LoRA student simulators trained with SFT and then DPO, scored turn by turn against real tutoring turns; prompting trails both.
• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference: rank-32 LoRA on Stable Diffusion v1.4 trained on images a CLIP-based preference score selected, with a negative-prompt marker; author-reported user-study gain.