A Survey on LoRA of Large Language Models

In plain words

Large language models are too big to fine-tune in full on ordinary hardware, because every weight needs gradients and optimizer state. LoRA gets around this by freezing the original weights and training a pair of small matrices that are added to each big weight matrix. Those small matrices are the only part that changes, so they are cheap to train, small to store, and easy to swap in and out of the same base model.

This survey collects the many follow-up ideas. Some make LoRA more accurate, some combine several of these small add-ons to handle several tasks, some make it faster or lighter, some use it when the training data sit on many private devices, and some show it working in real applications. It is a map of the area. It does not test anything itself, and it does not give a single recipe to copy.

TL;DR

A survey of LoRA (low-rank adaptation) for large language models. LoRA trains a low-rank product BA added to a frozen weight matrix, with a scaling factor α §2.1 eq. 1. The survey covers five branches: variants that improve downstream accuracy §3, mixing LoRA modules for cross-task generalization §4, efficiency §5, federated learning §6 and applications §7. It runs no experiments; its one results table copies GLUE numbers from earlier papers tab. 1.

Problem

Full fine-tuning of a large model needs memory for the weights, gradients, activations and optimizer state. For LLaMA2-7B the survey cites about 60 GB, which exceeds common consumer GPUs §1 §2.3. Other parameter-efficient methods either add modules or change the input, while intra-parameter methods change a small part of the existing weights §1. LoRA is named as the most used intra-parameter method §1.

The survey also states the open gap: LoRA can trail full fine-tuning on knowledge- and skill-heavy tasks such as code and mathematics §3.1. Its variants aim to close that gap, along with allocating ranks, improving training and combining modules §1 §3.

Method

This is a survey, so there is no single method to reimplement. The steps below follow the survey's own taxonomy fig. 1. The core LoRA formulation is the only equation it writes out.

1. Core LoRA. For a dense layer with weight W0 of shape d × k, LoRA sets W = W0 + αBA. W0 is frozen. B is d × r and A is r × k, with r much smaller than min(d, k). The trainable count is r(d + k), against d·k for the full layer §2.1 eq. 1.
2. Initialization. One of A and B is random Gaussian and the other is zero, and α controls update strength §2.1. The survey names opposite factors in §2.1 and §3.3.1; see Pitfalls.
3. Where it is applied. The original setting covers the query and value projections in attention. Later work reports gains from also adapting the feed-forward layers §2.1.
4. Pluggable and combinable. Trained LoRA parameters can be separated from the base model, shared, and combined across tasks §2.1 §1.
5. Theory. Work on the kernel view says LoRA in the lazy regime is nearly equivalent to full fine-tuning. An expressive-power result gives a rank threshold for approximating a smaller target model. A neural-tangent analysis suggests rank on the order of the square root of the number of data points. Freezing the down-projection A and tuning only B is reported to generalize better with about half the parameters §2.2.
6. Downstream accuracy variants §3:
   • Breaking the low-rank bottleneck §3.1. Stacking LoRAs over training (ReLoRA, COLA, MELoRA) §3.1.1; treating LoRA as a random gradient compressor with resampled projections (FLoRA) §3.1.2; updating the base weight from the change in the LoRA product (Delta-LoRA) §3.1.3.
   • Dynamic rank allocation §3.2. SVD-style control of singular values (AdaLoRA, SaLoRA, IncreLoRA) §3.2.1; single-rank components pruned or gated (DoRA as dynamic LoRA, AutoLoRA, SoRA, ALoRA) §3.2.2; rank sampled per step from a distribution (DyLoRA) §3.2.3.
   • Learning procedure §3.3. Initialization from principal singular components (PiSSA) or minor ones (MiLoRA) §3.3.1. Gradient-side changes: a Riemannian preconditioner, different learning rates for A and B (LoRA+), residual connections (ResLoRA, SIBO), gradient-free search (CMA-ES, FWA), and weight decomposition into direction and magnitude (DoRA, weight-decomposed) §3.3.2. Overfitting control with bi-level training (BiLoRA), dropout variants and HiddenKey §3.3.3.
   • Combining with other paradigms: Bayesian (Laplace-LoRA), in-context (PILLOW) and active learning (STAR) §3.4.
7. Cross-task mixing §4:
   • Manually weighted merges: plain averaging, hand-set factors, similarity-weighted token-level mixing, and classic model-fusion rules §4.1.
   • Learned weights: task-level search with CMA-ES (LoRAHub), with follow-ups on quantized modules (ComPEFT, L-LoRA); instance- and token-level gating (MixLoRA, X-LoRA) §4.2.
   • LoRA mixture of experts: a router over LoRA experts, trained jointly. Variants change initialization, task balance, or sparsity (Mixture-of-LoRAs, MultiLoRA, MLoRE, MTLoRA, MoLA, LLaVA-MoLE, SiRA, Octavius, Fast LoRA). I-LoRA uses two LoRAs for long- and short-term memory §4.3.
8. Efficiency §5:
   • Parameter reduction §5.1. Freezing parts of LoRA (LoRA-SP, LoRA-FA, AFLoRA, DropBP) and SVD-based fixed-basis variants (LoRA-XS, BYOM-LoRA) §5.1.1. Pruning, using LoRA outputs or gradients (LoRA-drop, LoRAPrune) §5.1.2. Sharing frozen or spectral parts across layers (VeRA, VB-LoRA, FourierFT) §5.1.3.
   • Quantization §5.2. Post-training: QLoRA stores the base in 4 bits, trains LoRA in higher precision and dequantizes at inference; QA-LoRA uses group-wise operators so that inference needs no dequantization §5.2.1. Quantization-aware: LoftQ alternates quantization with low-rank fitting; ApiQ initializes to preserve activations; L4Q designs the layer for joint training §5.2.2.
   • Parallel frameworks §5.3. Fine-tuning several LoRA jobs on one shared model (ASPEN) §5.3.1. Serving many adapters with batched kernels, paging and tensor parallelism (Punica, S-LoRA, CARASERVE) §5.3.2.
9. Federated learning §6:
   • Data heterogeneity: data-driven initialization from SVD of accumulated updates (SLoRA), SVD of pre-trained weights (FeDeRA), and freezing one factor (FFA-LoRA) §6.1.
   • Device heterogeneity: dynamic numbers of activated matrices (FedMS), rank adjustment with SVD redistribution (FlexLoRA), per-client ranks with sparsity-weighted aggregation (HETLORA) §6.2.
   • Model heterogeneity: LoRA as the carrier of shared and local knowledge across different client models (pFedLoRA) §6.3.
   • Privacy: LoRA inside a trusted execution environment with encrypted transfer (TEE work), a client-held trainable matrix (PrivateLoRA), and Gaussian noise on updates (DP-LoRA) §6.4.
10. Applications §7: language (traditional NLP, code, model alignment, vertical domains), vision (diffusion-based image generation, SAM-based segmentation) and multimodal models (audio, image and video text models).
11. Future directions §8: serving LoRA as a cloud service, continued pre-training in vertical domains, and LoRA-based agents with LoRA used for roles, tools and memory.

Pseudocode for the forward pass. The survey does not write this order of operations. It follows from eq. 1 with row-vector inputs, so the steps are inferred from the equation.

# LoRA forward pass for one dense layer  [§2.1] [eq. 1]
# Input:  x (batch, k); frozen W0 (d, k); B (d, r); A (r, k); alpha (scalar)
# Output: y (batch, d)

def lora_forward(x, W0, B, A, alpha):
    # Frozen path: the original layer, W0 gets no gradient  [§2.1]
    frozen_out = x @ W0.T                  # (batch, d)
    # Low-rank path: project to rank r, then back up to d  [eq. 1]
    low_rank_out = (x @ A.T) @ B.T         # (batch, d)
    # Add the scaled update  [eq. 1]
    return frozen_out + alpha * low_rank_out

Key equation, as written in the survey:

W = W_0 + \alpha B A eq. 1

Hyperparameters

The survey trains no models, so most values below are not stated. The few numbers it gives are worked examples or come from cited studies.

Results

The survey runs no experiments of its own. The table below copies GLUE results for RoBERTa-base from earlier papers. The caption says the numbers are taken from the cited literature tab. 1 §3.4. Metrics: Matthews correlation for CoLA, Pearson correlation for STS-B, accuracy for the rest. The paper prints the MRPC column as "MPRC" (see Pitfalls); this card uses MRPC.

Source for every row: tab. 1. Bold marks the best value per column in the printed table.

Other numbers the survey states, each from a cited study and not checked here §2 §3.1:

• LoRA is reported comparable to or better than full fine-tuning across 57 MMLU tasks §2.1.
• Full fine-tuning of LLaMA2-7B on one 24 GB GPU needs about 60 GB; LoRA needs about 23 GB, with the setup in the Hyperparameters table §2.3.
• A cited study finds the rank needed for full fine-tuning is 10 to 100 times higher than LoRA's §3.1.

Paper vs code

No official code. The abstract links ZJU-LLMs/Awesome-LoRAs, a curated paper list for updates and discussion, not implementation code. It is not cloned or recorded as code. The survey implements no method of its own, so there is no training code to compare against. The equation and worked numbers above come from the paper text.

Pitfalls

• Survey, not a benchmark. The results table is copied from earlier papers and is not reproduced here. Settings such as seeds, epochs and learning rates differ across those sources, and the survey does not state them.
• Initialization contradiction. §2.1 says B is Gaussian and A is zero. §3.3.1 says A is Gaussian and B is zero, and then argues Init[A] is better. The survey does not resolve this, so check the original LoRA setting before implementing.
• Name clash: DoRA. §3.2.2 uses DoRA for Dynamic Low-Rank Adaptation, a rank-allocation method, while §3.3.2 uses DoRA for Weight-Decomposed Low-Rank Adaptation, a magnitude-direction reparameterization. These are different methods.
• Parameter ratio is loose. §2.3 says LoRA adjusts "nearly one-thousandth" of the parameters of one layer. The survey's own numbers give 60,416 / 45,088,768 ≈ 0.13%, about 1/746.
• Rank gain is disputed inside the survey. §3.1.1 presents merge-and-reinit stacking as a way to raise rank. MELoRA, which it cites, argues that the merges can overlap so the rank increase is not guaranteed. The survey reports both claims without deciding.
• Memory numbers are from one study. The 60 GB and 23 GB figures, and the 1.9× speed-up, come from one cited study with batch size 1 on one GPU §2.3. They are not measured here.
• Uncited count. §4 says there are more than 20,000 LoRA plugins on Hugging Face. No source or date is given.
• Typo in table header. The printed column "MPRC" should read MRPC. The caption's metric rule (accuracy for every task except CoLA and STS-B) covers it.
• Scaling convention. Eq. 1 uses α alone and gives no division by r. The survey does not say how α is set, so an implementation that uses a different convention needs checking against the original paper.
• Figure numbers. Figs. 1 and 2 were confirmed in the compiled PDF. Figs. 3 to 5 were counted in source order and not checked against the PDF. The figures are drawings by the survey authors; none are reproduced on this card.
• Version. Card follows the arXiv v4 LaTeX source (updated 2024-10-24). Section, table and equation numbers were checked against the compiled PDF.

Lineage

• Builds on: LoRA itself, the method the survey formalizes in §2.1 (original paper, no card here). The survey grounds it in the low intrinsic dimensionality hypothesis §2.
• Related card: QLoRA, the 4-bit quantized LoRA the survey covers in §5.2.1.
• Compared against: the survey's Table 1 sets LoRA variants beside full fine-tuning and each other tab. 1.
• Superseded by: not stated.