wikillm

Paper · 2023-11-08 · cs.CL, cs.AI, cs.LG

Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models

GBLM-Pruner prunes a pretrained LLM in one pass, with no retraining or weight update. Each weight is scored by its magnitude times a calibration activation norm plus a scaled gradient norm over 128 C4 sequences; the lowest scores are zeroed in each output row. Author-reported: 50% unstructured LLaMA-2-7B WikiText perplexity 6.86, versus 6.92 for Wanda and 7.00 for SparseGPT.

Post-training pruningNetwork pruningpruningsparsityllm-compressiongradientspost-training

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A large language model has billions of numbers, called weights, and many of them can be set to zero without the model getting much worse. Pruning means choosing which ones to remove. The simple way is to remove the smallest weights. A better way also looks at how strongly each input channel is used on sample text.

This paper adds a third signal: the gradient. A gradient says how much the model's error would change if one weight moved a little. The method runs a small set of sample texts through the model once to collect these gradients. Then it goes through the layers one at a time. In every row of a weight matrix it zeroes the weights with the lowest combined score. It does not adjust the weights that remain.

The appeal is that no retraining is needed, and the paper argues that this is much cheaper than methods that recompute the remaining weights. The authors report that it beats the common baselines on most of their tests, though some margins are small.

TL;DR

  • Training-free, one-shot pruning of pretrained LLMs: no retraining and no weight update §1 §2.3.
  • Score per weight: magnitude times input-activation norm, plus a scaled gradient-norm term eq. 15.
  • Author-reported: at 50% unstructured sparsity it has the lowest perplexity of the four methods for all six models in tab. 3; at 2:4 it does not.
  • Official code exists and follows the method's core. Its activation norm differs from the paper's equation by √N, and α is fixed inside the gradient file (see Paper vs code).

Problem

Magnitude pruning ignores how inputs use a weight §2.2. SparseGPT adds a Hessian-based score and updates the remaining weights, which requires matrix inverses §1 §2.2. Wanda scores by magnitude times input-activation norm and needs no update, but the paper calls it heuristic §1 §2.2. The paper's premise is that gradients of a pretrained LLM, which earlier work dropped because they were assumed to vanish at a trained minimum, still carry useful information for unstructured pruning §2.3. The paper says gradient-based prior work targets structured pruning or needs task-specific fine-tuning §2.3 app. B.

Assumptions: calibration data drawn from the pretraining distribution (C4) §3.1; per-layer comparison within each output row §2.3 tab. 2; Hessian approximated with a diagonal and a uniform curvature constant §2.4 eq. 11.

Method

Prose overview. The method applies to every linear layer inside the transformer blocks. Embeddings and the final output layer are not pruned §3.1 prune.py. Four steps:

  1. Gradient pass, once per model. Run calibration sequences through the dense model with a language-modelling loss. For each sequence, take the gradient of every weight. Aggregate over sequences as an ℓp norm, with ℓ1 and ℓ2 tested §2.3 §3.7. The scaling factor α is applied to the gradients here, in the released code gradient_computation.py gradient_computation.py. The result does not depend on the sparsity level §3.9.
  2. Activation norms. For each linear layer, take the norm of each input channel over the calibration tokens §2.3 alg. 1.
  3. Score. Combine the terms as in eq. 15. Figure 1 shows the gradient-times-magnitude form, with the activation term optional fig. 1.
  4. Mask and zero. Inside each output row, zero the fraction s of weights with the smallest scores. For N:M sparsity, zero the n smallest in each group of m input weights §2.3 alg. 1 §3.1. The masked weights are set to zero; no other weight changes alg. 1.

The paper also derives the metric from Optimal Brain Surgeon with a first-order term §2.4 app. I app. L. The final form is found empirically, not derived: the paper drops the square on the activation term, uses absolute values, and uses ℓp aggregation §2.4 §3.7 tab. 8.

Pseudocode
# Score and mask for one linear layer  [alg. 1] [eq. 15]
# Inputs:  weight (d_out, d_in); grad_norm (d_out, d_in) with alpha applied;
#          act_norm (d_in,) per input channel
# Output:  mask (d_out, d_in), True where the weight is set to zero

def gblm_mask(weight, grad_norm, act_norm, sparsity):
    # Weight magnitude, used by both terms  [eq. 15]
    magnitude = abs(weight)                          # (d_out, d_in)
    # Score: |W| * ||X[:, j]||_2 + alpha * |W| * ||G[:, i, j]||_p  [eq. 15]
    score = magnitude * act_norm + magnitude * grad_norm
    # Number of weights to zero in every output row  [code: lib/prune.py#L300]
    count = floor(sparsity * d_in)
    mask = zeros_like(score, dtype=bool)
    for row in range(d_out):
        # Mark the `count` lowest scores of this row  [§2.3] [alg. 1]
        mask[row, indices_of_smallest(score[row], count)] = True
    return mask

N:M variant, as in the released code. Groups are blocks of m consecutive input columns within each row; the paper does not define the grouping prune.py.

Pseudocode
# N:M mask (2:4 or 4:8) per output row  [code: lib/prune.py#L270-L275]
# score: (d_out, d_in) from gblm_mask; n weights zeroed per group of m

def nm_mask(score, n, m):
    mask = zeros_like(score, dtype=bool)
    for start in range(0, d_in, m):
        # Lowest n scores of each row inside this group of m columns
        block = score[:, start:start + m]
        mask[:, start:start + m] = smallest_n_per_row(block, n)
    return mask

Layer-by-layer loop, as in the released code. The paper does not describe it, so it is code behaviour prune.py.

Pseudocode
# Sequential pass over the decoder layers  [code: lib/prune.py#L229-L308]
# grad_norms[layer][name]: from the one-time gradient pass on the dense model

def prune_model(layers, grad_norms, calib_inputs, sparsity):
    for layer_index, layer in enumerate(layers):
        # Activation norms from the current (partly pruned) model  [§2.3]
        act_norms = collect_act_norms(layer, calib_inputs)
        for name, linear in linear_layers(layer):
            # Score and mask this linear layer  [eq. 15]
            mask = gblm_mask(linear.weight,
                             grad_norms[layer_index][name],
                             act_norms[name],
                             sparsity)
            # Zero the masked weights; no weight update  [alg. 1]
            linear.weight[mask] = 0
        # Pruned outputs feed the next layer [code: lib/prune.py#L305-L308]
        calib_inputs = layer(calib_inputs)
    return layers

One-time gradient pass, as in the released code gradient_computation.py.

Pseudocode
# Gradient aggregation over calibration sequences  [§2.3] [§3.7]
# norm: "l1" (sum of absolute values) or "l2" (root of sum of squares)

def gradient_norms(model, sequences, alpha, norm):
    # Running sums per linear weight [code: gradient_computation.py#L136-L137]
    sum_abs = zeros_like_linear_weights(model)
    sum_sq = zeros_like_linear_weights(model)
    for sequence in sequences:
        # Causal language-modelling loss on the full sequence  [§3.1]
        loss = model(sequence, labels=sequence).loss
        loss.backward()
        for each linear weight with gradient g:
            scaled = alpha * g
            sum_abs += abs(scaled)
            sum_sq += scaled * scaled
        model.zero_grad()
    if norm == "l1":
        return sum_abs
    return sqrt(sum_sq)

Key equations, written out:

\mathbf{S}_{ij} = \lVert \mathbf{X}_{:,j} \rVert_2 \, |\mathbf{W}_{ij}| + \alpha \, |\mathbf{W}_{ij}| \, \lVert \mathbf{G}_{:,i,j} \rVert_p eq. 15

The theoretical result, before the empirical changes, is a change in error of the form (w_m \lVert x_m \rVert_2)^2 + w_m(-g_m) eq. 14. The paper's experiments do not use this form directly.

Hyperparameters

NameValueWhereNotes
Sparsity, main setting50% unstructured, per output row§2.3 §3.1 tab. 3Row comparison group is the chosen one tab. 2
Comparison groupoutput,1 (row-wise) selected; others comparedtab. 2 §2.3 app. ELayer-wise 7.45, input,1 10.16, input,128 7.64, output,1 6.86, output,128 7.47 (model not stated)
N:M patterns2:4 and 4:8§3.1 tab. 3Grouping geometry not stated in the paper; code groups consecutive input columns prune.py
Scaling factor α100§2.4 §3.1 tab. 13Chosen on "a held-out validation set"; split not named. Code default is 100 gradient_computation.py
Gradient norm pℓ1 is the best unstructured setting; ℓ2 is close§3.7 tab. 7 tab. 8Norm used for the headline tables is not stated (see Pitfalls)
Calibration set128 sequences × 2048 tokens, random from first C4 train shard§3.1 app. FSampling seed not stated in the paper; code default is 0 main.py
Gradient objectiveLanguage-modelling loss on the input sequence§3.1 gradient_computation.pyTheory uses squared error eq. 6; see Pitfalls
Gradient passOnce, dense model, independent of sparsity§3.9Code computes the gradient file before pruning gradient_computation.py
Hessian constantFactor 2 in the diagonal approximationeq. 11 app. LConstant "absorbed" into α for ranking app. L
Weight updateNone§1 §2.3 alg. 1
Perplexity evaluationWikiText, 2048-token windows, batch size 1§3.2 app. DPaper says validation split; code loads the test split (see Paper vs code)
Zero-shot tasksBoolQ, RTE, HellaSwag, WinoGrande, ARC-easy, OBQA; lm-eval-harness; shots not stated§3.3 app. DText says "six", appendix says "five"
K-shot tasksMMLU 5-shot; GPQA 5-shot; GPQA-D 5-shot; MATH 4-shot; GSM8K 5-shot§3.4Evaluation code not in the repository
Coding tasksMBPP 3-shot; HumanEval 0-shot§3.5Evaluation code not in the repository
PrecisionNot stated in the paper—Released code loads the model in fp16 main.py gradient_computation.py
HardwareNVIDIA A100 40/80 GB; A6000 for 2:4 speed§3.1 §3.9 tab. 11

Results

Author-reported. Setup: WikiText perplexity and zero-shot, K-shot and coding accuracy for LLaMA-1, LLaMA-2, Qwen and DeepSeek models; baselines are magnitude pruning, SparseGPT [arXiv:2301.00774] and Wanda §3.1 app. C.

Perplexity, WikiText, author-reported tab. 3. Rows GBLM-Pruner use the ℓ1 norm:

MethodSparsityLLaMA-2 7BLLaMA-2 13BLLaMA-2 70BLLaMA-1 7BLLaMA-1 13BLLaMA-1 30B
Dense05.474.883.325.685.094.10
Magnitude0.516.036.835.3617.2920.217.54
SparseGPT0.57.006.034.257.226.195.32
Wanda0.56.925.974.227.266.155.24
GBLM-Pruner0.56.865.884.177.156.115.18
Magnitude2:437.778.896.7642.5418.369.11
SparseGPT2:410.828.755.6810.889.067.12
Wanda2:412.119.005.4811.539.596.90
GBLM-Pruner2:411.918.805.4711.339.166.87
Magnitude4:815.917.325.8916.8313.877.62
SparseGPT4:88.467.014.918.457.446.18
Wanda4:88.607.004.778.577.415.97
GBLM-Pruner4:88.636.904.728.487.265.89

The paper states that SparseGPT is better than GBLM-Pruner and Wanda at 2:4 §3.2.

Zero-shot mean accuracy over six tasks, 50% unstructured tab. 1 (per-task rows are in the source table):

ModelDenseMagnitudeSparseGPTWandaGBLM-Pruner
LLaMA-1-7B67.4753.1261.7761.1763.07
LLaMA-1-13B69.9653.8666.2766.4566.65
LLaMA-1-30B72.5059.9869.3370.8471.17
LLaMA-2-13B70.0259.9967.5467.6267.74
Qwen3-14B74.65not reported72.4072.8673.16

STEM and general reasoning, mean over MMLU, GPQA-D, GPQA, MATH and GSM8K, 50% unstructured tab. 4:

ModelBaseSparseGPTWandaGBLM-Pruner
Llama-3.1-8B35.4424.6323.7324.76
Qwen3-8B47.2541.3041.8242.00
Qwen3-14B49.2345.8246.3147.58
Qwen2.5-7B46.1738.2837.5238.61

Coding, mean of MBPP and HumanEval, 50% unstructured tab. 5:

ModelBaseSparseGPTWandaGBLM-Pruner
Qwen3-8B64.2025.0034.8136.02
Qwen3-14B63.5437.5050.2650.67
DeepSeek-R1-Distill-Qwen-7B29.2912.7014.3015.90
DeepSeek-R1-Distill-Qwen-14B45.8522.1223.3224.53

The text gives the coding gain over Wanda as "up to 11.19%"; this matches the DeepSeek-R1-Distill-Qwen-7B mean (15.90 against 14.30) §1 tab. 5.

Vision-language models, mean of MMMU and MTVQA, 50% unstructured tab. 6:

ModelBaseSparseGPTWandaGBLM-Pruner
Qwen2.5-VL-3B36.4725.0028.7328.89
Qwen2.5-VL-7B39.5733.2434.3134.93

Metric ablation, WikiText perplexity, 50%, model not stated in the caption tab. 8:

Pruning metricPerplexity
|W| · |G_acc| (gradient accumulation)119.72
|W| · ‖G‖₁7.17
|W| · ‖G‖₂7.09
(|W| · ‖X‖₂)² + α · |W| · ‖G‖₁6.90
(|W| · ‖X‖₂)² + α · |W| · ‖G‖₂6.88
(|W| · ‖X‖₂)² − α · |W| · ‖G‖₁9743.65
(|W| · ‖X‖₂)² − α · |W| · ‖G‖₂9377.00
|W| · ‖X‖₂ + α · |W| · ‖G‖₁6.86
|W| · ‖X‖₂ + α · |W| · ‖G‖₂6.89

Scaling-factor sweep, LLaMA-2-7B perplexity tab. 13: α = 0.001: 6.920; 0.01: 6.919; 0.1: 6.921; 1: 6.912; 10: 6.890; 100: 6.858; 1000: 6.902; 10000: 6.926; 100000: 6.952. The spread is about 0.09 perplexity points.

Calibration robustness, LLaMA-2-7B, 50%: five random calibration sets of 128 sequences give 6.86, 6.87, 6.89, 6.86, 6.87 §3.7. The paper says perplexity is less sensitive to the number of calibration samples than SparseGPT's §3.7 fig. 3.

Higher sparsity, WikiText perplexity, Wanda vs GBLM-Pruner tab. 17: Qwen3-14B at 0.7: 33.12 vs 32.13; 0.8: 254.90 vs 192.85; 0.9: 10902.73 vs 5225.87; 0.95: 16872.77 vs 11845.89. LLaMA-3.1-8B at 0.7: 113.59 vs 111.48; 0.8: 1480.43 vs 835.78; 0.9: 13155.59 vs 7726.91.

Cost, A100 tab. 9 tab. 10: gradient pass 71.3 s and 62 GB for LLaMA-7B; 102.1 s and 96 GB for LLaMA-13B. Pruning-metric plus weight-removal time for LLaMA-7B and LLaMA-13B: SparseGPT 256,732.95 ms and 423,550.22 ms; Wanda 257.04 ms and 455.39 ms; GBLM-Pruner 403.76 ms and 561.30 ms. The paper states these as 636× and 754× faster than SparseGPT §3.9.

Vision Transformer ViT-B, ImageNet-1k classification accuracy at sparsity 0.5 / 0.6 / 0.7 / 0.8 tab. 12: dense 75.40; Wanda 64.54 / 43.65 / 7.92 / 0.20; magnitude 59.48 / 29.98 / 1.88 / 0.18; GBLM ℓ1 64.64 / 44.15 / 8.89 / 0.32; GBLM ℓ2 64.86 / 44.23 / 8.02 / 0.24. Calibration used 4,096 ImageNet training images app. A.

2:4 speed on an A6000 for LLaMA-1 linear layers tab. 11: q/k/v/o 3.49 to 2.14 ms (1.63×); up/gate 9.82 to 6.10 (1.61×); down 9.92 to 6.45 (1.54×). The text's "1.24×" overall speedup is not in a table §3.9.

Paper vs code

Official repository: VILA-Lab/GBLM-Pruner, cloned at commit b59221570017d2e1dac45cb1c16f1c54ba4d8ea1, MIT license. The paper and abstract link this repository. The README and model zoo link a second repository, RocktimJyotiDas/GBLM-Pruner, which we did not check against this commit.

  1. Activation normalisation differs by √N. The paper's activation term is the norm over all N·L calibration rows alg. 1 eq. 15. The code keeps, for each input channel, the mean over the N sequences of each sequence's squared token norm, then takes the square root layerwrapper.py prune.py. That equals the paper's norm divided by √N, with N = 128. This is derived by reading the code; we did not run it. Taken literally, the paper's α = 100 is a different weighting from the released code's α = 100. To get the code's ranking under the paper's normalisation, α would have to be about 100·√128 ≈ 1131.
  2. α is applied in the gradient script, not the pruning script. Gradients are multiplied by --scale (default 100) before aggregation gradient_computation.py gradient_computation.py. The pruning code uses the stored gradient norm as it is, with no α prune.py. Changing α therefore means recomputing or rescaling the gradient file. The run scripts do not take α.
  3. Gradient aggregation matches the paper's ℓ1 and ℓ2 definitions. ℓ1 is the sum of absolute values over sequences; ℓ2 is the root of the sum of squares gradient_computation.py gradient_computation.py.
  4. Perplexity split. The paper says WikiText validation data §3.2 app. D. The code loads the WikiText-2 test split eval.py data.py.
  5. Calibration seed. The gradient script fixes the sampling seed at 0 gradient_computation.py. The pruning script uses --seed prune.py. The paper states no seed for either. The run scripts use 0, so they agree by default.
  6. Gradients come from the dense model. The gradient file is computed once before pruning gradient_computation.py. Activation norms are recomputed through already-pruned earlier layers prune.py. The paper does not say which model the gradients come from.
  7. Only perplexity is reproducible from the repository. The zero-shot, K-shot, coding and VLM evaluations are not in the code, and eval.py only computes WikiText perplexity eval.py. main.py hard-codes LlamaTokenizer for every model main.py.
  8. Extra options not in the paper. An inverse-gradient variant prune.py prune.py and a Wanda-variant branch that bisects a threshold prune.py exist. Both are off by default main.py main.py. The --grad_norm argument is parsed and printed but never used main.py main.py.
  9. Masks match the paper's description. Unstructured pruning zeroes the first floor(s · d_in) entries of each row after a stable sort prune.py. The paper's "s% weights in each output row" matches this. The paper does not say how the count is rounded.
  10. Pruned layers match the paper. Only linear layers inside the decoder blocks are pruned; embeddings and the output head are skipped prune.py §3.1.

Pitfalls

  • Eq. 15 literally versus the code. An implementer who uses the paper's normalisation with α = 100 gets a much smaller weight on the gradient term than the released code does (see Paper vs code, item 1). Pick one convention and fix α for it. The α sweep tab. 13 does not say which normalisation it used.
  • The gradient objective and the theory differ. The derivation assumes a squared-error loss over outputs eq. 6. The experiments use the language- modelling loss on each sequence §3.1 gradient_computation.py. The paper does not discuss the mismatch.
  • Sign. The theoretical gradient term is w·(−g), which is signed eq. 14. The practical metric uses absolute gradient magnitudes "for stability" §2.4. Subtracting the gradient term instead of adding it collapses perplexity into the thousands tab. 8.
  • The Hessian step has a constant the paper never fixes. The paper writes the diagonal Hessian with a factor 2 eq. 11, then the appendix uses an unspecified constant c app. L. The constant is "absorbed" into α. So the activation-to-gradient ratio is set empirically, not by the derivation.
  • Which ℓp norm the headline tables use is not stated. The ablation says ℓ1 with the activation term gives the best unstructured result §3.7 tab. 8. Tables 1 and 4–6 label the method "Ours" with no norm. Table 3 uses ℓ1. The model zoo's unstructured and 4:8 perplexities match Table 3's ℓ1 rows MODEL_ZOO.md. Table 14 reports both ℓ1 and ℓ2 for chat models.
  • Gradient accumulation is a different metric. The "gradient accumulation" variant (|G_acc|) in Table 8 is not in the released gradient code, which keeps only ℓ1 and ℓ2 gradient_computation.py.
  • Precision and memory. The gradient pass loads the model in fp16 with no loss scaling in the script gradient_computation.py. The ℓ1 accumulator is fp16 gradient_computation.py. The gradient pass keeps two full-size accumulators per linear weight, one fp16 and one fp32 gradient_computation.py. Table 9 reports 62 GB and 96 GB on an A100 tab. 9.
  • Model-count and benchmark-count claims. Section 3.1 says "five models" §3.1. Table 3 has six model sizes, including LLaMA-2 7B and 70B tab. 3. The introduction claims "15+ model families and 20+ benchmarks" §1. The tables show about seven families and sixteen evaluation sets.
  • Zero-shot list contradicts itself. Section 3.3 says the zero-shot results include LLaMA-2-70B, LLaMA-3.1-8B and Qwen2.5-7B §3.3. Table 1 does not include them. The appendix says "five" tasks and lists six app. D.
  • Calibration split for α is unnamed. The paper says α is chosen on "a held-out validation set" §2.3 §3.1, and reports perplexity on WikiText validation §3.2. It does not say whether these are the same split.
  • Tables without a model label. Table 2 (comparison group) and Table 8 (metric ablation) do not name the model in their captions. Table 2's 6.86 for output,1 and Table 8's 6.86 for the chosen metric are consistent with the LLaMA-2-7B GBLM-Pruner ℓ1 value in Table 3 tab. 2 tab. 3 tab. 8.
  • N:M grouping is undefined. The paper says 2:4 and 4:8 but does not say how groups are formed. The code groups consecutive input columns within a row prune.py.
  • Numerically sensitive case: very high sparsity. Perplexity grows steeply above 0.8 tab. 17. The paper reports no variance across runs, apart from the five calibration sets §3.7.
  • Calibration size for ViT. The ViT experiment uses 4,096 ImageNet images, not 128 text sequences app. A. Do not carry the text setting over to images.

Lineage

  • Builds on: SparseGPT, the one-shot pruning baseline with a Hessian-based score and weight updates; Optimal Brain Surgeon and Optimal Brain Damage as the theoretical basis §2.2 §2.3.
  • Builds on: Wanda, arXiv:2306.11695, which scores by magnitude times input-activation norm without weight updates §2.2 app. C. This card has no Wanda page.
  • Compared against: magnitude pruning, SparseGPT and Wanda §3.1 app. C.
  • Superseded by: not known from the content of this wiki.

Source and license

Adapted from “Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models” by Rocktim Jyoti Das, Mukul Ranjan, Mingjie Sun, Liqun Ma, Zhiqiang Shen (arXiv:2311.04902), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/VILA-Lab/GBLM-Pruner at commit b59221570017 (MIT). Any quoted code is shown under that license.