--- kind: concept slug: network-pruning title: Network pruning summary: >- Removes the least useful parts of a model by a scoring rule, or rewires a sparse model during training, so it gets smaller or cheaper to run. Variants differ in unit (weights, kernels, filters, LLM groups, search edges, 3D primitives, neurons) and score (L1 magnitude, gradient, posterior variance, learned mask). Finer units keep more accuracy; coarser ones give regular structure hardware can use. tags: [compression, sparsity, efficiency] papers: ["1705.08922", "1905.04919", "2406.17074", "2305.11627", "2301.00774", "1908.09791", "1911.11134", "2605.06714", "2408.15865", "2305.14109", "2311.11883", "2410.11845", "2601.03290", "2309.11932", "2303.13569", "2110.00791", "2110.14430"] related: [dynamic-sparse-training, token-pruning, weight-quantization, post-training-quantization, neural-architecture-search, knowledge-distillation, low-rank-adaptation, gaussian-splatting, convolutional-neural-network, efficient-vision-backbone, object-detection, hyperparameter-optimization, bayesian-optimization, backdoor-attack, adversarial-robustness] status: draft updated_at: 2026-10-31 --- ## Overview Pruning scores each removable unit, cuts the lowest-scoring ones at a target share or threshold, and keeps the rest [arxiv:1705.08922 §3.3]. Most variants below share that loop. They differ in the unit that is removed, the score, and whether the cut is followed by retraining. The unit matters most: a single weight and a whole filter may save the same number of parameters while behaving very differently on real hardware [arxiv:1705.08922 §1]. Two entries below do not cut a trained model. Once-for-all training trains smaller sub-networks that share weights and selects one of them per target [arxiv:1908.09791 §3.3]. RigL trains a sparse network from a random start and rewires it during training, so no dense model is ever trained [arxiv:1911.11134 §1]. Two more entries put a cut inside a larger pipeline: a gradual schedule for a microcontroller detector [arxiv:2408.15865 §2], and a search that treats pruning as one of its settings [arxiv:2305.14109 tab. 1]. A learned-mask variant prunes neurons to remove a backdoor, without fine-tuning the network afterwards [arxiv:2110.14430 §3.3] [arxiv:2110.14430 §4.2]. ## Variants ### Magnitude pruning at four granularities [Exploring the Regularity of Sparse Structure in Convolutional Neural Networks](/papers/1705.08922) prunes convolution weights by the L1 norm of each grain and compares four grain sizes under one method and one schedule, so that only the grain changes [arxiv:1705.08922 §3.2] [arxiv:1705.08922 §4]. The grains are single values, sub-kernel vectors, kernels and filters. - Accuracy at matched density: on five ImageNet CNNs, fine-grained pruning has the highest top-5 accuracy in every row. Kernel pruning trails it by 0.27 to 1.21 points and vector pruning by 0.08 to 0.47 points [arxiv:1705.08922 tab. 1]. - Storage at the baseline accuracy: the ranking depends on the model. Fine grain has the lowest storage for AlexNet and DenseNet-121, vector grain for VGG-16, GoogLeNet and ResNet-50. Kernel grain has the highest storage in all five [arxiv:1705.08922 tab. 2]. - Memory references: in a simulated sparse-dataflow accelerator on VGG-16 convolution layers, vector pruning needs 65 to 70 percent of the references of fine pruning [arxiv:1705.08922 tab. 3]. - Filter grain is only described in the text. The paper reports that AlexNet loses about one point of validation accuracy at the first pruning stage [arxiv:1705.08922 §4]. ### Dependency-aware Bayesian pruning of a search graph [BayesNAS](/papers/1905.04919) uses pruning inside one-shot architecture search. Each candidate edge gets a prior whose variance combines the switches of its predecessor edges. The prior is updated with a Laplace approximation, and an edge is removed when its prior variance is at most 1/(2πe), the point where the entropy of a zero-mean Gaussian becomes non-positive [arxiv:1905.04919 §5.3]. The score is therefore a posterior quantity, not the raw weight magnitude [arxiv:1905.04919 §1]. The pruned network is then retrained from scratch [arxiv:1905.04919 §6.1]. The same machinery can also prune groups of convolution or fully connected weights, with a reweighted group-Lasso term, and the paper reports small MNIST and CIFAR-10 compression results with no run-to-run spread [arxiv:1905.04919 app. S2.2]. ### Structural pruning of large language models [LLM-Pruner: On the Structural Pruning of Large Language Models](/papers/2305.11627) targets LLaMA-family models and removes whole coupled groups of weights. It first follows dependency rules through the network, so that structures which break if one is removed are pruned as one group [arxiv:2305.11627 §3.1]. Each structure is scored from gradients on 10 calibration samples, using a first-order or second-order estimate of how the loss changes if it is zeroed. The scores are summed per group, and the lowest-scoring groups in each module are removed [arxiv:2305.11627 §3.2] [arxiv:2305.11627 eq. 4] [arxiv:2305.11627 app. B.1]. Recovery trains low-rank adapters on about 50k instruction samples for two epochs, then merges them back into the weights [arxiv:2305.11627 §3.3] [arxiv:2305.11627 app. B.2]. On LLaMA-7B at 20 percent of parameters removed, the authors report 60.07 average zero-shot accuracy against a re-run dense baseline of 63.25, which is 94.97 percent, with tuning [arxiv:2305.11627 tab. 1]. Without tuning the same setting keeps 89.8 percent [arxiv:2305.11627 tab. 1]. Removing the dependency step is very costly: WikiText2 perplexity after tuning rises from 17.58 to 13307.46 [arxiv:2305.11627 tab. 6]. At 50 percent the paper reports a strong drop that recovery only partly repairs [arxiv:2305.11627 app. C.5]. ### One-shot pruning of a GPT-family model with a weight update [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](/papers/2301.00774) prunes every linear layer inside the decoder blocks of a GPT-family model in one pass, with no retraining [arxiv:2301.00774 §1] [arxiv:2301.00774 §4.1]. It works from a Hessian of each layer's inputs, estimated on 128 calibration segments of 2048 tokens from C4 [arxiv:2301.00774 §4]. Unlike the cuts above, it does not only drop weights: after each removal, the surviving weights in the row are adjusted to offset the error [arxiv:2301.00774 §3.1]. The mask is chosen in blocks of 128 columns by a second-order score, the squared weight divided by the squared diagonal of the inverse Hessian, so sparsity can differ between columns [arxiv:2301.00774 §3.2] [arxiv:2301.00774 alg. 1]. - Accuracy at 50 percent unstructured sparsity: on OPT-175B, raw-WikiText2 perplexity is 8.21 against 8.35 dense, and magnitude pruning reaches about 4.3e4 [arxiv:2301.00774 tab. 1]. The zero-shot average is 70.52 against 70.29 dense [arxiv:2301.00774 tab. 2]. The authors call the 175B result a slight gain that depends on the dataset [arxiv:2301.00774 §4.1] [arxiv:2301.00774 app. C]. - Hardware-friendly patterns: 2:4 gives 8.74 and 4:8 gives 8.45 on the same WikiText2 setting [arxiv:2301.00774 tab. 1]. The method sets the block size to the group size for these patterns [arxiv:2301.00774 §3.3]. - Cost: a full 175B run takes about 4 hours on one A100 80GB [arxiv:2301.00774 §4]. The authors say the earlier accurate one-shot method, AdaPrune, takes about 1.3 hours at 350M and 4.3 hours at 1.3B, and extrapolate to hundreds of hours at 175B [arxiv:2301.00774 §2] [arxiv:2301.00774 §4.1]. - Speed: on CPU, OPT-2.7B end-to-end runs 1.82 times faster at 50 percent sparsity [arxiv:2301.00774 tab. 7]. On an Ampere GPU, 2:4 matrix products of OPT-175B shapes run 1.54 to 1.79 times faster than dense cuBLAS [arxiv:2301.00774 tab. 8]. The paper gives no end-to-end GPU speed for the large models [arxiv:2301.00774 app. E]. - Joint quantization: 50 percent sparsity with 4-bit weights gives 8.29 on OPT-175B, against 8.68 for GPTQ at 3 bits, which the authors compare at about the same storage [arxiv:2301.00774 §4.1] [arxiv:2301.00774 app. C]. The paper's text prunes layers in order, block by block, and feeds the pruned outputs of each block into the next [arxiv:2301.00774 §4]. The card for this paper records a difference in how the released code collects Hessians inside a block; see the contradictions below before copying the order. The paper's update is written with entries of the inverse Hessian, but its algorithm uses the Cholesky factor of that inverse [arxiv:2301.00774 alg. 1]. Follow the algorithm for an implementation. ### Once-for-all sub-networks instead of a cut [Once-for-All: Train One Network and Specialize it for Efficient Deployment](/papers/1908.09791) trains one super-network whose sub-networks differ in depth, expand ratio, kernel size and input resolution [arxiv:1908.09791 §3.2]. The authors describe this as a generalised form of pruning, because it also shrinks depth, kernel size and resolution, not only width [arxiv:1908.09791 §3.3]. Width is reduced by keeping the channels ranked most important by L1 norm [arxiv:1908.09791 §3.3]. Training is progressive: the largest sub-network first, then smaller kernel, depth and width choices added in stages with distillation from the largest network [arxiv:1908.09791 §3.3] [arxiv:1908.09791 app. B]. A device then gets a sub-network chosen by a search over an accuracy predictor and a latency table, with no training per device [arxiv:1908.09791 §3.4] [arxiv:1908.09791 app. A]. Without fine-tuning, the specialized model reaches 76.0 percent ImageNet top-1 at 230M MACs, against 75.2 percent for MobileNetV3-Large at 219M MACs [arxiv:1908.09791 tab. 1]. Without progressive shrinking, the authors report 72.4 percent at 235M MACs [arxiv:1908.09791 tab. 1]. ### Rewiring a sparse network during training [Rigging the Lottery: Making All Tickets Winners](/papers/1911.11134) keeps a fixed number of weights active, with fixed parameter count and fixed training FLOPs, from the first step [arxiv:1911.11134 §1] [arxiv:1911.11134 §3]. Every 100 iterations it drops the active weights with the smallest magnitude and grows the same number at inactive positions with the largest dense gradient. New connections start at zero [arxiv:1911.11134 §3] [arxiv:1911.11134 §4]. The share replaced each time follows a cosine decay from an initial fraction of 0.3 [arxiv:1911.11134 §3 (2)] [arxiv:1911.11134 §4]. Growth needs the dense gradient at update steps only, computed layer by layer [arxiv:1911.11134 §3]. On ImageNet ResNet-50 at 80 percent sparsity, the authors report 74.6 percent top-1 at 0.23 times the dense training FLOPs. A static random mask reaches 70.6 and SET 72.9 at the same cost, and the dense model 76.8 [arxiv:1911.11134 fig. 2]. With 5 times the training steps, RigL reaches 76.6 at 1.14 times the training FLOPs [arxiv:1911.11134 fig. 2]. The magnitude-pruning row in that figure is taken from another paper and was not rerun, and it reaches 75.6 at 0.56 times the training FLOPs [arxiv:1911.11134 fig. 2]. On character-level WikiText-103 the authors report that RigL does not match magnitude pruning [arxiv:1911.11134 §4.2]. On CIFAR-10 they report that it matches pruning with fewer training resources [arxiv:1911.11134 §4.3]. The [paper card](/papers/1911.11134) lists where the released code differs from the text, for example on how regrown weights are initialised. ### Iterative L1 pruning at the end of training on a microcontroller [microYOLO: Towards Single-Shot Object Detection on Microcontrollers](/papers/2408.15865) prunes every convolutional and linear layer of a small YOLO-style detector with an L1-norm criterion, in an iterative and gradual schedule [arxiv:2408.15865 §2]. In the detection runs the pruning covers only the last 100 of 400 epochs, every 20 epochs [arxiv:2408.15865 §3]. The target sparsity and the step size are not stated, and it is not stated whether the first cut falls on epoch 300 or 301 [arxiv:2408.15865 §2] [arxiv:2408.15865 §3]. Pruning is followed by 8-bit post-training quantization, with no quantization-aware training [arxiv:2408.15865 §2]. On the fridge task the model reaches 56.4 percent mAP50 averaged over seeds, with no baseline detector reported [arxiv:2408.15865 §3.1] [arxiv:2408.15865 fig. 1]. The deployed fridge model needs 752 KB of flash and 324 KB of RAM, at 3.46 FPS on an OpenMV H7 R2 with a Cortex-M7 [arxiv:2408.15865 tab. 2]. The authors report that the fridge task loses mAP50 during the pruning epochs [arxiv:2408.15865 §3.1], and that more aggressive pruning reaches up to 8 FPS at a clearly lower mAP50 [arxiv:2408.15865 §3.3]. The deployed model is the pruned and quantized one, so these numbers do not isolate the effect of pruning. ### Pruning as one searched setting [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](/papers/2305.14109) does not fix a pruning recipe. It searches pruning together with training and quantization settings, with a budget of 150 samples per run [arxiv:2305.14109 §5]. The pruning settings are a start fraction in [0.0, 0.6], an end fraction in [0.8, 0.95], between 1 and 20 steps, and a sparsity in [0.1, 0.99] for each prunable convolution, so the search space grows with depth [arxiv:2305.14109 tab. 1] [arxiv:2305.14109 §5]. The schedule is the automated gradual pruning of Zhu et al. [arxiv:2305.14109 §5]. Each sample trains a network, prunes and quantizes it, then generates code to measure its memory [arxiv:2305.14109 §4]. The paper reports the highest hypervolume after about 50 to 75 samples on two tasks, against five baselines, but the values are only in a figure [arxiv:2305.14109 §5.1] [arxiv:2305.14109 fig. 2]. One of its own designs uses 276 KB of RAM [arxiv:2305.14109 tab. 3], above the 256 KB limit that the experiments state [arxiv:2305.14109 §5]. ### Learned neuron masks to remove a backdoor [Adversarial Neuron Pruning Purifies Backdoored Deep Models](/papers/2110.14430) repairs a model whose trigger is unknown, using a small clean set. It gives each neuron a mask in [0, 1], optimised while every neuron's weight and bias are perturbed in the worst-case direction, then prunes the neurons whose mask falls below 0.2 [arxiv:2110.14430 §3.3] [arxiv:2110.14430 §4.1]. The premise is that backdoor-related neurons are dormant on clean inputs, so adversarial perturbation switches them on and the masks learn to drop them [arxiv:2110.14430 §3.2]. The mask objective weights clean loss against worst-case loss with a trade-off α, set to 0.2 on CIFAR-10 [arxiv:2110.14430 eq. 8] [arxiv:2110.14430 §4.1]. In the paper's CIFAR-10 / ResNet-18 benchmark with 500 clean images, the attack success rate falls below 1 percent on five of six attacks, and CLB stays at 3.98 percent. The mean clean-accuracy drop is 1.47 points [arxiv:2110.14430 tab. 1]. Fine-tuning at learning rate 0.01 gives a mean accuracy drop of 3.53 points in the same table, and fine-pruning gives 2.00 [arxiv:2110.14430 tab. 1]. - Cost: no fine-tuning, so no weight changes beyond the pruned neurons [arxiv:2110.14430 §4.2]. The authors report about 2.5 times the run time of fine-tuning, with the hardware not named [arxiv:2110.14430 tab. 2] [arxiv:2110.14430 §4.2]. - Clean data: with 10 percent of the training set, ACC stays at 93.01 to 93.86 percent across attacks. With 50 images, ACC falls to 85.66 to 89.54 percent [arxiv:2110.14430 tab. 7] [arxiv:2110.14430 tab. 9]. - Sensitivity: the good threshold band is roughly 0.2 to 0.5 for ResNet-18, ResNet-50 and VGG-19, and MobileNet-V2 needs a smaller budget [arxiv:2110.14430 §4.3] [arxiv:2110.14430 fig. 3]. With 20 mask iterations, the ASR rises again after about 10 percent of neurons are pruned, and the effect goes away at 100 or more iterations [arxiv:2110.14430 §4.4] [arxiv:2110.14430 fig. 4]. ### Taxonomy and results from surveys Surveys run no experiments of their own, or quote results from other works. Their numbers are leads to the original papers, not measurements to copy. [The edge-vision survey](/papers/2605.06714) runs no experiments of its own. It sorts pruning into weight, neuron, channel and filter forms, and quotes results from the works it cites [arxiv:2605.06714 §4.1]. Its quoted examples include a channel-pruned ResNet-50 with half the parameters at 99 percent of the original ImageNet accuracy, and a filter-pruned ResNet-50 that runs 2 times faster at 98.6 percent of the original [arxiv:2605.06714 §4.1.3] [arxiv:2605.06714 §4.1.4]. The numbers come from different setups and are not comparable with the rows above. Use the survey as a map to the original papers. [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](/papers/2311.11883) calls magnitude pruning with a cubic sparsity schedule the main recipe for tiny models, and lists structured pruning of filters, neurons or layers as the alternative [arxiv:2311.11883 §4.2]. Its 90 percent sparsity with acceptable loss is a cited claim, not a measurement [arxiv:2311.11883 §4.2]. The schedule is given in the implementation sketch below. [A Review on Edge Large Language Models: Design, Execution, and Applications](/papers/2410.11845) groups pruning into structured forms that remove heads, channels or layers, and unstructured forms that remove single weights [arxiv:2410.11845 §3.2]. Its qualitative advice is structured pruning for strict memory with moderate compute, and unstructured pruning for maximum compression with some accuracy loss tolerated [arxiv:2410.11845 §3.6]. [Lightweight Transformer Architectures for Edge Devices in Real-Time Applications](/papers/2601.03290) lists a pipeline with structured pruning of 30 to 40 percent of attention heads and 20 to 30 percent of MLP channels [arxiv:2601.03290 §VIII-H]. It gives no protocol, dataset or sample sizes for these figures. The paper's own step reductions, compounded by our arithmetic, give about 3 to 3.6 times smaller models, not the 8 to 12 times it reports; the paper does not show how that was derived [arxiv:2601.03290 §VIII-H]. [A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932) splits pruning into weight, neuron and structured variants, applied during or after training [arxiv:2309.11932 §5.1] [arxiv:2309.11932 §5.1.1]. [TinyML: Tools, Applications, Challenges, and Future Research Directions](/papers/2303.13569) quotes a speech-enhancement RNN with structured pruning and integer quantization, at 11.9 times smaller size and 2.9 times fewer operations [arxiv:2303.13569 §III-A1]. Pruning and quantization are combined there, so the figure does not isolate pruning, and it is quoted from another work. ### Mentioned without numbers [Optimizing Neural Network for Computer Vision task in Edge Device](/papers/2110.00791) says a gesture model is structurally pruned on a representative dataset, but gives no ratio, no criterion and no before-and-after parameter count [arxiv:2110.00791 §3]. Use it as a pointer to the idea, not as evidence. ### Pruning primitives of a 3D scene [Reducing the Memory Footprint of 3D Gaussian Splatting](/papers/2406.17074) applies pruning to a scene of Gaussians rather than to network weights. A Gaussian counts as redundant when many others lie in a sphere sized by its smallest pixel footprint, and the lower-opacity half of the redundant ones is removed. The lowest 3 percent by opacity are removed as well, with the cut capped at opacity 0.05 [arxiv:2406.17074 §4.1]. Pruning is combined with dropping view-dependent colour bands per Gaussian and with codebook quantisation [arxiv:2406.17074 §4.2] [arxiv:2406.17074 §4.3]. On its three dataset groups the paper reports 25.7 to 34.8 times smaller files, with PSNR moving between −0.32 and +0.16 dB against the authors' re-run baseline [arxiv:2406.17074 tab. 1]. The authors' own ablation keeps 43 percent of the primitives with all rules, 48 percent with the opacity rule only and 56 percent with the redundancy rule only [arxiv:2406.17074 tab. 3]. The paper's code is under a non-commercial licence (see the card), so implement from the paper's description. ## When to use which | Situation | Use | Why | |---|---|---| | Smallest file, the target hardware runs sparse weights well | Fine-grained magnitude pruning | Best accuracy at matched density, and lowest storage for AlexNet and DenseNet-121 [arxiv:1705.08922 tab. 1] [arxiv:1705.08922 tab. 2] | | Need speed on ordinary dense hardware and accept some loss | Kernel or filter grain | Regular structure is easier to run on general hardware, at a higher accuracy cost [arxiv:1705.08922 §1]. Kernel grain has the highest storage of the four [arxiv:1705.08922 tab. 2] | | Custom accelerator where memory traffic dominates | Vector grain | Fewer output memory references than fine grain in simulation [arxiv:1705.08922 tab. 3]. Simulation only, not measured on silicon | | Compressing a pretrained LLM with little original data and no task-specific tuning | Dependency-group structural pruning with LoRA recovery | Needs 10 calibration samples and an instruction set for recovery, and the authors report 94.97 percent of baseline at 20 percent removal [arxiv:2305.11627 tab. 1]. Above 20 percent the results are weaker [arxiv:2305.11627 app. C.5] | | Edge LLM with strict memory and moderate compute | Structured pruning with weight-only quantization | The survey's qualitative advice, with no measured numbers [arxiv:2410.11845 §3.6]. For measured results see the LLM-pruning rows | | One model must serve many devices with different latency limits | Once-for-all sub-networks | One training run, then per-device search without retraining [arxiv:1908.09791 §3.4]. Training costs about 1200 GPU hours on V100 [arxiv:1908.09791 tab. 1] | | Architecture search with a limited GPU budget | BayesNAS-style pruning of the search graph | Gradient-based search with a posterior prune rule, reported at about 0.2 GPU days on CIFAR-10 [arxiv:1905.04919 tab. 1]. See the contradiction below on search cost | | Object detector on a Cortex-M7 with about 1 MB of flash | Iterative L1 pruning in the last part of training, then 8-bit post-training quantization | Reports 3.46 FPS, 752 KB flash and 324 KB RAM at 128 x 128 [arxiv:2408.15865 tab. 2]. Accuracy is modest and task-dependent, with no baseline detector [arxiv:2408.15865 §3.1] | | Hard ROM, RAM and FLOPs limits, and budget for about 150 full training runs | Search over pruning with training and quantization settings | Pruning is searched rather than fixed. The authors report higher hypervolume than five baselines on two tasks, in a figure only [arxiv:2305.14109 §5.1] [arxiv:2305.14109 fig. 2] | | Shipping or streaming a 3D Gaussian scene | Redundancy and opacity pruning with quantisation | Tens of times smaller files at small PSNR change in the authors' tests [arxiv:2406.17074 tab. 1] | | Pruning a GPT-family model where retraining is too costly, with a small generic text set for calibration | One-shot pruning with a weight update (SparseGPT) | No retraining. Reports 8.21 against 8.35 dense perplexity at 50 percent on OPT-175B [arxiv:2301.00774 tab. 1]. Needs one 80 GB GPU for hours, and the speed numbers are for matrix products or CPU runs [arxiv:2301.00774 app. E] | | Need a sparse model trained from scratch, with no dense model first | Drop-and-grow rewiring (RigL) | Fixed sparse training cost [arxiv:1911.11134 §3]. Reports 74.6 percent top-1 at 80 percent sparsity and 0.23 times the dense training FLOPs [arxiv:1911.11134 fig. 2]. Behind magnitude pruning on character-level WikiText-103 [arxiv:1911.11134 §4.2] | | Repairing a model of unknown origin for a backdoor, with a small clean set and no fine-tuning | Adversarial neuron pruning (ANP) | Single pruning pass, no fine-tuning. ASR below 1 percent on five of six CIFAR-10 attacks with 500 clean images, at a 1.47-point mean ACC drop [arxiv:2110.14430 tab. 1]. It is a repair method, not a compression one, and its numbers come from CIFAR-10 with ResNet-18 | | Choosing among compression families for an edge device | Read the survey first | Maps the families and points to original papers; it gives no recipe [arxiv:2605.06714 §4] | Constraints that these results do not cover: training memory, which the 3D scene paper says does not drop during training [arxiv:2406.17074 §5]; latency on the paper's target hardware, which the Gaussian paper measures only as rasteriser frame rate [arxiv:2406.17074 §6]; latency for LLM pruning, which the LLM paper measures on one setup only [arxiv:2305.11627 tab. 3]; and any result beyond the datasets, models and simulator the papers used. The RigL numbers are author-reported at ImageNet and CIFAR-10 scale, and its pruning baselines are numbers from other papers, not reruns [arxiv:1911.11134 fig. 2]. The microcontroller detector's deployed numbers come from the pruned and quantized model together, and the search paper's hypervolume comparison has no numbers in its text [arxiv:2408.15865 tab. 2] [arxiv:2305.14109 §5.1]. The ANP backdoor results cover one architecture and dataset, and the released code runs only some of the attacks in the paper's table [arxiv:2110.14430 tab. 1] [code: train_backdoor_cifar.py#L29-L31]. ## Implementation sketch Common core of magnitude pruning. The variants differ in how `groups` is built and in whether the score is recomputed between stages. The LLM variant builds its groups from the dependency graph and sums per-structure gradient scores over each group [arxiv:2305.11627 §3.1] [arxiv:2305.11627 §3.2]. ```pseudocode # Magnitude pruning of one layer at a chosen grain [arxiv:1705.08922 §3.3] # Input: weights (tensor), grain in {single, vector, kernel, filter}, # fraction_removed in [0, 1) # Output: boolean mask, True = keep def magnitude_mask(weights, grain, fraction_removed): # Cut the tensor into grains of the chosen size [arxiv:1705.08922 §3.2] groups = split_into_grains(weights, grain) # Score each grain by the sum of absolute values [arxiv:1705.08922 eq. 1] scores = [sum(abs(w) for w in g) for g in groups] # Remove the lowest-scoring share; the paper's wording is ambiguous, so # confirm the reading before implementing [arxiv:1705.08922 §3.3] n_remove = round(fraction_removed * len(groups)) removed = argsort(scores)[:n_remove] return build_mask(groups, removed) ``` The gradual schedule used by the microcontroller and tiny-model entries sets how much to remove at each step. It is separate from the score above. ```pseudocode # Target sparsity at step t of a cubic gradual schedule [arxiv:2311.11883 eq. 6] # Input: step t; s0, sf in [0, 1]; t0 start step; n steps; dt interval # Output: sparsity to reach at step t def target_sparsity(t, s0, sf, t0, n, dt): # Share of the pruning window already passed [arxiv:2311.11883 eq. 6] progress = (t - t0) / (n * dt) # Cubic decay: most of the cut happens early in the window [arxiv:2311.11883 §4.2] return sf + (s0 - sf) * (1 - progress) ** 3 ``` The survey does not say how to clamp `progress` outside the window, or whether `dt` counts steps or epochs; see the contradictions below. The BayesNAS keep rule replaces the score and the cut: ```pseudocode # Keep rule for one edge of a one-shot search graph [arxiv:1905.04919 §5.3] # Input: prior variance gamma of the edge # Output: True = keep the edge def keep_edge(gamma): # Entropy of a zero-mean Gaussian is 0.5 * ln(2 * pi * e * gamma) entropy = 0.5 * log(2 * pi * e * gamma) # Remove the edge when the entropy is not positive [arxiv:1905.04919 §5.3] return entropy > 0 ``` RigL is not a cut of a trained model, so its update is a separate block below. ```pseudocode # One RigL mask update for one layer [arxiv:1911.11134 §3, alg. 1] # Input: weights, mask (bool), dense_grad (same shape, zeros included), # k (connections to replace this step) # Output: updated weights and mask def rigl_update(weights, mask, dense_grad, k): # Drop the k active weights of smallest magnitude [arxiv:1911.11134 §3] dropped = smallest_k(abs(weights), k, where=mask) mask[dropped] = False # Grow among positions inactive after the drop; dropped ones may return # [arxiv:1911.11134 §3] grown = largest_k(abs(dense_grad), k, where=~mask) # New connections start at zero, so the output does not jump # [arxiv:1911.11134 §3] weights[grown] = 0 mask[grown] = True return weights, mask ``` ## Contradictions - **Memory-reference savings in one paper.** The conclusion says coarse grain saves about twice the output memory accesses of fine grain [arxiv:1705.08922 §8]. The VGG-16 table gives 30 to 35 percent savings, roughly 1.4 to 1.5 times [arxiv:1705.08922 tab. 3]. The AlexNet table gives 60.5 percent of dense references for fine grain and 34.6 percent for vector grain, about 1.75 times [arxiv:1705.08922 tab. 4]. The three numbers come from different models and setups; use the table values. - **Search cost in BayesNAS.** The table reports 0.2 GPU days [arxiv:1905.04919 tab. 1]. The experiments section says one search takes about 3 hours on a single GPU [arxiv:1905.04919 §6.1], which is about 0.125 GPU days. The paper does not resolve this. - **Size headline in the 3D scene paper.** The introduction gives about 27 times smaller files [arxiv:2406.17074 §1]. The per-dataset ratios in the table are 25.7, 27.6 and 34.8 [arxiv:2406.17074 tab. 1], and the paper does not state how the headline is computed. - **Paper and code on the base-colour test.** The paper's threshold quantity is a weighted sum of squared deviations with a threshold of 0.04 [arxiv:2406.17074 §4.2]. The card records that the released code takes a square root and averages over channels before comparing to the threshold, so the two are not the same quantity. - **Search time for the LLM pruner.** The abstract gives about 3 hours on a single GPU [arxiv:2305.11627 §1]. The appendix gives about 2.5 hours on one RTX 4090 for the recovery step [arxiv:2305.11627 app. B.2]. The two are not the same measurement, and the paper does not reconcile them. - **Block order in SparseGPT.** The paper says layers are pruned sequentially, block by block, with pruned outputs carried forward [arxiv:2301.00774 §4]. The [paper card](/papers/2301.00774) records that the released code takes all Hessians of one block from a single forward pass before pruning any linear layer in it. The two readings give different Hessians and so different masks. The paper does not settle this. - **Learning rates in the once-for-all paper.** The paper gives absolute learning rates for the shrinking stages [arxiv:1908.09791 app. B]. The released code sets a per-GPU base rate and multiplies by the GPU count [code: train_ofa_net.py#L46] [code: train_ofa_net.py#L164]. The two agree only at 32 GPUs, and the paper does not say how the rates scale. - **Units of the pruning schedule in one survey.** The caption of its pruning figure says the rate is over epochs, but its equation indexes steps, and the interval has no stated unit [arxiv:2311.11883 fig. 6] [arxiv:2311.11883 eq. 6]. - **Clean-data size in the ANP paper.** The appendix says ANP always keeps ASR below 5 percent with 0.1 percent of the clean data [arxiv:2110.14430 app. C.3]. Its table gives an ASR of 5.70 for Blend at that size [arxiv:2110.14430 tab. 9]. The paper does not reconcile them. - **Noise budget in the ANP paper and code.** The algorithm clips the weight and bias noise back to the budget after each ascent step [arxiv:2110.14430 alg. 1, lines 8-9]. The released code starts the noise uniformly in the budget, takes one signed step of the full budget, and never clamps the noise, so one step can reach twice the budget [code: models/anp_batchnorm.py#L26-L33] [code: optimize_mask_cifar.py#L142-L145]. Its effect on the reported numbers is unknown. - **Head pruning share in one survey.** Its pipeline gives 30 to 40 percent of attention heads [arxiv:2601.03290 §VIII-H]. A cited BERT result gives 95 to 97 percent retention at 40 to 50 percent pruning [arxiv:2601.03290 §III-B]. The two ranges come from different statements, and the survey does not reconcile them. ## Papers - [Exploring the Regularity of Sparse Structure in Convolutional Neural Networks](/papers/1705.08922): compares weight, vector, kernel and filter pruning at matched density, storage and simulated memory references. - [BayesNAS: A Bayesian Approach for Neural Architecture Search](/papers/1905.04919): prunes a one-shot search graph by posterior variance rather than weight magnitude. - [Reducing the Memory Footprint of 3D Gaussian Splatting](/papers/2406.17074): prunes redundant and low-opacity Gaussians alongside band reduction and codebooks for smaller scenes. - [LLM-Pruner: On the Structural Pruning of Large Language Models](/papers/2305.11627): removes dependency-coupled groups of LLaMA-family weights, scored by gradient estimates, then recovers with LoRA. - [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](/papers/2301.00774): prunes GPT-family layers in one pass without retraining, using a Hessian-based mask and weight update. - [Once-for-All: Train One Network and Specialize it for Efficient Deployment](/papers/1908.09791): trains one weight-sharing network whose sub-networks are selected per device, with channel ranking for width. - [Rigging the Lottery: Making All Tickets Winners](/papers/1911.11134): trains a fixed-sparsity network from a random start, rewiring it every 100 iterations by drop-smallest and grow-largest-gradient. - [microYOLO: Towards Single-Shot Object Detection on Microcontrollers](/papers/2408.15865): prunes a small detector with iterative L1 pruning in the last part of training, then quantizes it to 8 bits for a Cortex-M7. - [Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML](/papers/2305.14109): treats pruning schedule and sparsity as searched settings next to training and quantization settings. - [Adversarial Neuron Pruning Purifies Backdoored Deep Models](/papers/2110.14430): learns a per-neuron mask under adversarial weight and bias perturbation, then prunes low-mask neurons to remove a backdoor without fine-tuning. - [Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review](/papers/2311.11883): review of tiny-model compression with a cubic gradual magnitude pruning schedule and a survey of structured and Bayesian sparsity. - [A Review on Edge Large Language Models: Design, Execution, and Applications](/papers/2410.11845): survey that separates structured and unstructured pruning for edge LLMs, with qualitative selection advice. - [Lightweight Transformer Architectures for Edge Devices in Real-Time Applications](/papers/2601.03290): survey with a pruning step in a six-step pipeline; its numbers are not backed by a stated protocol. - [A Machine Learning-oriented Survey on Tiny Machine Learning](/papers/2309.11932): survey that places pruning among tiny-model optimization methods. - [TinyML: Tools, Applications, Challenges, and Future Research Directions](/papers/2303.13569): survey that quotes a pruning-plus-quantization speech model with reported size reduction. - [Optimizing Neural Network for Computer Vision task in Edge Device](/papers/2110.00791): mentions structured pruning of a gesture model without quantifying it. - [The edge-vision survey](/papers/2605.06714): surveys pruning and other edge compression routes with quoted, not reproduced, results.