Network pruning

Overview

Pruning scores each removable unit, cuts the lowest-scoring ones at a target share or threshold, and keeps the rest 1705.08922 §3.3. Most variants below share that loop. They differ in the unit that is removed, the score, and whether the cut is followed by retraining. The unit matters most: a single weight and a whole filter may save the same number of parameters while behaving very differently on real hardware 1705.08922 §1. Two entries below do not cut a trained model. Once-for-all training trains smaller sub-networks that share weights and selects one of them per target 1908.09791 §3.3. RigL trains a sparse network from a random start and rewires it during training, so no dense model is ever trained 1911.11134 §1. Two more entries put a cut inside a larger pipeline: a gradual schedule for a microcontroller detector 2408.15865 §2, and a search that treats pruning as one of its settings 2305.14109 tab. 1. A learned-mask variant prunes neurons to remove a backdoor, without fine-tuning the network afterwards 2110.14430 §3.3 2110.14430 §4.2.

Variants

Magnitude pruning at four granularities

Exploring the Regularity of Sparse Structure in Convolutional Neural Networks prunes convolution weights by the L1 norm of each grain and compares four grain sizes under one method and one schedule, so that only the grain changes 1705.08922 §3.2 1705.08922 §4. The grains are single values, sub-kernel vectors, kernels and filters.

• Accuracy at matched density: on five ImageNet CNNs, fine-grained pruning has the highest top-5 accuracy in every row. Kernel pruning trails it by 0.27 to 1.21 points and vector pruning by 0.08 to 0.47 points 1705.08922 tab. 1.
• Storage at the baseline accuracy: the ranking depends on the model. Fine grain has the lowest storage for AlexNet and DenseNet-121, vector grain for VGG-16, GoogLeNet and ResNet-50. Kernel grain has the highest storage in all five 1705.08922 tab. 2.
• Memory references: in a simulated sparse-dataflow accelerator on VGG-16 convolution layers, vector pruning needs 65 to 70 percent of the references of fine pruning 1705.08922 tab. 3.
• Filter grain is only described in the text. The paper reports that AlexNet loses about one point of validation accuracy at the first pruning stage 1705.08922 §4.

Dependency-aware Bayesian pruning of a search graph

BayesNAS uses pruning inside one-shot architecture search. Each candidate edge gets a prior whose variance combines the switches of its predecessor edges. The prior is updated with a Laplace approximation, and an edge is removed when its prior variance is at most 1/(2πe), the point where the entropy of a zero-mean Gaussian becomes non-positive 1905.04919 §5.3. The score is therefore a posterior quantity, not the raw weight magnitude 1905.04919 §1. The pruned network is then retrained from scratch 1905.04919 §6.1. The same machinery can also prune groups of convolution or fully connected weights, with a reweighted group-Lasso term, and the paper reports small MNIST and CIFAR-10 compression results with no run-to-run spread 1905.04919 app. S2.2.

Structural pruning of large language models

LLM-Pruner: On the Structural Pruning of Large Language Models targets LLaMA-family models and removes whole coupled groups of weights. It first follows dependency rules through the network, so that structures which break if one is removed are pruned as one group 2305.11627 §3.1. Each structure is scored from gradients on 10 calibration samples, using a first-order or second-order estimate of how the loss changes if it is zeroed. The scores are summed per group, and the lowest-scoring groups in each module are removed 2305.11627 §3.2 2305.11627 eq. 4 [arxiv:2305.11627 app. B.1]. Recovery trains low-rank adapters on about 50k instruction samples for two epochs, then merges them back into the weights 2305.11627 §3.3 2305.11627 app. B.2. On LLaMA-7B at 20 percent of parameters removed, the authors report 60.07 average zero-shot accuracy against a re-run dense baseline of 63.25, which is 94.97 percent, with tuning 2305.11627 tab. 1. Without tuning the same setting keeps 89.8 percent 2305.11627 tab. 1. Removing the dependency step is very costly: WikiText2 perplexity after tuning rises from 17.58 to 13307.46 2305.11627 tab. 6. At 50 percent the paper reports a strong drop that recovery only partly repairs 2305.11627 app. C.5.

One-shot pruning of a GPT-family model with a weight update

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot prunes every linear layer inside the decoder blocks of a GPT-family model in one pass, with no retraining 2301.00774 §1 2301.00774 §4.1. It works from a Hessian of each layer's inputs, estimated on 128 calibration segments of 2048 tokens from C4 2301.00774 §4. Unlike the cuts above, it does not only drop weights: after each removal, the surviving weights in the row are adjusted to offset the error 2301.00774 §3.1. The mask is chosen in blocks of 128 columns by a second-order score, the squared weight divided by the squared diagonal of the inverse Hessian, so sparsity can differ between columns 2301.00774 §3.2 2301.00774 alg. 1.

• Accuracy at 50 percent unstructured sparsity: on OPT-175B, raw-WikiText2 perplexity is 8.21 against 8.35 dense, and magnitude pruning reaches about 4.3e4 2301.00774 tab. 1. The zero-shot average is 70.52 against 70.29 dense 2301.00774 tab. 2. The authors call the 175B result a slight gain that depends on the dataset 2301.00774 §4.1 2301.00774 app. C.
• Hardware-friendly patterns: 2:4 gives 8.74 and 4:8 gives 8.45 on the same WikiText2 setting 2301.00774 tab. 1. The method sets the block size to the group size for these patterns 2301.00774 §3.3.
• Cost: a full 175B run takes about 4 hours on one A100 80GB 2301.00774 §4. The authors say the earlier accurate one-shot method, AdaPrune, takes about 1.3 hours at 350M and 4.3 hours at 1.3B, and extrapolate to hundreds of hours at 175B 2301.00774 §2 2301.00774 §4.1.
• Speed: on CPU, OPT-2.7B end-to-end runs 1.82 times faster at 50 percent sparsity 2301.00774 tab. 7. On an Ampere GPU, 2:4 matrix products of OPT-175B shapes run 1.54 to 1.79 times faster than dense cuBLAS 2301.00774 tab. 8. The paper gives no end-to-end GPU speed for the large models 2301.00774 app. E.
• Joint quantization: 50 percent sparsity with 4-bit weights gives 8.29 on OPT-175B, against 8.68 for GPTQ at 3 bits, which the authors compare at about the same storage 2301.00774 §4.1 2301.00774 app. C.

The paper's text prunes layers in order, block by block, and feeds the pruned outputs of each block into the next 2301.00774 §4. The card for this paper records a difference in how the released code collects Hessians inside a block; see the contradictions below before copying the order. The paper's update is written with entries of the inverse Hessian, but its algorithm uses the Cholesky factor of that inverse 2301.00774 alg. 1. Follow the algorithm for an implementation.

Once-for-all sub-networks instead of a cut

Once-for-All: Train One Network and Specialize it for Efficient Deployment trains one super-network whose sub-networks differ in depth, expand ratio, kernel size and input resolution 1908.09791 §3.2. The authors describe this as a generalised form of pruning, because it also shrinks depth, kernel size and resolution, not only width 1908.09791 §3.3. Width is reduced by keeping the channels ranked most important by L1 norm 1908.09791 §3.3. Training is progressive: the largest sub-network first, then smaller kernel, depth and width choices added in stages with distillation from the largest network 1908.09791 §3.3 1908.09791 app. B. A device then gets a sub-network chosen by a search over an accuracy predictor and a latency table, with no training per device 1908.09791 §3.4 1908.09791 app. A. Without fine-tuning, the specialized model reaches 76.0 percent ImageNet top-1 at 230M MACs, against 75.2 percent for MobileNetV3-Large at 219M MACs 1908.09791 tab. 1. Without progressive shrinking, the authors report 72.4 percent at 235M MACs 1908.09791 tab. 1.

Rewiring a sparse network during training

Rigging the Lottery: Making All Tickets Winners keeps a fixed number of weights active, with fixed parameter count and fixed training FLOPs, from the first step 1911.11134 §1 1911.11134 §3. Every 100 iterations it drops the active weights with the smallest magnitude and grows the same number at inactive positions with the largest dense gradient. New connections start at zero 1911.11134 §3 1911.11134 §4. The share replaced each time follows a cosine decay from an initial fraction of 0.3 1911.11134 §3 (2) 1911.11134 §4. Growth needs the dense gradient at update steps only, computed layer by layer 1911.11134 §3.

On ImageNet ResNet-50 at 80 percent sparsity, the authors report 74.6 percent top-1 at 0.23 times the dense training FLOPs. A static random mask reaches 70.6 and SET 72.9 at the same cost, and the dense model 76.8 1911.11134 fig. 2. With 5 times the training steps, RigL reaches 76.6 at 1.14 times the training FLOPs 1911.11134 fig. 2. The magnitude-pruning row in that figure is taken from another paper and was not rerun, and it reaches 75.6 at 0.56 times the training FLOPs 1911.11134 fig. 2. On character-level WikiText-103 the authors report that RigL does not match magnitude pruning [arxiv:1911.11134 §4.2]. On CIFAR-10 they report that it matches pruning with fewer training resources 1911.11134 §4.3. The paper card lists where the released code differs from the text, for example on how regrown weights are initialised.

Iterative L1 pruning at the end of training on a microcontroller

microYOLO: Towards Single-Shot Object Detection on Microcontrollers prunes every convolutional and linear layer of a small YOLO-style detector with an L1-norm criterion, in an iterative and gradual schedule 2408.15865 §2. In the detection runs the pruning covers only the last 100 of 400 epochs, every 20 epochs 2408.15865 §3. The target sparsity and the step size are not stated, and it is not stated whether the first cut falls on epoch 300 or 301 2408.15865 §2 2408.15865 §3. Pruning is followed by 8-bit post-training quantization, with no quantization-aware training 2408.15865 §2. On the fridge task the model reaches 56.4 percent mAP50 averaged over seeds, with no baseline detector reported 2408.15865 §3.1 2408.15865 fig. 1. The deployed fridge model needs 752 KB of flash and 324 KB of RAM, at 3.46 FPS on an OpenMV H7 R2 with a Cortex-M7 2408.15865 tab. 2. The authors report that the fridge task loses mAP50 during the pruning epochs 2408.15865 §3.1, and that more aggressive pruning reaches up to 8 FPS at a clearly lower mAP50 2408.15865 §3.3. The deployed model is the pruned and quantized one, so these numbers do not isolate the effect of pruning.

Pruning as one searched setting

Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML does not fix a pruning recipe. It searches pruning together with training and quantization settings, with a budget of 150 samples per run 2305.14109 §5. The pruning settings are a start fraction in [0.0, 0.6], an end fraction in [0.8, 0.95], between 1 and 20 steps, and a sparsity in [0.1, 0.99] for each prunable convolution, so the search space grows with depth [arxiv:2305.14109 tab. 1] 2305.14109 §5. The schedule is the automated gradual pruning of Zhu et al. 2305.14109 §5. Each sample trains a network, prunes and quantizes it, then generates code to measure its memory 2305.14109 §4. The paper reports the highest hypervolume after about 50 to 75 samples on two tasks, against five baselines, but the values are only in a figure 2305.14109 §5.1 2305.14109 fig. 2. One of its own designs uses 276 KB of RAM 2305.14109 tab. 3, above the 256 KB limit that the experiments state 2305.14109 §5.

Learned neuron masks to remove a backdoor

Adversarial Neuron Pruning Purifies Backdoored Deep Models repairs a model whose trigger is unknown, using a small clean set. It gives each neuron a mask in [0, 1], optimised while every neuron's weight and bias are perturbed in the worst-case direction, then prunes the neurons whose mask falls below 0.2 2110.14430 §3.3 2110.14430 §4.1. The premise is that backdoor-related neurons are dormant on clean inputs, so adversarial perturbation switches them on and the masks learn to drop them 2110.14430 §3.2. The mask objective weights clean loss against worst-case loss with a trade-off α, set to 0.2 on CIFAR-10 2110.14430 eq. 8 2110.14430 §4.1. In the paper's CIFAR-10 / ResNet-18 benchmark with 500 clean images, the attack success rate falls below 1 percent on five of six attacks, and CLB stays at 3.98 percent. The mean clean-accuracy drop is 1.47 points 2110.14430 tab. 1. Fine-tuning at learning rate 0.01 gives a mean accuracy drop of 3.53 points in the same table, and fine-pruning gives 2.00 2110.14430 tab. 1.

• Cost: no fine-tuning, so no weight changes beyond the pruned neurons 2110.14430 §4.2. The authors report about 2.5 times the run time of fine-tuning, with the hardware not named 2110.14430 tab. 2 2110.14430 §4.2.
• Clean data: with 10 percent of the training set, ACC stays at 93.01 to 93.86 percent across attacks. With 50 images, ACC falls to 85.66 to 89.54 percent 2110.14430 tab. 7 2110.14430 tab. 9.
• Sensitivity: the good threshold band is roughly 0.2 to 0.5 for ResNet-18, ResNet-50 and VGG-19, and MobileNet-V2 needs a smaller budget 2110.14430 §4.3 2110.14430 fig. 3. With 20 mask iterations, the ASR rises again after about 10 percent of neurons are pruned, and the effect goes away at 100 or more iterations 2110.14430 §4.4 2110.14430 fig. 4.

Taxonomy and results from surveys

Surveys run no experiments of their own, or quote results from other works. Their numbers are leads to the original papers, not measurements to copy.

The edge-vision survey runs no experiments of its own. It sorts pruning into weight, neuron, channel and filter forms, and quotes results from the works it cites 2605.06714 §4.1. Its quoted examples include a channel-pruned ResNet-50 with half the parameters at 99 percent of the original ImageNet accuracy, and a filter-pruned ResNet-50 that runs 2 times faster at 98.6 percent of the original 2605.06714 §4.1.3 2605.06714 §4.1.4. The numbers come from different setups and are not comparable with the rows above. Use the survey as a map to the original papers.

Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review calls magnitude pruning with a cubic sparsity schedule the main recipe for tiny models, and lists structured pruning of filters, neurons or layers as the alternative 2311.11883 §4.2. Its 90 percent sparsity with acceptable loss is a cited claim, not a measurement 2311.11883 §4.2. The schedule is given in the implementation sketch below.

A Review on Edge Large Language Models: Design, Execution, and Applications groups pruning into structured forms that remove heads, channels or layers, and unstructured forms that remove single weights 2410.11845 §3.2. Its qualitative advice is structured pruning for strict memory with moderate compute, and unstructured pruning for maximum compression with some accuracy loss tolerated 2410.11845 §3.6.

Lightweight Transformer Architectures for Edge Devices in Real-Time Applications lists a pipeline with structured pruning of 30 to 40 percent of attention heads and 20 to 30 percent of MLP channels 2601.03290 §VIII-H. It gives no protocol, dataset or sample sizes for these figures. The paper's own step reductions, compounded by our arithmetic, give about 3 to 3.6 times smaller models, not the 8 to 12 times it reports; the paper does not show how that was derived 2601.03290 §VIII-H.

A Machine Learning-oriented Survey on Tiny Machine Learning splits pruning into weight, neuron and structured variants, applied during or after training 2309.11932 §5.1 2309.11932 §5.1.1.

TinyML: Tools, Applications, Challenges, and Future Research Directions quotes a speech-enhancement RNN with structured pruning and integer quantization, at 11.9 times smaller size and 2.9 times fewer operations 2303.13569 §III-A1. Pruning and quantization are combined there, so the figure does not isolate pruning, and it is quoted from another work.

Mentioned without numbers

Optimizing Neural Network for Computer Vision task in Edge Device says a gesture model is structurally pruned on a representative dataset, but gives no ratio, no criterion and no before-and-after parameter count 2110.00791 §3. Use it as a pointer to the idea, not as evidence.

Pruning primitives of a 3D scene

Reducing the Memory Footprint of 3D Gaussian Splatting applies pruning to a scene of Gaussians rather than to network weights. A Gaussian counts as redundant when many others lie in a sphere sized by its smallest pixel footprint, and the lower-opacity half of the redundant ones is removed. The lowest 3 percent by opacity are removed as well, with the cut capped at opacity 0.05 2406.17074 §4.1. Pruning is combined with dropping view-dependent colour bands per Gaussian and with codebook quantisation 2406.17074 §4.2 2406.17074 §4.3. On its three dataset groups the paper reports 25.7 to 34.8 times smaller files, with PSNR moving between −0.32 and +0.16 dB against the authors' re-run baseline 2406.17074 tab. 1. The authors' own ablation keeps 43 percent of the primitives with all rules, 48 percent with the opacity rule only and 56 percent with the redundancy rule only 2406.17074 tab. 3. The paper's code is under a non-commercial licence (see the card), so implement from the paper's description.

When to use which

Constraints that these results do not cover: training memory, which the 3D scene paper says does not drop during training 2406.17074 §5; latency on the paper's target hardware, which the Gaussian paper measures only as rasteriser frame rate 2406.17074 §6; latency for LLM pruning, which the LLM paper measures on one setup only 2305.11627 tab. 3; and any result beyond the datasets, models and simulator the papers used. The RigL numbers are author-reported at ImageNet and CIFAR-10 scale, and its pruning baselines are numbers from other papers, not reruns 1911.11134 fig. 2. The microcontroller detector's deployed numbers come from the pruned and quantized model together, and the search paper's hypervolume comparison has no numbers in its text 2408.15865 tab. 2 2305.14109 §5.1. The ANP backdoor results cover one architecture and dataset, and the released code runs only some of the attacks in the paper's table 2110.14430 tab. 1 train_backdoor_cifar.py.

Implementation sketch

Common core of magnitude pruning. The variants differ in how groups is built and in whether the score is recomputed between stages. The LLM variant builds its groups from the dependency graph and sums per-structure gradient scores over each group 2305.11627 §3.1 2305.11627 §3.2.

# Magnitude pruning of one layer at a chosen grain  [arxiv:1705.08922 §3.3]
# Input:  weights (tensor), grain in {single, vector, kernel, filter},
#         fraction_removed in [0, 1)
# Output: boolean mask, True = keep

def magnitude_mask(weights, grain, fraction_removed):
    # Cut the tensor into grains of the chosen size  [arxiv:1705.08922 §3.2]
    groups = split_into_grains(weights, grain)
    # Score each grain by the sum of absolute values  [arxiv:1705.08922 eq. 1]
    scores = [sum(abs(w) for w in g) for g in groups]
    # Remove the lowest-scoring share; the paper's wording is ambiguous, so
    # confirm the reading before implementing  [arxiv:1705.08922 §3.3]
    n_remove = round(fraction_removed * len(groups))
    removed = argsort(scores)[:n_remove]
    return build_mask(groups, removed)

The gradual schedule used by the microcontroller and tiny-model entries sets how much to remove at each step. It is separate from the score above.

# Target sparsity at step t of a cubic gradual schedule  [arxiv:2311.11883 eq. 6]
# Input:  step t; s0, sf in [0, 1]; t0 start step; n steps; dt interval
# Output: sparsity to reach at step t

def target_sparsity(t, s0, sf, t0, n, dt):
    # Share of the pruning window already passed  [arxiv:2311.11883 eq. 6]
    progress = (t - t0) / (n * dt)
    # Cubic decay: most of the cut happens early in the window  [arxiv:2311.11883 §4.2]
    return sf + (s0 - sf) * (1 - progress) ** 3

The survey does not say how to clamp progress outside the window, or whether dt counts steps or epochs; see the contradictions below.

The BayesNAS keep rule replaces the score and the cut:

# Keep rule for one edge of a one-shot search graph  [arxiv:1905.04919 §5.3]
# Input:  prior variance gamma of the edge
# Output: True = keep the edge

def keep_edge(gamma):
    # Entropy of a zero-mean Gaussian is 0.5 * ln(2 * pi * e * gamma)
    entropy = 0.5 * log(2 * pi * e * gamma)
    # Remove the edge when the entropy is not positive  [arxiv:1905.04919 §5.3]
    return entropy > 0

RigL is not a cut of a trained model, so its update is a separate block below.

# One RigL mask update for one layer  [arxiv:1911.11134 §3, alg. 1]
# Input:  weights, mask (bool), dense_grad (same shape, zeros included),
#         k (connections to replace this step)
# Output: updated weights and mask

def rigl_update(weights, mask, dense_grad, k):
    # Drop the k active weights of smallest magnitude  [arxiv:1911.11134 §3]
    dropped = smallest_k(abs(weights), k, where=mask)
    mask[dropped] = False
    # Grow among positions inactive after the drop; dropped ones may return
    # [arxiv:1911.11134 §3]
    grown = largest_k(abs(dense_grad), k, where=~mask)
    # New connections start at zero, so the output does not jump
    # [arxiv:1911.11134 §3]
    weights[grown] = 0
    mask[grown] = True
    return weights, mask

Contradictions

• Memory-reference savings in one paper. The conclusion says coarse grain saves about twice the output memory accesses of fine grain 1705.08922 §8. The VGG-16 table gives 30 to 35 percent savings, roughly 1.4 to 1.5 times 1705.08922 tab. 3. The AlexNet table gives 60.5 percent of dense references for fine grain and 34.6 percent for vector grain, about 1.75 times 1705.08922 tab. 4. The three numbers come from different models and setups; use the table values.
• Search cost in BayesNAS. The table reports 0.2 GPU days 1905.04919 tab. 1. The experiments section says one search takes about 3 hours on a single GPU 1905.04919 §6.1, which is about 0.125 GPU days. The paper does not resolve this.
• Size headline in the 3D scene paper. The introduction gives about 27 times smaller files 2406.17074 §1. The per-dataset ratios in the table are 25.7, 27.6 and 34.8 2406.17074 tab. 1, and the paper does not state how the headline is computed.
• Paper and code on the base-colour test. The paper's threshold quantity is a weighted sum of squared deviations with a threshold of 0.04 2406.17074 §4.2. The card records that the released code takes a square root and averages over channels before comparing to the threshold, so the two are not the same quantity.
• Search time for the LLM pruner. The abstract gives about 3 hours on a single GPU 2305.11627 §1. The appendix gives about 2.5 hours on one RTX 4090 for the recovery step 2305.11627 app. B.2. The two are not the same measurement, and the paper does not reconcile them.
• Block order in SparseGPT. The paper says layers are pruned sequentially, block by block, with pruned outputs carried forward 2301.00774 §4. The paper card records that the released code takes all Hessians of one block from a single forward pass before pruning any linear layer in it. The two readings give different Hessians and so different masks. The paper does not settle this.
• Learning rates in the once-for-all paper. The paper gives absolute learning rates for the shrinking stages 1908.09791 app. B. The released code sets a per-GPU base rate and multiplies by the GPU count train_ofa_net.py train_ofa_net.py. The two agree only at 32 GPUs, and the paper does not say how the rates scale.
• Units of the pruning schedule in one survey. The caption of its pruning figure says the rate is over epochs, but its equation indexes steps, and the interval has no stated unit 2311.11883 fig. 6 2311.11883 eq. 6.
• Clean-data size in the ANP paper. The appendix says ANP always keeps ASR below 5 percent with 0.1 percent of the clean data 2110.14430 app. C.3. Its table gives an ASR of 5.70 for Blend at that size 2110.14430 tab. 9. The paper does not reconcile them.
• Noise budget in the ANP paper and code. The algorithm clips the weight and bias noise back to the budget after each ascent step [arxiv:2110.14430 alg. 1, lines 8-9]. The released code starts the noise uniformly in the budget, takes one signed step of the full budget, and never clamps the noise, so one step can reach twice the budget anp_batchnorm.py optimize_mask_cifar.py. Its effect on the reported numbers is unknown.
• Head pruning share in one survey. Its pipeline gives 30 to 40 percent of attention heads 2601.03290 §VIII-H. A cited BERT result gives 95 to 97 percent retention at 40 to 50 percent pruning 2601.03290 §III-B. The two ranges come from different statements, and the survey does not reconcile them.

Papers

• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks: compares weight, vector, kernel and filter pruning at matched density, storage and simulated memory references.
• BayesNAS: A Bayesian Approach for Neural Architecture Search: prunes a one-shot search graph by posterior variance rather than weight magnitude.
• Reducing the Memory Footprint of 3D Gaussian Splatting: prunes redundant and low-opacity Gaussians alongside band reduction and codebooks for smaller scenes.
• LLM-Pruner: On the Structural Pruning of Large Language Models: removes dependency-coupled groups of LLaMA-family weights, scored by gradient estimates, then recovers with LoRA.
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot: prunes GPT-family layers in one pass without retraining, using a Hessian-based mask and weight update.
• Once-for-All: Train One Network and Specialize it for Efficient Deployment: trains one weight-sharing network whose sub-networks are selected per device, with channel ranking for width.
• Rigging the Lottery: Making All Tickets Winners: trains a fixed-sparsity network from a random start, rewiring it every 100 iterations by drop-smallest and grow-largest-gradient.
• microYOLO: Towards Single-Shot Object Detection on Microcontrollers: prunes a small detector with iterative L1 pruning in the last part of training, then quantizes it to 8 bits for a Cortex-M7.
• Combining Multi-Objective Bayesian Optimization with Reinforcement Learning for TinyML: treats pruning schedule and sparsity as searched settings next to training and quantization settings.
• Adversarial Neuron Pruning Purifies Backdoored Deep Models: learns a per-neuron mask under adversarial weight and bias perturbation, then prunes low-mask neurons to remove a backdoor without fine-tuning.
• Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review: review of tiny-model compression with a cubic gradual magnitude pruning schedule and a survey of structured and Bayesian sparsity.
• A Review on Edge Large Language Models: Design, Execution, and Applications: survey that separates structured and unstructured pruning for edge LLMs, with qualitative selection advice.
• Lightweight Transformer Architectures for Edge Devices in Real-Time Applications: survey with a pruning step in a six-step pipeline; its numbers are not backed by a stated protocol.
• A Machine Learning-oriented Survey on Tiny Machine Learning: survey that places pruning among tiny-model optimization methods.
• TinyML: Tools, Applications, Challenges, and Future Research Directions: survey that quotes a pruning-plus-quantization speech model with reported size reduction.
• Optimizing Neural Network for Computer Vision task in Edge Device: mentions structured pruning of a gesture model without quantifying it.
• The edge-vision survey: surveys pruning and other edge compression routes with quoted, not reproduced, results.