CHIP: CHannel Independence-based Pruning for Compact Neural Networks

In plain words

Pruning removes parts of a neural network so that it is smaller and faster to run. A common way to choose what to remove is to look at each filter's own output, one channel at a time. This paper looks at how the channels relate to each other instead. If the output of one channel can be rebuilt from the outputs of the others, that channel adds little, and its filter can be removed.

The method measures this with a single score per channel, computed once from a small batch of training images. It then removes the filters with the lowest scores, copies the remaining weights into a smaller network, and fine-tunes it. The authors report that accuracy is kept or even slightly raised on CIFAR-10 and ImageNet while size and compute drop, but these numbers come from the authors.

TL;DR

CHIP ranks the filters of each convolutional layer by "channel independence", the drop in the nuclear norm of the layer's stacked feature maps when one channel is zeroed. It keeps the top-scoring filters, copies their weights, and fine-tunes with SGD. The scoring is one-shot: it needs about 640 images and no training. The code is public but has no license file, and it does not run the VGG-16 setting as shipped. Use the paper for the idea and the code for the exact settings.

Problem

Filter pruning gives speedups on ordinary hardware because it removes whole filters, but the choice of which filters to remove decides accuracy §1. Earlier feature-guided methods score each feature map using only that map's own information (its rank, its scaling factor, or a geometric median over filters) §1 §3.1. The authors argue that information shared across channels is under-used, and that it gives a more stable and more global ranking §1 §3.1.

Method

Setup. For layer l with c^l output channels, the feature maps are a tensor \mathcal{A}^l of shape c^l \times h \times w. Feature-guided pruning chooses which maps to keep and removes the matching filters §2, eq. 2. CHIP reshapes the maps into a matrix with one row per channel and measures how independent each row is §3.2.

Per-channel score. Let A^l be the c^l \times hw matrix and M_i^l the mask that zeroes row i. The channel independence of row i is the nuclear-norm drop when that row is removed §3.2, eq. 3:

CI(A^l_i) = \|A^l\|_* - \|M^l_i \odot A^l\|_* eq. 3

Nuclear norm was chosen over rank because rank changes barely distinguish rows, while the nuclear norm does §3.2 fig. 2.

Several channels at once. Exhaustive search over all subsets to remove m of c^l channels is intractable; the paper gives the cost as O(C(N,\kappa)), which the approximation reduces to O(N) §3.2. The approximation sums the single-channel scores §3.2, eq. 4:

CI(\{A^l_{b_i}\}_{i=1}^m) \approx \sum_{i=1}^m CI(A^l_{b_i}) eq. 4

Stability across inputs. The per-channel score is averaged over batches of input images. The paper reports that the average is stable from batch to batch §3.2 fig. 4, and gives Pearson correlations between batch vectors for one ResNet-56 layer §6.4 tab. 4.

Procedure (one shot). Algorithm 1 runs per layer alg. 1. The code runs all layers from the same pretrained model, then fine-tunes the pruned network.

# CHIP channel scoring and selection for one layer  [alg. 1]
# Input:  feature maps (N, C, H, W) from N samples; number to keep K
# Output: sorted indices of the K kept channels

def select_channels(feature_maps, keep):
    # Flatten each sample to a (C, H*W) matrix, one row per channel  [alg. 1]
    per_sample_ci = []
    for sample in feature_maps:
        matrix = reshape(sample, (C, H * W))
        full_norm = nuclear_norm(matrix)
        sample_ci = zeros(C)
        for i in range(C):
            # Zero row i and measure the drop in nuclear norm  [§3.2] [eq. 3]
            masked = matrix with row i set to 0
            sample_ci[i] = full_norm - nuclear_norm(masked)
        per_sample_ci.append(sample_ci)
    # Average the scores over the samples  [alg. 1] [§4.1]
    ci = mean(per_sample_ci, axis=0)
    # Keep the K channels with the largest scores; remove the rest  [alg. 1]
    keep_idx = argsort(ci)[C - K:]
    return sorted(keep_idx)

The procedure, step by step:

1. Run the pretrained network on 5 batches of 128 training images (640 images) and record each layer's feature maps §4.1 calculate_feature_maps.py calculate_feature_maps.py.
2. For each layer, compute the score of every channel per image and average over images and batches calculate_ci.py calculate_ci.py.
3. Keep the \kappa^l highest-scoring filters of each layer; the rest are removed alg. 1 prune_finetune_cifar.py.
4. Build the smaller network: copy the kept filters, and keep only the input channels that survive from the previous layer prune_finetune_cifar.py prune_finetune_cifar.py.
5. Fine-tune with SGD alg. 1 §4.1.

The paper gives no per-layer keep ratios in its text. The configurations used for the table are in the README, as sparsity strings per layer README.md README.md.

Equations and figures. Eq. 1 is the general pruning objective §2, eq. 1. Eq. 2 is the feature-guided objective §2, eq. 2. Figures: fig. 1 contrasts intra- and inter-channel views §1; fig. 2 compares rank and nuclear-norm change §3.2; fig. 3 shows the pipeline example §3.2; fig. 4 shows batch stability §3.2.

Hyperparameters

Results

Author-reported. Setup: CIFAR-10 (ResNet-56, ResNet-110, VGG-16) and ImageNet (ResNet-50), fine-tuning as above §4.1. The number of runs and the variance are not stated. Parameters and FLOPs are as printed in the paper tab. 1 tab. 2.

CIFAR-10 tab. 1:

The paper's closest comparison rows at similar size tab. 1: HRank on ResNet-56 at 0.27M (68.1%) reaches 90.72 (−2.54); HRank on ResNet-110 at 0.53M (68.7%) reaches 92.65 (−0.85); HRank on VGG-16 at 1.78M (92.0%) reaches 91.23 (−2.73) tab. 1.

ImageNet, ResNet-50 tab. 2, baseline top-1 76.15, top-5 92.87:

Comparisons in the text §4.3: CHIP matches SCOP's top-1 in the high-compression region (75.26) and is ahead of SCOP in the moderate region, with the text giving +0.12 (see Pitfalls).

Paper vs code

Repository: Eclipsess/CHIP_NeurIPS2021, cloned at commit 418041b7805b88efb5c7cdd08b0b8050dc965335. The paper's abstract links this repository. The README says the code is based on HRankPlus README.md. No LICENSE file is in the repository.

• Weight decay (CIFAR-10). Paper 0.05 §4.1; code default and README 0.005 prune_finetune_cifar.py README.md.
• Epochs and batch (CIFAR-10). Paper 300 epochs, batch 128 §4.1; README 400 epochs, batch 256 README.md; code defaults 200 epochs, batch 256 prune_finetune_cifar.py.
• Momentum and learning rate (ImageNet). Paper momentum 0.99 and LR 0.1 §4.1; code default momentum 0.9 prune_finetune_imagenet.py; README momentum 0.99 but LR 0.01 README.md.
• Label smoothing (ImageNet). Not in the paper. The README and the code default use 0.1 README.md prune_finetune_imagenet.py, and the training loop always uses the smoothed loss prune_finetune_imagenet.py.
• Learning-rate schedule. Not in the paper. Code uses cosine decay with a 5-epoch linear warm-up prune_finetune_cifar.py prune_finetune_cifar.py.
• Feature maps. The paper defines A^l as the layer's output feature maps §2, eq. 2 without saying before or after the activation. The code hooks the ReLU modules, so it uses post-activation outputs calculate_feature_maps.py.
• Score computation. Rows are rounded to 4 decimals before the nuclear norms are computed calculate_ci.py. Not in the paper.
• Residual blocks. Stage-output filters of the residual convolutions are pruned in stages 2 and 3 in the README configurations resnet_cifar10.py README.md. The identity shortcut is zero-padded resnet_cifar10.py, and the loader selects each second-convolution's filters by its own score prune_finetune_cifar.py. The paper does not discuss this. I did not run the model to check whether the kept channels of the residual path line up.
• ResNet-50 hook list. For the first block of each stage, the relu3 hook is registered a second time, so four of the 53 saved feature-map layers come from a second forward pass calculate_feature_maps.py calculate_ci.py.
• VGG-16. The paper reports VGG-16 results tab. 1, and the README lists them README.md. The shipped prune and feature-map scripts only import the ResNet models prune_finetune_cifar.py calculate_feature_maps.py, so eval(args.arch) cannot build VGG-16 from these scripts prune_finetune_cifar.py. I read this from the imports and did not run it.
• ResNet-56 row. The README row at 0.24M gives 70.0% parameter and 74.1% FLOP reduction, 34.78M FLOPs and 92.43% top-1 README.md; the paper's row at the same size reports 71.8%, 72.3%, 92.05% tab. 1.
• ResNet-110 row. README 0.54M at 68.3% gives 93.23% README.md; paper 93.63% tab. 1.
• ImageNet rows. README 40.8% gives 76.41% top-1 README.md; paper 76.30%. README 44.2% gives 76.35% README.md; paper 76.15%. The README says some results are better than the paper README.md.
• Mask learning. The supplement describes an extra learning stage for the pruning mask (no gain reported) §6.5. The repository has no script for it.
• Parameter and FLOP counting. The CIFAR script counts with thop on a 32×32 input prune_finetune_cifar.py. The paper does not state its counting method.

Pitfalls

• Which feature maps. The score depends on whether maps are taken before or after the ReLU. The code uses post-ReLU outputs calculate_feature_maps.py. Match the code unless you have a reason not to.
• Scores come from the unpruned network. All layers are scored from the pretrained model's feature maps, and the script then prunes every layer at once calculate_feature_maps.py prune_finetune_cifar.py. No layer is re-scored after upstream pruning. The paper's Algorithm 1 is written per layer, and it does not say which feature maps it uses.
• Cost. The score needs one nuclear norm per channel per sample, each an SVD of a C \times HW matrix, computed in a Python loop calculate_ci.py. The paper's O(N) count refers to channels; it does not count the SVD cost. On large layers this is the expensive part, and the paper gives no timing.
• Calibration data. The CIFAR calibration pass uses the augmented training loader, and the sampling is shuffled with no seed set calculate_feature_maps.py cifar10.py.
• Hard-coded path. The score reader expects _repeat5 in its directory name calculate_ci.py, so changing --repeat for the feature-map step breaks it.
• Test-set model selection. The CIFAR "validation" loader is the test split cifar10.py, and the script keeps the best top-1 across epochs prune_finetune_cifar.py. The paper does not say whether it reports the final or the best epoch. Expect best-of-epochs numbers unless you change the script.
• Text versus table. The text gives a +0.12 top-1 margin over SCOP at moderate compression §4.3. The table row for SCOP at 42.8% params gives 75.95, and CHIP's 76.30 is +0.35 above it tab. 2.
• Per-layer ratios. The paper gives no per-layer keep ratios. Using only the paper's averages, you cannot reproduce the table. Use the README strings README.md README.md.
• Batch stability. The reported correlations between batch-level score vectors are for one ResNet-56 layer with 64 channels §6.4 tab. 4. They do not cover every layer.
• Worked example. The 3×4 example in the supplement gives per-row scores §6.1 tab. 3. I did not recompute these values.

Lineage

• Builds on: feature-guided pruning, in particular HRank §1 §4.2 and SCOP §1 §4.3. The repository is based on HRankPlus README.md. The paper's bibliography gives no arXiv identifiers for these works, so they are not linked.
• Compared against: ℓ1-norm, NISP, GAL, LASSO, HRank, SCOP, FPGM, Taylor, PFP, ThiNet, SFP, Autopruner, C-SGD, RRBP and SSS, depending on dataset tab. 1 tab. 2.
• Superseded by: not stated.