Activation sparsity

Overview

In a feed-forward block of a Transformer LLM, most neurons produce little or no output for a given token. Activation sparsity means computing only the neurons that matter for the current input and skipping the rest. The PowerInfer paper argues that this sparsity is not random: a fixed core of neurons fires across many inputs, and a long tail fires rarely 2312.12456 §3.1. The paper reports that this pattern is strongest in ReLU-family models and weaker in SiLU-family models 2312.12456 §4, §8.4.

The shared mechanism has three parts. A profile on general text finds which neurons are hot 2312.12456 §4.1, §6.1. A small predictor per layer guesses the active neurons for the current input 2312.12456 §5.1. An engine then computes only the predicted-active neurons, on whichever device holds them 2312.12456 §5.3. So far only one source in the library covers this concept, so the variants below are those of a single paper.

Variants

Hot and cold neuron split with per-layer predictors

Neurons are ranked by how often they fire on profiling text. The frequent ones are placed on the GPU and the rest on the CPU. Each layer has an MLP predictor that scores neurons for the current token, and only predicted-active neurons run 2312.12456 §4, §5.1. Placement is chosen by an integer linear program over activation counts and memory limits 2312.12456 §6.3. The reported result is a speed-up for batch size 1 on one consumer GPU: an average of 7.23x over llama.cpp on an RTX 4090 for FP16 models, with up to 11.69x for Falcon-40B 2312.12456 §8.2, fig. 10. Card: PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU.

When to use which

Implementation sketch

The core is the same for any variant: split the active set by placement, then compute each part on its device. The sketch follows the paper's description of the hybrid FFN step 2312.12456 §4.2, §5.3.

# Predict the active set for one FFN block  [arxiv:2312.12456 §5.1]
# Input:  x (d,) FFN input; predictor weights per layer
# Output: active (f,) bool, one flag per FFN neuron

def predict_active(x, predictor):
    # Small MLP with one hidden layer and ReLU  [arxiv:2312.12456 §5.1]
    scores = predictor(x)                              # (f,)
    return scores >= threshold                         # threshold: see below

# Compute the block on a split placement  [arxiv:2312.12456 §4.2, §5.3]
# Input:  x (d,); active (f,) bool; hot_mask (f,) bool from offline profile
# Output: y (d,)

def hybrid_ffn(x, active, hot_mask):
    # Hot active neurons run on the GPU  [arxiv:2312.12456 §4.2]
    gpu_part = neuron_rows(x, active & hot_mask)       # GPU
    # Cold active neurons run on the CPU  [arxiv:2312.12456 §5.3]
    cpu_part = neuron_rows(x, active & ~hot_mask)      # CPU
    # The CPU result is copied and merged on the GPU  [arxiv:2312.12456 §5.3]
    return gpu_part + cpu_part

Implementation notes that vary by variant:

• The predictor threshold is not stated in the paper 2312.12456 §5.1. Any value must be set on the predictor's own score scale.
• The placement solver is an integer linear program over activation counts 2312.12456 §6.3. A simpler greedy ranking would need checking against the paper's constraints.
• The paper's predictor sizing is iterative and stops at a perplexity within 0.1% of the dense baseline on WikiText-2 2312.12456 §5.1.

Contradictions

• Share of neurons active per token. The paper says about 10% of MLP neurons are active per iteration in one place 2312.12456 §3.2, and about 20% per token for OPT-30B in the evaluation 2312.12456 §8.2. The paper does not reconcile these.
• Predictor size. The paper gives about 6% of parameters 2312.12456 §5.1, a 7.09% figure in its overhead section 2312.12456 §8.3.4, and a per-model range of 4.88% to 8.68% 2312.12456 tab. 6. The table range is the most specific of the three.
• A100 comparison. The abstract reports 82% of an A100's token rate for OPT-30B 2312.12456 abs.. The A100 section reports an 18% slowdown for the same model 2312.12456 §8.3.5. These are one measurement framed two ways.
• Released code. The released implementation differs from the paper in several places, including the batch size for the solver, the constraints used, and the activation functions it supports. These are listed on the card's Paper vs code section. Reproductions should follow the code's behaviour where it matters.

Papers

• PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU: hot and cold neuron split for consumer-GPU inference, with per-layer MLP predictors and an offline placement solver.