--- kind: concept slug: activation-sparsity title: Activation sparsity summary: >- Skipping the neurons that stay near zero for a given input, so that only the active ones are computed or moved. The core idea is to find a stable set of frequently active "hot" neurons and to predict the active set per token. The trade-off: the gain depends on the activation function, and it needs an offline profile and a predictor per layer. tags: [llm-inference, sparsity, offloading, consumer-gpu] papers: ["2312.12456"] related: [llm-serving, on-device-inference, weight-quantization, network-pruning, mixture-of-experts, dynamic-sparse-training] status: draft updated_at: 2026-11-01 --- ## Overview In a feed-forward block of a Transformer LLM, most neurons produce little or no output for a given token. Activation sparsity means computing only the neurons that matter for the current input and skipping the rest. The PowerInfer paper argues that this sparsity is not random: a fixed core of neurons fires across many inputs, and a long tail fires rarely [arxiv:2312.12456 §3.1]. The paper reports that this pattern is strongest in ReLU-family models and weaker in SiLU-family models [arxiv:2312.12456 §4, §8.4]. The shared mechanism has three parts. A profile on general text finds which neurons are hot [arxiv:2312.12456 §4.1, §6.1]. A small predictor per layer guesses the active neurons for the current input [arxiv:2312.12456 §5.1]. An engine then computes only the predicted-active neurons, on whichever device holds them [arxiv:2312.12456 §5.3]. So far only one source in the library covers this concept, so the variants below are those of a single paper. ## Variants ### Hot and cold neuron split with per-layer predictors Neurons are ranked by how often they fire on profiling text. The frequent ones are placed on the GPU and the rest on the CPU. Each layer has an MLP predictor that scores neurons for the current token, and only predicted-active neurons run [arxiv:2312.12456 §4, §5.1]. Placement is chosen by an integer linear program over activation counts and memory limits [arxiv:2312.12456 §6.3]. The reported result is a speed-up for batch size 1 on one consumer GPU: an average of 7.23x over llama.cpp on an RTX 4090 for FP16 models, with up to 11.69x for Falcon-40B [arxiv:2312.12456 §8.2, fig. 10]. Card: [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](/papers/2312.12456). ## When to use which | Situation | Use | Why | |---|---|---| | Model is ReLU-family (ReLU, ReGLU) and its weights do not fit in one consumer GPU | Hot and cold split with per-layer predictors | The paper frames its design around this case and reports the largest gains for ReLU-family models [arxiv:2312.12456 §3.2, §4] | | Model is SiLU-family (SwiGLU-style) | Expect smaller gains and check on the target model first | About 50% sparsity is cited for this family, against much higher sparsity for ReLU-family models [arxiv:2312.12456 §4, §8.4] | | Weights fit in GPU memory on a data-centre GPU | Compare with dense serving first | On an A100 the paper's engine is reported 18% to 23% slower than vLLM at input length 1 [arxiv:2312.12456 §8.3.5, fig. 18] | | Batch size above one | Read the batch study before adopting | The reported batch results cover sizes below 32. The average speed-up there is 6.08x, and it is 4.38x at batch 32 [arxiv:2312.12456 §8.2, fig. 14] | | Limited time to adopt and no custom kernels | Plan for engineering effort | The reported engine is about 4,200 lines of C++ and CUDA added to llama.cpp, plus an offline profiler and solver [arxiv:2312.12456 §7] | | Predictor training is not affordable | Budget a one-time training run | The paper says predictor training often takes several hours and can be shortened with several GPUs [arxiv:2312.12456 §7] | ## Implementation sketch The core is the same for any variant: split the active set by placement, then compute each part on its device. The sketch follows the paper's description of the hybrid FFN step [arxiv:2312.12456 §4.2, §5.3]. ```pseudocode # Predict the active set for one FFN block [arxiv:2312.12456 §5.1] # Input: x (d,) FFN input; predictor weights per layer # Output: active (f,) bool, one flag per FFN neuron def predict_active(x, predictor): # Small MLP with one hidden layer and ReLU [arxiv:2312.12456 §5.1] scores = predictor(x) # (f,) return scores >= threshold # threshold: see below ``` ```pseudocode # Compute the block on a split placement [arxiv:2312.12456 §4.2, §5.3] # Input: x (d,); active (f,) bool; hot_mask (f,) bool from offline profile # Output: y (d,) def hybrid_ffn(x, active, hot_mask): # Hot active neurons run on the GPU [arxiv:2312.12456 §4.2] gpu_part = neuron_rows(x, active & hot_mask) # GPU # Cold active neurons run on the CPU [arxiv:2312.12456 §5.3] cpu_part = neuron_rows(x, active & ~hot_mask) # CPU # The CPU result is copied and merged on the GPU [arxiv:2312.12456 §5.3] return gpu_part + cpu_part ``` Implementation notes that vary by variant: - The predictor threshold is not stated in the paper [arxiv:2312.12456 §5.1]. Any value must be set on the predictor's own score scale. - The placement solver is an integer linear program over activation counts [arxiv:2312.12456 §6.3]. A simpler greedy ranking would need checking against the paper's constraints. - The paper's predictor sizing is iterative and stops at a perplexity within 0.1% of the dense baseline on WikiText-2 [arxiv:2312.12456 §5.1]. ## Contradictions - **Share of neurons active per token.** The paper says about 10% of MLP neurons are active per iteration in one place [arxiv:2312.12456 §3.2], and about 20% per token for OPT-30B in the evaluation [arxiv:2312.12456 §8.2]. The paper does not reconcile these. - **Predictor size.** The paper gives about 6% of parameters [arxiv:2312.12456 §5.1], a 7.09% figure in its overhead section [arxiv:2312.12456 §8.3.4], and a per-model range of 4.88% to 8.68% [arxiv:2312.12456 tab. 6]. The table range is the most specific of the three. - **A100 comparison.** The abstract reports 82% of an A100's token rate for OPT-30B [arxiv:2312.12456 abs.]. The A100 section reports an 18% slowdown for the same model [arxiv:2312.12456 §8.3.5]. These are one measurement framed two ways. - **Released code.** The released implementation differs from the paper in several places, including the batch size for the solver, the constraints used, and the activation functions it supports. These are listed on the [card's Paper vs code section](/papers/2312.12456). Reproductions should follow the code's behaviour where it matters. ## Papers - [PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU](/papers/2312.12456): hot and cold neuron split for consumer-GPU inference, with per-layer MLP predictors and an offline placement solver.