PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

In plain words

Running a large language model on a home computer is slow, because most of the model's weights do not fit in the memory of a consumer graphics card. The usual fix is to move some of the weights to ordinary system memory, but then every token pays for copying data over a slow link.

The paper's observation is that the work inside an LLM is lopsided. For any given word, only a small share of the feed-forward neurons does anything useful, and a fixed core of neurons does most of the firing across many different inputs. The system therefore keeps that frequently used core on the fast GPU, and leaves the rest in system memory where the CPU computes them directly. Small helper networks guess which neurons will fire for the current input, so the machine can skip the rest. A separate offline step decides which neurons belong on the GPU, using measurements taken from sample text.

The authors report that this runs much faster than llama.cpp on a single high-end consumer GPU, with the biggest gains for ReLU-style models, which fire the fewest neurons. Most of the evidence is in the authors' own benchmarks.

TL;DR

PowerInfer splits each feed-forward block between a GPU and a CPU by neuron §4. Neurons that are frequently active across profiled inputs are preloaded to the GPU ("hot"), and the rest are computed on the CPU ("cold") §3.1, §4. A small per-layer MLP predictor decides which neurons to compute for the current input §5.1. An offline integer linear program places neurons, using activation counts and per-unit cost §6, eq. 1–8. The method depends on strong activation sparsity: the authors say ReLU-family models fit best §4. The numbers are author-reported on two desktop machines, one with an RTX 4090 and one with an RTX 2080 Ti §8.1.

Problem

A dense LLM at batch size one is limited by how fast the weights can be read from memory. When the weights do not fit in GPU memory, either part of the model runs on the CPU (slow CPU compute) or weights are streamed to the GPU for each layer (slow transfers). The authors' motivation for the design rests on the costs of these two options §3.2. Approaches that predict sparse activations with fixed-size predictors either need large predictors in GPU memory or lose accuracy when the predictor is made small §5.1. The authors also state that existing sparse-matrix libraries and compilers do not handle dynamic, per-token neuron sparsity on a hybrid CPU-GPU setup well §5.4.

The design assumes that LLM activation is locality-heavy and that the ReLU-family models make it sparse. The authors report that for models with other activations, such as SiLU-based ones, sparsity is roughly 50% and the gain is smaller §4, §8.4.

Method

Insight 1: power-law activation. On about one million tokens of Wikipedia, 26%, 43% and 69% of MLP neurons account for 80% of activations in OPT-30B, LLaMA2 (ReGLU)-70B and LLaMA2 (SwiGLU)-70B. Across the whole model, the corresponding shares are about 17%, 26% and 75% §3.1, fig. 4. The authors call neurons that fire consistently "hot" and the rest "cold". Reported overlap of the top 20% most-active neurons across different task types is above 90% §3.1, fig. 5.

Insight 2: computing cold neurons on the CPU is faster than moving them. For batch sizes below 32, the authors measure that transferring a CPU-resident slice of weights and computing it on the GPU is slower than computing it directly on the CPU with AVX2 §3.2, fig. 6.

Offline profiling and placement. The profiler runs the model on general text (Wikipedia and C4 are named) and counts how often each neuron is activated §4.1, §6.1. A solver then decides which neurons go to the GPU. It uses the activation counts as each neuron's impact §6.2, eq. 1 and maximises the total impact placed on the GPU §6.3, eq. 2, subject to:

• each neuron on exactly one unit §6.3, eq. 3;
• a per-layer minimum number of GPU neurons, so that a layer is only split when the transfer overhead is worth paying §6.3.1, eq. 4;
• per-unit memory limits §6.3.2, eq. 6;
• an on/off indicator per layer that either allows zero GPU neurons or at least the minimum §6.3.2, eq. 7–8.

Neuron counts are grouped into batches of 64 neurons with similar impact, which the authors say reduces the problem from millions of neurons to tens of thousands and brings solve time to about 10 s §6.3.3.

# Offline placement of neurons, as an ILP  [§6.1–§6.3] [eq. 1–8]
# Input:  freq (L, f) activation counts per layer and neuron
#         gpu_budget (bytes), min_gpu_neurons (per layer C_l)
# Output: per-layer count of neurons to keep on the GPU

def place_neurons(freq, gpu_budget, min_gpu_neurons):
    # Each neuron's impact is its activation frequency  [§6.2, eq. 1]
    impact = freq                                      # (L, f)
    # Group neurons into batches of similar impact, 64 per batch  [§6.3.3]
    batches = group_by_impact(impact, batch_size=64)
    # Maximise impact on the GPU  [§6.3, eq. 2]
    # Each neuron on exactly one unit  [§6.3, eq. 3]
    # Layer split only if GPU count >= C_l  [§6.3.1, eq. 4; §6.3.2, eq. 7–8]
    # GPU bytes stay under the budget  [§6.3.2, eq. 6]
    return solve_ilp(batches, gpu_budget, min_gpu_neurons)

Predictors. Each Transformer layer has one MLP predictor with an input layer, one hidden layer and an output layer §5.1. The hidden size is set by the layer's sparsity and then adjusted iteratively: the authors increase or shrink it until the perplexity of the model with the predictor on WikiText-2 is within 0.1% of the dense baseline §5.1. Training uses general corpora and no downstream task data §7. The predictor is run on the same input as the FFN block llama.cpp.

# Predict active neurons for one FFN block  [§5.1]
# Input:  x (d,) FFN input; W1 (h, d), W2 (f, h) predictor weights
# Output: scores (f,); a neuron is computed when score >= threshold

def predict_active(x, W1, W2):
    # Hidden layer of the per-layer MLP predictor, ReLU activation  [§5.1]
    hidden = relu(W1 @ x)                              # (h,)
    # One score per FFN neuron  [§5.1] [code: llama.cpp#L4687-L4692]
    scores = W2 @ hidden                               # (f,)
    return scores

Hybrid execution. Before inference, the engine builds a dependency graph of operators and runs CPU and GPU executors that take operators from a shared queue §5.3. For each FFN block, the GPU computes the predicted-active hot neurons and the CPU computes the predicted-active cold neurons §4.2, §5.3. The CPU results are sent to the GPU, which merges them §5.3. Each side also has a neuron table that maps neuron slots back to the original matrix order, reported at about 9 MB for OPT-175B §5.2.

# One sparse FFN block on the hybrid engine  [§4.2, §5.2, §5.3]
# Input:  x (d,); scores (f,) from the predictor; hot_mask (f,) bool
# Output: y (d,)

def hybrid_ffn(x, scores, hot_mask, threshold):
    # Keep only neurons the predictor marks as active  [§5.1, §5.3]
    active = scores >= threshold                       # (f,) bool
    # Split the active set by placement  [§5.2]
    gpu_neurons = active & hot_mask                    # computed on GPU
    cpu_neurons = active & ~hot_mask                   # computed on CPU
    # Each side touches only its active rows and columns  [§5.4]
    partial_gpu = ffn_rows(x, gpu_neurons)             # GPU executor
    partial_cpu = ffn_rows(x, cpu_neurons)             # CPU executor
    # CPU result is copied over and merged on the GPU  [§5.3]
    return partial_gpu + partial_cpu

The neuron-aware operators work on single rows or columns of a weight matrix, first check whether the neuron is active, and then compute only active ones §5.4. On the GPU, each thread block handles a disjoint set of neurons so that no cross-block synchronisation is needed §5.4. On the CPU, neurons are split into batches across cores and computed with vector extensions §5.4.

Implementation. The online engine is 4,200 lines of C++ and CUDA added to llama.cpp, with 10 neuron-aware operators. The offline profiler and solver are about 400 lines of Python added to transformers. The KV cache stays in CPU memory §7. The authors state that when the input is long, the model switches to dense GPU computation in the prefill stage §8.2.

Hyperparameters

Results

Author-reported. Setup: batch size 1 and prompts of 8 to 128 tokens on PC-High (RTX 4090) and PC-Low (RTX 2080 Ti), baselines llama.cpp and SpecInfer §8.1, §8.2. The speedups are relative to llama.cpp unless stated.

Paper vs code

The official code is Tiiny-AI/PowerInfer at code_commit, MIT-licensed. The paper's own text links no code. The repository's README cites this paper with its four authors and links the SJTU-IPADS project board, and the SJTU-IPADS/PowerInfer URL now resolves to this repository, so it is treated as the authors' repository. The current owner is Tiiny-AI, which I could not verify as an author's organisation. The code is a fork of llama.cpp and the README describes later work, so it does not match the paper exactly.

• Batch size in the solver. The paper groups neurons in batches of 64 §6.3.3. The CLI defaults to 256 [code: powerinfer-py/powerinfer/main.py#L17].
• Communication constraint. The paper derives a per-layer minimum from profiled compute and synchronisation times §6.3.1, eq. 4–5. The solver takes one user-supplied constant for all layers, and the CLI default is 0, so the constraint is effectively off by default solver.py [code: powerinfer-py/powerinfer/main.py#L18]. The code never computes compute or sync times.
• Memory constraint. The paper constrains each processing unit separately with per-neuron memory sizes §6.3.2, eq. 6. The solver has one cap on the total number of GPU neurons and treats every neuron as the same size solver.py. The CLI help calls the capacity a VRAM figure, but the solver uses it as a neuron-count budget [code: powerinfer-py/powerinfer/main.py#L14].
• Indicator constant. The paper leaves K as "sufficiently large" §6.3.2. The code uses 1,000,000 solver.py.
• Solver output. The solver returns per-layer counts solver.py. The export then picks the top-k neurons per layer by activation count export_split.py. The paper describes the ILP as assigning individual neurons.
• Solver timeout. The paper reports about 10 s §6.3.3. The code sets a 30 s time limit on GLPK solver.py.
• Predictor training. The paper describes an iterative sizing procedure on WikiText-2 §5.1. The README lists "Release predictor training code" as not done README.md. The repository has no training code. Predictor weights are published separately on Hugging Face README.md.
• Profiler. The paper describes a profiling kernel and a GPU count table §6.1. The repository has no profiler code. It ships pre-profiled activation_*.pt files with the model downloads README.md README.md.
• Predictor threshold. The paper says a neuron is computed when predicted active, without a threshold value §4.2, §5.1. The code skips a neuron when its score is below a threshold, default 0.0, which can also be set from the GGUF file or an environment variable llama.cpp llama.cpp ggml-cuda.cu ggml.c.
• Activation functions. The sparse FFN path passes ReLU as the only activation llama.cpp and asserts for any other activation llama.cpp. The README says only ReLU, ReGLU and squared ReLU are supported README.md. The SiLU-family results in the paper tab. 8 therefore could not be reproduced from this code without changes.
• Prefill switch. The paper says long inputs trigger dense GPU computation in prefill §8.2. In the graph builder, the sparse path is selected per model, not per prompt length llama.cpp llama.cpp. I did not find the prefill switch described in the paper.
• Attention predictors. The paper measures attention-block sparsity §3.2, fig. 6. The code has MLP predictors only llama.cpp.
• Hybrid merge. The paper merges CPU results on the GPU §4.1, §5.3. The CPU path only handles neurons with gpu_idx equal to 0 and an active predictor score ggml.c.

Pitfalls

• Predictor size is stated three ways. The paper says about 6% of parameters §5.1, 7.09% of weights in the overhead section §8.3.4, and 4.88% to 8.68% per model in Table 6 tab. 6. Plan for the Table 6 range.
• Batch size for the solver. Paper: 64 §6.3.3. Code: 256 [code: powerinfer-py/powerinfer/main.py#L17]. Results will depend on which one you use.
• Predictor threshold is not in the paper. The code default of 0.0 is the only value available llama.cpp. If you use a different predictor, its score scale matters.
• Activation sparsity in the 10% vs 20% sense. The Insight 2 footnote says about 10% of MLP neurons are active per iteration §3.2. The evaluation says about 20% of neurons are activated per token for OPT-30B §8.2. The paper does not reconcile these.
• Two ways to report the A100 gap. The abstract says PowerInfer reaches 82% of an A100's token rate for OPT-30B [abs.]. The A100 section reports an 18% slowdown for the same model §8.3.5. These are two framings of one measurement and should not be mixed.
• Accuracy table. For LLaMA(ReGLU)-70B, MMLU and GSM8K are both listed as 62.30% tab. 7. The average (66.99%) is consistent with that, but the duplicate value could be a typo.
• Table 8 needs ReLU-style models. The SiLU results cannot be reproduced with the released sparse path (see Paper vs code).
• Speedup depends on sparsity. The paper says the gains are largest for ReLU-family models and smaller for SiLU-family ones, where a larger share of neurons stays active and CPU compute becomes the bottleneck §4, §8.4.
• Long prompts. For prompts with very short outputs (fewer than 8 tokens), the paper reports gains of 1.07x to 4x, because the prompt phase activates many distinct neurons and reduces sparsity §8.2.
• Predictor placement. The paper says predictors are preferentially placed on the GPU, which it credits for keeping the overhead low §8.3.4.
• Setup cost. The paper says predictor training often takes several hours, is a one-time task, and can be sped up with several GPUs §7. It does not state the cost of profiling or of the ILP beyond the roughly 10 s solve time §6.3.3.
• Results are author-reported on two specific machines, with llama.cpp as the main baseline.

Lineage

• Builds on: llama.cpp (the implementation base, 4,200 added lines) §7; DejaVu, which uses fixed-size predictors and is cited as liu2023deja §5.1.
• Compared against: llama.cpp and SpecInfer (speculative decoding) §8.1; vLLM on an A100 §8.3.5; FlexGen and DejaVu, which the paper dismisses for high latency §8.1; PIT and PyTorch sparse kernels for operators §8.3.3.