---
kind: concept
slug: sparse-bayesian-learning
title: Sparse Bayesian learning
summary: >-
  Sparsity from a Bayesian prior: each weight gets a zero-mean Gaussian whose variance is learned, and weights whose variance collapses are pruned. Used for neural architecture search and structural pruning. Trade-offs: the search caches feature maps in memory, and the only known source is one author-reported paper with unresolved inconsistencies.
tags: [bayesian-learning, sparsity, pruning, nas]
papers: ["1905.04919"]
related: [differentiable-architecture-search, network-pruning, neural-architecture-search, uncertainty-estimation]
status: draft
updated_at: 2026-10-19
---

## Overview

Sparse Bayesian learning gives each weight its own prior variance and lets
the data decide how large that variance should be. A weight whose variance
shrinks towards zero is effectively removed from the model. The paper frames
architecture search as this kind of sparse learning over an over-parameterised
network, where each candidate operation on an edge is one weight [arxiv:1905.04919
§3] [arxiv:1905.04919 §5.1].

The prior is hierarchical. Each variance has a Gamma hyper-prior, which yields
a Student-t marginal and favours sparsity. The paper sets the hyper-prior
parameters to zero, which makes them flat [arxiv:1905.04919 §5.1]. The
variance of an edge is also tied to the switches of the edges that feed into
the same node, so a node whose inputs are switched off also weakens the
variance of the edges it would feed [arxiv:1905.04919 §4.1]
[arxiv:1905.04919 eq. 6].

Pruning uses the posterior variance, not the weight magnitude. The paper
states that magnitude depends on weight scale and on the architecture choice,
which is why it prefers the variance-based rule [arxiv:1905.04919 §1]
[arxiv:1905.04919 §2]. A Laplace approximation with a Hessian term gives the
posterior update [arxiv:1905.04919 §5.2].

## Variants

### Cell-based architecture search (BayesNAS)

Sparse Bayesian learning over the operations of a cell. The search trains an
over-parameterised network for a short time, updates the variances once per
iteration, and prunes edges whose variance is at most $1/(2\pi e)$
[arxiv:1905.04919 §5.3] [arxiv:1905.04919 alg. 1]. The reported CIFAR-10 test
error is 2.81% for the variant with $\lambda_w = 0.005$, with cutout and an
author-reported search cost of 0.2 GPU days [arxiv:1905.04919 tab. 1].
Card: [BayesNAS: A Bayesian Approach for Neural Architecture Search](/papers/1905.04919).

### Proxy and proxyless search

Proxy search ties switches and re-weighting coefficients within a group of
identical cells, so the search is on a smaller network that is then
transferred [arxiv:1905.04919 app. S1.3]. Proxyless search runs the same
procedure on each cell separately, using a PyramidNet backbone with a tree
cell of nine paths [arxiv:1905.04919 §6.2]. The paper reports 2.41% CIFAR-10
test error for the proxyless variant with a reported search cost of 0.1 GPU
days [arxiv:1905.04919 tab. 1]. These are author-reported numbers.

### Structural compression of trained networks

The same machinery prunes groups of weights in a trained network: shape-wise,
row-wise, column-wise or filter-wise groups, using a re-weighted group Lasso
[arxiv:1905.04919 app. S2] [arxiv:1905.04919 app. S2.1]. The paper reports
results for LeNet-300-100, LeNet-5 and ResNet-18 that trade a small error
increase for fewer FLOPs in the MNIST cases [arxiv:1905.04919 app. S2.2]. The
MNIST and CIFAR-10 results report no spread across runs.

## When to use which

| Situation | Use | Why |
|---|---|---|
| Need a gradient-based architecture search that keeps a magnitude-free prune rule | Cell-based search | The prune rule uses the posterior variance rather than weight size [arxiv:1905.04919 §1] [arxiv:1905.04919 §5.3]. |
| Single GPU with limited memory | Check first | Feature maps are cached to compute the Hessian, which the authors call inefficient, and that memory sets the search batch size [arxiv:1905.04919 §8] [arxiv:1905.04919 §6.1]. |
| Search time is the main constraint | Cell-based or proxyless search | The paper reports 0.2 GPU days (cell-based) and 0.1 GPU days (proxyless), but also about 3 hours on one GPU, which conflicts with these figures (see below) [arxiv:1905.04919 tab. 1] [arxiv:1905.04919 §6.1]. |
| Need to shrink an already trained network | Structural compression | This is the setting the group-Lasso variant targets, and it does not need a search [arxiv:1905.04919 app. S2]. |
| Need exact, reproducible numbers | Not established | No official code was found, and the paper's own hyperparameters conflict in places (see below) [arxiv:1905.04919 §6.1] [arxiv:1905.04919 app. S4.1]. |

## Implementation sketch

Adapted from the search loop in the paper. Variants differ in how switches
are tied (cell-based vs proxy), in the Hessian approximation, and in whether
the loop is over architecture weights or over the groups of a trained network.

```pseudocode
# Sparse Bayesian search over one over-parameterised network  [alg. 1]
# Initialise architecture weights, variances and l1 weights to 1  [alg. 1]
arch, gamma, omega = ones(n), ones(n), ones(n)
for t in range(1, T + 1):
    # Train weights on the re-weighted loss; schedule not stated  [§5.3]
    net, arch = sgd_step(data, net, arch, lambda_w * omega)
    # Hessian of each architecture weight, from cached feature maps  [app. S3]
    hess = architecture_hessian(net, arch, data)
    # Posterior variance, using the previous gamma  [alg. 1]
    post = 1 / (1 / gamma + hess)
    # Re-weighting coefficient for the next l1 step  [alg. 1]
    omega = sqrt(gamma - post) / gamma
    # Switch of each architecture weight  [alg. 1]
    switch = abs(arch / omega)
    # Variance from the predecessor switches and the own switch  [eq. 6] [§4.1]
    gamma = 1 / (1 / pred_sum(switch) + 1 / switch)
# Keep edges whose entropy is positive  [§5.3]
keep = gamma > 1 / (2 * pi * e)
```

Notes: the number of iterations $T$ is not stated for the CIFAR-10 search
[arxiv:1905.04919 §5.3]. The paper does not say which Hessian approximation
the experiments use [arxiv:1905.04919 app. S3.1] [arxiv:1905.04919 app. S3.2].

## Contradictions

Only one card covers this slug, so there are no disagreements between papers.
Within BayesNAS, the reported numbers are inconsistent in three places, and
each side is cited:

- Search cost: 0.2 GPU days in the abstract and [arxiv:1905.04919 tab. 1],
  against "about 3 hours on a single GPU" in [arxiv:1905.04919 §6.1].
- Search batch size: 18 in the main text [arxiv:1905.04919 §6.1], 16 in the
  appendix [arxiv:1905.04919 app. S4.1].
- Exact FC-layer Hessian cost: the formula gives about 8.03 M operations
  for $n = m = 100$, while the text states 107.97 M [arxiv:1905.04919 app. S3.1].

## Papers

- [BayesNAS: A Bayesian Approach for Neural Architecture Search](/papers/1905.04919):
  applies sparse Bayesian learning with a hierarchical ARD-style prior to
  one-shot architecture search and structural compression.