---
kind: concept
slug: neural-network-gaussian-process
title: Neural network Gaussian process
summary: >-
  A wide network with random Gaussian weights behaves like a Gaussian process, so its predictions come with a variance and need no gradient training. A recursion over layers computes the kernel. Variants differ by layer type (plain or residual convolution) and depth. Trade-off: exact Bayesian predictions with few settings, but an N×N kernel matrix and lower accuracy than trained residual networks.
tags: [gaussian-processes, infinite-width, kernels, uncertainty]
papers: ["1808.05587"]
related: [gaussian-process, neural-tangent-kernel, convolutional-neural-network, residual-learning, bayesian-neural-network]
status: draft
updated_at: 2026-11-05
---

## Overview

A neural network with randomly drawn weights and biases, in the limit of very
many channels per layer, defines a Gaussian process over its inputs. The
network's kernel, the covariance between the outputs for two inputs, can then
be computed without training any weights [arxiv:1808.05587 §2] [arxiv:1808.05587 §3].
The shared invariant is that, under the prior, only the variances at
corresponding locations and the covariance between two inputs at each layer
are needed. The kernel is a recursion over layers, and the Gaussian process
posterior then gives an exact prediction [arxiv:1808.05587 §3.2] [arxiv:1808.05587 §4].

The original argument covers fully connected networks [arxiv:1808.05587 §1].
The paper under this slug extends it to convolutional and residual networks,
which use translation-invariant filters and shortcut connections. The kernel
for these needs only the three kernel entries at each location, not the
covariance between all pairs of locations [arxiv:1808.05587 §3.2].

## Variants

All three variants come from one paper and share the same recursion. They
differ in layer type, depth and skip pattern, and the paper's hyperparameter
table sets each one [arxiv:1808.05587 tab. 2].

### Convolutional kernel (ConvNet GP)

A stack of convolutional layers with a ReLU nonlinearity and no skip
connections. The paper's setting uses 7 layers with 7×7 filters [arxiv:1808.05587 tab. 2].
Its reported MNIST test error is 1.03% with 27 kernels in the random search [arxiv:1808.05587 tab. 1].
Card: [Deep Convolutional Networks as shallow Gaussian Processes](/papers/1808.05587).

### Residual convolutional kernel (Residual CNN GP)

The same convolutional recursion with a skip connection at every layer, with
4×4 filters and 9 layers [arxiv:1808.05587 tab. 2]. The paper reports 0.93%
test error on MNIST [arxiv:1808.05587 tab. 1]. Card: [Deep Convolutional Networks as shallow Gaussian Processes](/papers/1808.05587).

### Deep residual kernel (ResNet GP, 32 layers)

A 32-layer residual kernel with 3×3 filters, a skip every two layers, and
mostly stride-1 layers [arxiv:1808.05587 tab. 2]. Its bias variance is zero and
its weight variance is fixed rather than searched [arxiv:1808.05587 tab. 2]
[arxiv:1808.05587 §4]. This is the most accurate kernel in the paper: 0.84% test
error and 0.68% validation error on MNIST [arxiv:1808.05587 tab. 1]. The paper
reports the cost of computing its kernel matrix and the validation and test
kernels as 3 h 40 min on two Tesla P100 GPUs [arxiv:1808.05587 §4]. Card: [Deep Convolutional Networks as shallow Gaussian Processes](/papers/1808.05587).

## Efficiency

The cost of computing the kernel for one pair of inputs is about one forward
pass of a network with one filter per layer [arxiv:1808.05587 §3.3]. For N
training examples, L layers and input dimension D, the paper gives O(N²·L·D)
for the kernel matrix and O(N³) for inverting it [arxiv:1808.05587 §4]. For
the MNIST runs, the kernel matrix dominates the cost: inversion and error
evaluation took about 43 s on one GPU against the 3 h 40 min for kernel
computation [arxiv:1808.05587 §4].

## When to use which

| Situation | Use | Why |
|---|---|---|
| You need predictions with uncertainty and no gradient training | Any variant | The kernel is computed from random-weight statistics, and the posterior is exact GP regression [arxiv:1808.05587 §4] |
| You want the best accuracy among these kernels on images | ResNet GP, 32 layers | Lowest reported test error, 0.84% against 1.03% and 0.93% for the shallower kernels [arxiv:1808.05587 tab. 1] |
| You want a shallower image kernel with less compute | ConvNet GP or Residual CNN GP | Fewer layers. Both beat the NNGP baseline in the paper's table, 1.03% and 0.93% against 1.21% [arxiv:1808.05587 tab. 1]. The paper does not report their compute cost separately |
| The N×N kernel matrix does not fit in memory or time | Do not use this method as is | Kernel and inverse costs grow with N, N² and N³ [arxiv:1808.05587 §4]. Use an approximation or a trained network |
| You need the best accuracy on images, period | A trained residual network | The standard ResNet row in the paper's table, a result the paper cites from another work, reports 0.41% test error, against 0.84% for the kernel [arxiv:1808.05587 tab. 1] |
| You need a calibrated probabilistic classifier | Use a proper classification likelihood | The main experiments use regression on ±1 targets. Calibration is checked separately with a classification likelihood and inducing points [arxiv:1808.05587 app. §7.3] |

## Implementation sketch

Pseudocode of the common core for the convolutional recursion. The three
kernel entries per location are propagated through the layers. The ReLU
expectation is written as a helper so that the constant can be chosen
explicitly (see Contradictions).

```pseudocode
# Convolutional NNGP kernel for one pair of inputs x, x2
# Input: images (C0, H0*W0), number of layers L, sigma_w2, sigma_b2
# Output: scalar kernel value k(x, x2)

def nngp_conv_kernel(x, x2, L, sigma_w2, sigma_b2):
    # First layer: average over input channels, then sum over each filter
    # patch; this gives the three kernel entries at each location [arxiv:1808.05587 eq. 10]
    k_xx = sigma_b2 + sigma_w2 * patch_sum(mean_channels(x * x))
    k_yy = sigma_b2 + sigma_w2 * patch_sum(mean_channels(x2 * x2))
    k_xy = sigma_b2 + sigma_w2 * patch_sum(mean_channels(x * x2))
    for _ in range(L):
        # Expected product of post-nonlinearity activations for the pair  [arxiv:1808.05587 eq. 13]
        v_xy = relu_expectation(k_xy, k_xx, k_yy, scale=scale_choice)
        # Propagate each entry through the next filter  [arxiv:1808.05587 eq. 11]
        k_xx = sigma_b2 + sigma_w2 * patch_sum(k_xx)
        k_yy = sigma_b2 + sigma_w2 * patch_sum(k_yy)
        k_xy = sigma_b2 + sigma_w2 * patch_sum(v_xy)
    # Read-out: one number per pair; normalisation not stated [arxiv:1808.05587 alg. 1]
    return mean(k_xy)
```

For residual kernels, a skip connection adds the kernel stored from the
layer it spans back, in place of the usual plain update
[arxiv:1808.05587 eq. 14] [arxiv:1808.05587 eq. 15].
Pooling is not part of the method. Average pooling would need covariances
between locations within a filter, which the efficient form avoids, so the
paper ends with a dense layer instead [arxiv:1808.05587 §4].

Prediction is exact Gaussian process regression on the training kernel
matrix, with one-hot targets in {−1, +1} and the largest output as the class
[arxiv:1808.05587 §4]. The kernel itself has no gradient-trained parameters,
so no optimiser or learning rate applies [arxiv:1808.05587 §4]. Hyperparameters
were chosen on a validation split, with no retraining afterwards [arxiv:1808.05587 §4].

## Contradictions

- **Nonlinearity constant.** The text defines the nonlinearity as max(0, x)
  [arxiv:1808.05587 §3.2]. The closed form in [arxiv:1808.05587 eq. 13] matches
  √2·max(0, x) instead, and a card check finds the paper does not say which the
  ResNet kernel uses. Plugging plain ReLU into [arxiv:1808.05587 eq. 13] halves
  the expected products. See [Deep Convolutional Networks as shallow Gaussian Processes](/papers/1808.05587)
  for the code's choice. Unresolved.
- **Weight-scale note for the ResNet.** Table 2 says the ResNet weight
  variance matters only for numerical stability, because a plain ReLU stack
  can absorb it [arxiv:1808.05587 tab. 2]. With residual sums in
  [arxiv:1808.05587 eq. 15], the branch scale changes the balance against the
  shortcut, so this claim does not hold for the residual recursion. The card
  finds the same. Unresolved.
- **Output normalisation.** The paper's read-out is one location with no stated
  normalisation [arxiv:1808.05587 alg. 1]. A constant factor on the whole
  kernel leaves a noiseless GP argmax unchanged, but a per-layer factor does not.
  The paper does not say which is used.
- **Search size.** The paper reports 27 kernels per row [arxiv:1808.05587 tab. 1].
  The card notes the run configuration in the public code does not obviously
  match this count; see [Deep Convolutional Networks as shallow Gaussian Processes](/papers/1808.05587).

## Papers

- [Deep Convolutional Networks as shallow Gaussian Processes](/papers/1808.05587): gives the convolutional and residual kernels, their recursion, and MNIST results with the ResNet kernel at 0.84% test error.