Gaussian process

Overview

A Gaussian process (GP) is a prior over functions. Its prediction at a new input comes with a variance. The ten cards use GPs for different tasks, and only some of them use the variance.

The uncertainty use rests on an argument about distance. The SNGP paper says the variance should grow for inputs far from the training data, and calls this distance awareness 2006.10108 §1 2006.10108 §2. It places a GP prior over the training logits with an RBF kernel 2006.10108 §3.1. It also says an exact GP is too costly for large classification problems 2006.10108 §3.1, and that applying a GP to high-dimensional data needs a learned feature extractor, which can destroy the distance property 2006.10108 §1.

The other nine cards use GPs for other purposes:

• Bayesian optimization. EGP-TS uses a weighted mixture of GPs, one per kernel in a fixed dictionary, as its surrogate 2205.14090 §3. HdSafeBO fits one GP to an objective and one to a safety function, and searches a latent trust region 2412.20350 §3. The locomotion paper fits a GP to the cost of detector thresholds and searches it with a mean-and-spread acquisition 2411.12573 §2.5.2.
• Online terrain model. SALON fits a GP from visual features, speed and roughness, and converts its prediction into a risk-aware value 2412.07826 §3.4.
• Trajectory smoothing. GSI fits a GP regression to each coordinate of a tracked object after filling short gaps linearly 2202.13514 §IV-B.
• Human motion. HRI-MPC fits a GP regression to earlier human reaching data. It predicts where a person switches from the fast part of a reach to the careful part, which is where control is handed over 2412.13474 §2.2.
• Policy gradients. BPG treats the gradient of the expected return as a Bayesian integral, with a GP prior on its integrand and Bayesian quadrature for the posterior 2604.27563 §3 2604.27563 §4.
• Dense matching. RoMa keeps a GP as its match encoder, following DKM. The GP is a feature encoder in that paper, not an uncertainty model 2305.15404 §3.1 2305.15404 app. B.
• Interpretable regression. The additive kernel sums products of one-dimensional kernels over all interaction orders, with one variance per order. The fitted variances show which orders carry the signal 1112.4394 §3.2 1112.4394 tab. 1. The full treatment is on the additive kernel page.

Variants

Exact GP on learned features

A GP with an RBF kernel over hidden features, with the kernel matrix computed exactly. The SNGP paper rules this out for large classification problems because the kernel is too costly 2006.10108 §3.1. No card here runs it at scale.

Additive kernel over interaction orders (additive GP)

An exact GP whose kernel is a sum over interaction orders. The order-n term is σ_n² times the sum, over all n-subsets of inputs, of the product of their one-dimensional base kernels 1112.4394 §3.1 1112.4394 eq. 1– 1112.4394 eq. 3. Newton–Girard identities evaluate all orders in O(D²) 1112.4394 §3.3 1112.4394 eq. 6. The Gram matrix costs O(N²DR) for maximum order R, and the authors note that the first few orders usually matter most, so R can be lowered 1112.4394 §3.4. The experiments use R = 10, or D when D < 10, chosen for speed [arxiv:1112.4394 §5.2]. Hyperparameters are fitted by maximising the marginal likelihood 1112.4394 §5.2. Card: Additive Gaussian Processes. The additive kernel page covers the formulas and the Newton–Girard recursion.

Random-feature GP output layer (SNGP)

The output layer becomes a GP. Its kernel is approximated by a fixed random projection followed by a cosine, so only the output weights are learned, and the posterior is an ordinary Bayesian linear model over those features 2006.10108 §3.1. The paper's own posterior uses a Laplace approximation, with a per-class precision accumulated over a final pass of the training data 2006.10108 §3.1 2006.10108 alg. 1. The variant also normalizes the spectral norm of the hidden weight matrices, so that each residual block is close to distance-preserving 2006.10108 §3.2. Card: SNGP.

Mixture of GPs for Bayesian optimization (EGP-TS)

The surrogate is a weighted mixture of GPs, one component per kernel in a dictionary. The prior weight of each component is 1/M in the experiments 2205.14090 §3 2205.14090 §5. After each observation, each component's weight is multiplied by the likelihood of the new value under that component and then renormalized 2205.14090 §3.1 2205.14090 eq. 13.

Each query uses Thompson sampling. The method draws one component in proportion to its weight, draws a function from that component's posterior, and queries the maximiser of that function 2205.14090 §3 [arxiv:2205.14090 eq. 10]. Random Fourier features make the draw cheap, with 50 frequencies per component in the experiments 2205.14090 §5 2205.14090 §8. Card: EGP-TS.

Safe Bayesian optimization with GP surrogates (HdSafeBO)

Two GPs, one for the objective f and one for the safety function g, are fitted with the standard posterior mean and variance 2412.20350 eq. 2. The search runs in a low-dimensional latent space given by PCA or an isometric VAE, with an isometric map back to the input space 2412.20350 §4.3. Inside a trust region around the best safe point, a candidate is kept only if its optimistic safety bound, μ + βσ, passes a threshold 2412.20350 §4.1 2412.20350 alg. 1. The paper chooses β so that Φ(β) ≤ 1 − α, which gives a per-step probability bound on safety 2412.20350 §4.1 prop. 4.1 2412.20350 eq. 3. The next point is the maximiser of one Thompson sample of f over the safe set 2412.20350 alg. 1 2412.20350 §4. The experiments use β = 2 and a Matérn kernel whose ν is not stated 2412.20350 app. B.3. The trust region grows after a run of safe improvements and shrinks after failures 2412.20350 §4.2. Card: HdSafeBO.

GP surrogate for tuning detector thresholds (locomotion BO)

The objective of the threshold search is replaced by a GP surrogate. The GP has an RBF kernel, fitted with GPy, over (threshold pair, cost) samples 2411.12573 §2.5.2. The cost is computed from labelled transitions: it penalises a missed first transition and a wrong Walk output after that transition, and it adds a regularisation term based on each threshold's squared distance to its limit 2411.12573 alg. 2 2411.12573 eq. 10. The acquisition is k·mean(S) ± (1 − k)·std(S), with k = 0.5, and the next threshold minimises it over the search bounds 2411.12573 eq. 8 2411.12573 eq. 9. The search runs for 30 iterations, and the paper compares it with a grid search of 100 evaluations 2411.12573 §4.3. Card: Locomotion mode transitions.

Online terrain model with a risk measure (SALON)

A GP with an RBF kernel takes a cell's visual feature and speed, and predicts the mean and variance of terrain roughness 2412.07826 §3.4 2412.07826 eq. 3. The risk-adjusted roughness adds a multiple of the variance, set by the user's risk tolerance α_R 2412.07826 eq. 4. The speed limit is the same kind of value, computed from a GP conditioned on a roughness cap R_max 2412.07826 eq. 5. The authors prefer a GP over an MLP because it adapts faster and its predictions are stable [arxiv:2412.07826 §3.4]. The training set is a buffer of samples gathered while the robot drives. The buffer is pruned by removing samples from the most common class and speed, which raises the average pairwise distance between buffer points from 3.47 to 4.79 in the first scenario 2412.07826 §3.3 2412.07826 tab. 1. Card: SALON.

GP regression for gap filling (GSI)

Before smoothing, gaps shorter than 20 frames are filled linearly 2202.13514 §V-A. Each track's x, y, w and h are then smoothed by a GP with an RBF kernel. The length scale depends on the track length l as λ = τ · log(τ³ / l) with τ = 10 2202.13514 §IV-B [arxiv:2202.13514 eq. 15]. The smoothed values are the posterior mean at the track's frames 2202.13514 §IV-B 2202.13514 eq. 14. The paper gives no noise value for this regression 2202.13514 §IV-B. Card: StrongSORT.

GP regression for handover timing (HRI-MPC)

A GP regression is fitted to earlier experimental data and predicts the position of the transition between the two sub-movements of a reach. Its inputs are the distance to the goal and the goal size, normalised by arm length [arxiv:2412.13474 §2.2] 2412.13474 fig. 2. In the handover task the robot keeps 400 N/m of translational stiffness until that point, then sets it to 0 [arxiv:2412.13474 §5.2]. The kernel and the training split are not stated. Card: HRI-MPC.

Exact GP posterior over policy gradients (BPG)

The gradient is a GP posterior over the score-function integrand. With the Fisher kernel k(ξ, ξ') = uᵀ G⁻¹ u', the posterior mean is B C y and the covariance is G − B C Bᵀ, where C = (K + Σ)⁻¹ is formed over the sampled paths 2604.27563 §4.2 2604.27563 prop. 4 [arxiv:2604.27563 alg. 1]. A second model uses the quadratic Fisher kernel (1 + uᵀ G⁻¹ u')², which makes the covariance a scalar times the identity, with b₀ = 1 + n for n policy parameters 2604.27563 §4.1 2604.27563 prop. 3. A natural-gradient step multiplies the posterior mean by G⁻¹ [arxiv:2604.27563 alg. 2]. Card: BPG and BAC.

GPTD critic (BAC)

The critic is a GP over the state-action value function, fitted with GPTD. The GPTD model treats each reward as Q(z_i) − γ Q(z_{i+1}) plus noise 2604.27563 §7.1. The kernel adds a state kernel to a Fisher kernel on state-action pairs 2604.27563 §7.2. The gradient posterior then uses the score vectors of the visited pairs and the critic's weights [arxiv:2604.27563 §7.2] 2604.27563 prop. 6. Card: BPG and BAC.

GP match encoder in dense matching (RoMa)

RoMa's match encoder is a GP, carried over from DKM. It works on the frozen coarse features and outputs 512-dimensional embeddings, using an exponential cosine kernel 2305.15404 §3.1 2305.15404 app. B. The appendix describes the GP output as a posterior over coordinates in the second image's embedding 2305.15404 app. B. A Transformer decoder reads the GP output with the DINOv2 features and predicts logits over 64 × 64 anchors plus one matchability logit, instead of regressing the warp directly 2305.15404 §3.3 2305.15404 app. B. The ablation has no row that removes the GP alone, so its own contribution is not measured 2305.15404 tab. 2. The paper gives an inverse temperature of 10 for the kernel; the released code uses a different parameterisation (see the RoMa card and Contradictions). Card: RoMa.

When to use which

The results are author-reported. The SNGP rows come from one paper's tables. The paper does not give the hardware for each latency row 2006.10108 tab. 2 2006.10108 tab. 4. The EGP-TS results are plots, not numbers in the text 2205.14090 §5. The SALON, HRI-MPC and locomotion results come from one robot or one operator each.

Implementation sketch

The common core is the GP posterior mean and variance. The mean is a kernel-weighted combination of the observed values, and the variance shrinks near the data 2202.13514 eq. 14. The variance line is the standard GP predictive form; the cards name the quantities, but none writes this line out.

# GP regression posterior at test inputs  [arxiv:2202.13514 eq. 14]
# Kernel k(x, x'); noise variance sigma2 (not stated in the GSI paper)
def gp_posterior(x_train, y_train, x_test, sigma2):
    # 1. Gram matrices of training and test inputs
    K = kernel_matrix(x_train, x_train)                  # (N, N)
    k_star = kernel_matrix(x_test, x_train)              # (T, N)
    # 2. Invert the noisy Gram matrix once
    inv = inverse(K + sigma2 * identity(N))              # (N, N)
    # 3. Posterior mean  [eq. 14]
    mean = k_star @ inv @ y_train                        # (T,)
    # 4. Posterior variance, prior minus what the data explain
    var = kernel_diag(x_test) - rowsum((k_star @ inv) * k_star)   # (T,)
    return mean, var

The SNGP variant replaces the exact kernel with random features, so the prediction is one forward pass plus a quadratic form. Where the paper's equations disagree, the sketch uses the form from the appendix; see Contradictions.

# Random-feature GP output layer, prediction for one input
# Frozen random projection W_L (D_L, D_h) and bias b_L (D_L,)  [arxiv:2006.10108 §3.1]
# Learned: output weights beta (D_L, K) and per-class covariance  [alg. 1, alg. 2]
def gp_predict(x):
    # 1. Hidden features from the network body  [§3.2]
    h = hidden_network(x)                            # (D_h,)
    # 2. Random Fourier features approximating an RBF kernel  [§3.1]
    phi = sqrt(2 / D_L) * cos(W_L @ h + b_L)         # (D_L,)
    # 3. Posterior mean of each class logit  [alg. 2]
    mean = phi @ beta                                # (K,)
    # 4. Posterior variance of each class logit  [alg. 2]
    var = [phi @ cov[k] @ phi for k in range(K)]     # (K,)
    return mean, var

Contradictions

Across the ten cards, no two papers report conflicting numbers for the same quantity. They measure different things (OOD AUPR and calibration, simple regret, safe objective and violation, tracking HOTA and IDF1, policy learning curves, exoskeleton transition accuracy, navigation interventions, handover timing, dense-matching mAA and AUC), so their numbers should not be compared directly.

Within and around individual papers:

• Additive GP optimizer and initialisation. The paper gives L-BFGS for 500 iterations 1112.4394 §5.2. The card reads the released code as passing a function-evaluation budget instead, and as storing the initial order variance so that it is squared on use. These readings come from the code, not from runs. See the card's Paper vs code section before reusing the paper's settings Additive Gaussian Processes.
• SNGP precision update. §3.1 and appendix C give the minibatch update with the weights on the previous and new precision swapped. With the paper's discount factor of 0.999 the two forms differ strongly 2006.10108 §3.1 2006.10108 app. C.
• SNGP initial precision. Algorithm 1 starts the precision at the identity; appendix C starts it at a scaled identity with a small factor 2006.10108 alg. 1 2006.10108 app. C.
• SNGP sign of the projection. §3.1 writes the projection with a minus sign on the weights, while algorithm 2 does not 2006.10108 §3.1 2006.10108 alg. 2. The weights are frozen, so either sign gives a valid model if it is used consistently in training and prediction; mixing them does not.
• SNGP prediction method. The paper's results average 10 Monte Carlo samples of the logits 2006.10108 §5. The released code's default is a mean-field approximation with a tunable factor that the paper does not state 2006.10108 app. A. The card records the code-side details in full at the card.
• SNGP learning rate for CIFAR-100. Appendix C gives 0.08 [arxiv:2006.10108 app. C]. The released code's default is also 0.08, but its docstring tells CIFAR-100 users to set 0.04 (see the card's Paper vs code section). Check the run configuration before reusing either value.
• SNGP OOD score. The CIFAR tables use the Dempster-Shafer uncertainty of the logits 2006.10108 app. C. The released code marks the Monte Carlo version of that score as degenerate and falls back to a different score by default 2006.10108 app. C. Compare OOD numbers only with the same score.
• EGP-TS refits. The paper refits hyperparameters and weights every 50 iterations 2205.14090 §5. The released notebook turns refits off by default, and its random-feature scaling does not match the RBF length scale (see the card's Paper vs code section). Results from the notebook as released may not reproduce the paper's settings.
• EGP-TS regret theory. The bound is sublinear for the exact-kernel method 2205.14090 §4. Whether it holds for the random-feature sampler used in the experiments is not covered by the paper.
• HdSafeBO safety threshold. Section 3 defines a safe point as g > 0. The candidate filter in algorithm 1 compares μ + βσ with a threshold h, while the update in algorithm 2 uses 0 2412.20350 alg. 1 2412.20350 alg. 2. The experiments use different thresholds in different units [arxiv:2412.20350 app. B.4] 2412.20350 app. B.5. Choose one convention and apply it everywhere.
• HdSafeBO bound scale in code. The paper's bound is μ + βσ [arxiv:2412.20350 §3]. The released code computes the bound through a library call whose scale the card could not check, so the effective multiplier may differ. See the card's Paper vs code section before reusing either form.
• HdSafeBO trust region. The paper does not give the trust-region constants 2412.20350 app. B.3. The released code's restart value and success rule differ from the algorithm (see the card's Paper vs code section), so the trust region may shrink in practice where the paper expects it to grow.
• HdSafeBO guarantee. The per-step probability bound depends on α, which the paper does not state 2412.20350 §4.1. The guarantee also assumes the embedding is accurate; the IRVAE's linear distance correlation is 0.9729, so the isometry does not strictly hold 2412.20350 tab. 4 2412.20350 app. A.3.
• Locomotion acquisition sign. The acquisition is written with ± on the standard deviation, and the paper does not state which sign it uses 2411.12573 eq. 8 2411.12573 §2.5.2.
• Locomotion subject S2. Results say S2's W-S was missed before BO and detected after 2411.12573 §4.2. The discussion says S2's W-S and S-W improved from 80% and 60% to 100% 2411.12573 §5.3. The two statements do not agree.
• Locomotion penalty shape. The text applies the limit penalty when a threshold exceeds 55° or falls below 5°, while the equation penalises the squared distance to the limit for every component 2411.12573 §2.5.2 2411.12573 eq. 10.
• SALON online fitting. The paper describes an RBF GP without online refitting 2412.07826 §3.4. The released code refits the GP every 10 cycles and fine-tunes the speed GP with SGD (see the card's Paper vs code section). The speed GP also uses a subset of the feature columns, while the paper conditions speed on the feature and on R_max 2412.07826 eq. 5.
• GSI length scale. Equation 15 has no bounds 2202.13514 eq. 15. The released code clips the length scale to [0.1, 100] (see the card's Paper vs code section), so the implemented kernel differs from the formula for very long or very short tracks.
• BPG covariance with estimated Fisher. Proposition 3 holds exactly only for the true Fisher matrix 2604.27563 app. A. The paper estimates it in its experiments 2604.27563 §4.3 2604.27563 §7.2, so the reported covariances are approximate.
• RoMa kernel temperature. The paper gives an inverse temperature of 10 for the exponential cosine kernel 2305.15404 app. B. The released code sets T = 0.2 and computes exp((cos − 1)/T), which the card reads as an inverse temperature of 5 (see the RoMa card). Check which convention a reimplementation uses.

Not stated in the BPG paper: the noise matrix Σ for noise-free rewards. For the second model the Gram matrix has rank at most n, so it is singular for M > n unless a nugget is added. This follows from the kernel form, and the paper gives no value 2604.27563 §4.2 2604.27563 §6.2.1.

Papers

• Additive Gaussian Processes (arXiv:1112.4394): a GP whose kernel sums products of one-dimensional kernels over all interaction orders, with one variance per order, compared with linear regression, GP-GAM, HKL and an SE-GP with ARD on regression and classification benchmarks.
• SNGP (arXiv:2006.10108): a deterministic network with spectral-normalized hidden layers and a random-feature GP output layer, compared with deep ensembles on CIFAR and CLINC OOS.
• EGP-TS (arXiv:2205.14090): Bayesian optimization with a weighted mixture of GPs and Thompson sampling via random features, compared with single-GP and bandit baselines on synthetic functions and robot pushing.
• HdSafeBO (arXiv:2412.20350): safe Bayesian optimization of controllers with up to thousands of parameters, using GP surrogates for objective and safety in a latent trust region, compared with LineBO, SCBO, CONFIG, cEI and CMA-ES on synthetic functions.
• Locomotion mode transitions (arXiv:2411.12573): personalizing a threshold-based transition detector on two exoskeletons, with GP-surrogate Bayesian optimization of thresholds against labelled user data, compared with grid search and a statistics-based rescaling.
• SALON (arXiv:2412.07826): online off-road traversability learning with a GP over foundation-model features and bandpower roughness, with CVaR-based cost and speed maps, compared with geometric, GANav, HDIF and Velociraptor costs.
• StrongSORT (arXiv:2202.13514): a tracker whose optional GSI step smooths each track with GP regression to fill gaps, reported with AFLink.
• HRI-MPC (arXiv:2412.13474): planning for human-robot co-manipulation with a human motor-control objective, using a GP regression to predict the handover point, with qualitative results on a Franka arm.
• Bayesian policy gradient and actor-critic (arXiv:2604.27563): GP and Bayesian-quadrature estimates of the policy gradient (BPG), and a GPTD critic (BAC), compared with Monte Carlo policy gradient.
• RoMa (arXiv:2305.15404): dense two-view feature matching with a GP match encoder, a Transformer anchor decoder and a robust refinement loss, compared with DKM, LoFTR, ASpanFormer and others on MegaDepth, ScanNet, WxBS, IMC2022 and InLoc.