Bayesian quadrature

Overview

Bayesian quadrature treats an integral as an unknown quantity to infer. For rho = integral of f(x) g(x) dx, it models f as a Gaussian process (GP), then conditions on noisy evaluations of f at points x_i. The result is a Gaussian posterior over rho, with a mean and a variance 2604.27563 §3. The points may come from any distribution, not only from g. The paper names this as the advantage over Monte-Carlo averaging 2604.27563 §3. The cost is that the prior-mean term and the two integrals b and b_0 must be solvable in closed form for the chosen kernel and factorisation 2604.27563 §3.

Policy gradients are the application in the one card here. The gradient of the expected return is an integral over trajectories, so the paper places the GP on that integrand and uses Fisher kernels, which make the integrals analytic 2604.27563 §4. A Bayesian actor-critic applies the same machinery to a critic that learns from single transitions 2604.27563 §7.

Variants

Plain BQ for one integral

The general form in 2604.27563 §3: a GP prior on f, a Gram matrix over the samples, and the posterior formulas given in the implementation sketch below. The paper extends the formulas to vector-valued integrals. It assumes each component is independent and shares one kernel 2604.27563 §3.1 [prop. 1] [prop. 2].

Model 1: vector GP with a quadratic Fisher kernel

The deterministic factor is the trajectory probability, and the GP models the return times the score. The kernel is the square of one plus a Fisher inner product 2604.27563 §4.1 [prop. 3]. The posterior mean of the gradient is a weighted sum of the measurements, and the posterior covariance is a scalar times the identity, so correlations between gradient components are ignored 2604.27563 §4.1 tab. 1. Card: Bayesian policy gradient and actor-critic algorithms.

Model 2: scalar GP with a Fisher kernel

The GP models the return, and the deterministic factor is the gradient of the trajectory probability. The kernel is the Fisher inner product of the score vectors 2604.27563 §4.2. The posterior mean and the full posterior covariance come out as matrix products of the score matrix, which the paper says is simpler to implement with one prior mean, one kernel and one noise matrix 2604.27563 §4.2 [prop. 4] tab. 1. Card: Bayesian policy gradient and actor-critic algorithms.

Sparse dictionary variant

A new trajectory joins the dictionary only if its residual kernel distance exceeds a threshold tau 2604.27563 §4.4. The sparse posterior in [prop. 5] needs the noise matrix to be invertible. The paper says sparsification makes the method faster and less memory-hungry, may reduce accuracy, and helps when the kernel matrix is near-singular 2604.27563 §4.4. The text does not state a value for tau 2604.27563 §4.4 §7.3.

Natural and variance-scaled steps

The update in alg. 2 uses the posterior mean as the step. A natural-gradient variant first multiplies it by the inverse Fisher matrix [arxiv:2604.27563 alg. 2]. The paper also reports a variant that scales the step by the posterior covariance, which it calls BPG-var, and finds it beats BPG and Monte-Carlo policy gradient at all four sample sizes tested on the LQR [arxiv:2604.27563 §6.2.2] fig. 5. Both are author-reported on simulated tasks.

Bayesian actor-critic (BAC)

The observation unit is a single state-action transition, so the method can use the Markov property that whole-trajectory BQ cannot 2604.27563 §1 §7. The action-value function gets a GP prior whose kernel adds a state kernel to the Fisher kernel on state-action pairs. A GPTD model supplies the posterior mean and covariance of the critic 2604.27563 §7.1 §7.2. The gradient posterior is then the score matrix times the critic weights, with covariance G minus a correction term 2604.27563 §7.2 [prop. 6]. The Fisher matrix is estimated from the data, and the paper gives a stationary estimate and a discounted-state estimator, G-EST 2604.27563 §7.2 alg. 3. Card: Bayesian policy gradient and actor-critic algorithms.

When to use which

The one card supports only the comparisons below. All results are author-reported on simulated tasks, and most of them appear only in figures the card does not transcribe 2604.27563 §6 §8.

Whichever variant you pick, the closed forms are exact only if the Fisher matrix is the true expectation. With an estimated Fisher matrix, the identities used for b and b_0 become approximations 2604.27563 app. A §4.3. The authors also note that the kernel restriction can add approximation error that is hard to quantify, and that they did not see a significant error in their experiments 2604.27563 §4 §7.2.

Implementation sketch

The posterior of the generic integral, in the form given in [arxiv:2604.27563 §3]. Variant-specific steps (the score and Fisher kernel, the sparse dictionary, the GPTD critic) are left out.

# Bayesian quadrature for rho = integral of f(x) g(x) dx  [§3]
# Inputs: points x_1..x_M, noisy values y_i of f(x_i), noise matrix Sigma
def bq_posterior(xs, ys, kernel, g, f_bar, noise):
    # Gram matrix of the GP prior on f  [§3]
    K = kernel_matrix(xs, xs)                       # (M, M)
    # Weights C = (K + Sigma)^-1  [§3]
    C = inverse(K + noise)                          # (M, M)
    # Prior mean of rho: rho_0 = integral of f_bar g  [§3]
    rho_0 = integrate(lambda x: f_bar(x) * g(x))    # scalar
    # b_i = integral of k(x, x_i) g(x) dx  [§3]
    b = [integrate(lambda x: kernel(x, xi) * g(x)) for xi in xs]
    # b_0 = double integral of k(x, x') g(x) g(x')  [§3]
    b_0 = integrate2(lambda x, y: kernel(x, y) * g(x) * g(y))
    # Posterior mean and variance of rho  [§3]
    mean = rho_0 + b @ C @ (ys - f_bar(xs))         # scalar
    var = b_0 - b @ C @ b                           # scalar
    return mean, var

Notes for implementers:

• The Fisher matrix is assumed known in the algorithm. In practice it is estimated, either from the sampled trajectories or from a fitted transition model 2604.27563 §4.3.
• The text does not give the noise matrix for noise-free rewards. For Model 2 the Gram matrix has rank at most the number of policy parameters, so it is singular once M exceeds that number unless a noise term or nugget is added. This is an inference from the kernel form 2604.27563 §4.2.
• The LQR is written as a cost to minimise, while the update rules are written as ascent. Pick one sign convention and apply it throughout 2604.27563 §6.2 alg. 2.

Contradictions

None across papers: this is the only card that lists the slug. Inside the one paper, two points need care and are not resolved here.

• The text says BPG is similar to BAC at M = 100 2604.27563 §8.1. The random-walk policy runs use only M = 1, 25, 50 and 75 2604.27563 §8.1 tab. 4, so that claim is not tested in the reported experiments.

Papers

• Bayesian policy gradient and actor-critic algorithms: Bayesian quadrature over policy gradients (BPG, Models 1 and 2, sparse variant) and a Bayesian actor-critic (BAC) with a GPTD critic.