--- kind: concept slug: bayesian-quadrature title: Bayesian quadrature summary: >- Bayesian quadrature (BQ) estimates an integral by putting a Gaussian-process prior on the integrand and returning a posterior mean and variance, so the samples can come from any distribution. Only one card so far: a Bayesian policy-gradient paper applies it to policy gradients and reports, author-reported, fewer samples than Monte-Carlo on small test tasks. tags: [integration, gaussian-process, policy-gradient, sample-efficiency, fisher-kernel] papers: ["2604.27563"] related: [gaussian-process, bayesian-optimization, policy-gradient, actor-critic, importance-weighting] status: draft updated_at: 2026-10-24 --- ## Overview Bayesian quadrature treats an integral as an unknown quantity to infer. For rho = integral of f(x) g(x) dx, it models f as a Gaussian process (GP), then conditions on noisy evaluations of f at points x_i. The result is a Gaussian posterior over rho, with a mean and a variance [arxiv:2604.27563 §3]. The points may come from any distribution, not only from g. The paper names this as the advantage over Monte-Carlo averaging [arxiv:2604.27563 §3]. The cost is that the prior-mean term and the two integrals b and b_0 must be solvable in closed form for the chosen kernel and factorisation [arxiv:2604.27563 §3]. Policy gradients are the application in the one card here. The gradient of the expected return is an integral over trajectories, so the paper places the GP on that integrand and uses Fisher kernels, which make the integrals analytic [arxiv:2604.27563 §4]. A Bayesian actor-critic applies the same machinery to a critic that learns from single transitions [arxiv:2604.27563 §7]. ## Variants ### Plain BQ for one integral The general form in [arxiv:2604.27563 §3]: a GP prior on f, a Gram matrix over the samples, and the posterior formulas given in the implementation sketch below. The paper extends the formulas to vector-valued integrals. It assumes each component is independent and shares one kernel [arxiv:2604.27563 §3.1] [prop. 1] [prop. 2]. ### Model 1: vector GP with a quadratic Fisher kernel The deterministic factor is the trajectory probability, and the GP models the return times the score. The kernel is the square of one plus a Fisher inner product [arxiv:2604.27563 §4.1] [prop. 3]. The posterior mean of the gradient is a weighted sum of the measurements, and the posterior covariance is a scalar times the identity, so correlations between gradient components are ignored [arxiv:2604.27563 §4.1] [tab. 1]. Card: [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563). ### Model 2: scalar GP with a Fisher kernel The GP models the return, and the deterministic factor is the gradient of the trajectory probability. The kernel is the Fisher inner product of the score vectors [arxiv:2604.27563 §4.2]. The posterior mean and the full posterior covariance come out as matrix products of the score matrix, which the paper says is simpler to implement with one prior mean, one kernel and one noise matrix [arxiv:2604.27563 §4.2] [prop. 4] [tab. 1]. Card: [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563). ### Sparse dictionary variant A new trajectory joins the dictionary only if its residual kernel distance exceeds a threshold tau [arxiv:2604.27563 §4.4]. The sparse posterior in [prop. 5] needs the noise matrix to be invertible. The paper says sparsification makes the method faster and less memory-hungry, may reduce accuracy, and helps when the kernel matrix is near-singular [arxiv:2604.27563 §4.4]. The text does not state a value for tau [arxiv:2604.27563 §4.4] [§7.3]. ### Natural and variance-scaled steps The update in [alg. 2] uses the posterior mean as the step. A natural-gradient variant first multiplies it by the inverse Fisher matrix [arxiv:2604.27563 alg. 2]. The paper also reports a variant that scales the step by the posterior covariance, which it calls BPG-var, and finds it beats BPG and Monte-Carlo policy gradient at all four sample sizes tested on the LQR [arxiv:2604.27563 §6.2.2] [fig. 5]. Both are author-reported on simulated tasks. ### Bayesian actor-critic (BAC) The observation unit is a single state-action transition, so the method can use the Markov property that whole-trajectory BQ cannot [arxiv:2604.27563 §1] [§7]. The action-value function gets a GP prior whose kernel adds a state kernel to the Fisher kernel on state-action pairs. A GPTD model supplies the posterior mean and covariance of the critic [arxiv:2604.27563 §7.1] [§7.2]. The gradient posterior is then the score matrix times the critic weights, with covariance G minus a correction term [arxiv:2604.27563 §7.2] [prop. 6]. The Fisher matrix is estimated from the data, and the paper gives a stationary estimate and a discounted-state estimator, G-EST [arxiv:2604.27563 §7.2] [alg. 3]. Card: [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563). ## When to use which The one card supports only the comparisons below. All results are author-reported on simulated tasks, and most of them appear only in figures the card does not transcribe [arxiv:2604.27563 §6] [§8]. | Situation | Use | Why | |---|---|---| | Gradient estimates from few trajectories, fixed episode length, and you can compute the Fisher matrix and the kernel integrals | BPG, Model 1 or 2 | On the bandit with M = 10, the standard deviation of the first gradient component is 0.050 for BQ and 0.438 for Monte-Carlo [arxiv:2604.27563 tab. 2]. The text says BQ's variance is about one order of magnitude lower at M = 10 [arxiv:2604.27563 §6.1] | | Policy optimisation on the LQR with M = 5 | Monte-Carlo policy gradient | At M = 5, only the Monte-Carlo method is better; BPG wins at M = 10, 20 and 40 [arxiv:2604.27563 §6.2.2] [fig. 3] | | You need correlations between gradient components | Model 2 | Model 1's covariance is a scalar times the identity [arxiv:2604.27563 §4.1] [§4.2] | | Trajectory lengths vary between episodes | BAC | The authors say BQ's performance degrades in that case, and they used BAC for mountain car and ship steering [arxiv:2604.27563 §8] | | Need a critic and per-transition updates | BAC | It uses single transitions and a GPTD critic [arxiv:2604.27563 §7] | | Large number of samples, limited memory | Sparse variant | Faster and smaller, with a possible loss of accuracy [arxiv:2604.27563 §4.4] | | Mountain car, few episodes per update | BAC | Reaches a good policy with M = 5, where Monte-Carlo needs M = 40 to catch up [arxiv:2604.27563 §8.2] | | Ship steering | BAC | Higher success ratio than Monte-Carlo for all M tested, though Monte-Carlo is slightly better early at M = 5 [arxiv:2604.27563 §8.3] | Whichever variant you pick, the closed forms are exact only if the Fisher matrix is the true expectation. With an estimated Fisher matrix, the identities used for b and b_0 become approximations [arxiv:2604.27563 app. A] [§4.3]. The authors also note that the kernel restriction can add approximation error that is hard to quantify, and that they did not see a significant error in their experiments [arxiv:2604.27563 §4] [§7.2]. ## Implementation sketch The posterior of the generic integral, in the form given in [arxiv:2604.27563 §3]. Variant-specific steps (the score and Fisher kernel, the sparse dictionary, the GPTD critic) are left out. ```pseudocode # Bayesian quadrature for rho = integral of f(x) g(x) dx [§3] # Inputs: points x_1..x_M, noisy values y_i of f(x_i), noise matrix Sigma def bq_posterior(xs, ys, kernel, g, f_bar, noise): # Gram matrix of the GP prior on f [§3] K = kernel_matrix(xs, xs) # (M, M) # Weights C = (K + Sigma)^-1 [§3] C = inverse(K + noise) # (M, M) # Prior mean of rho: rho_0 = integral of f_bar g [§3] rho_0 = integrate(lambda x: f_bar(x) * g(x)) # scalar # b_i = integral of k(x, x_i) g(x) dx [§3] b = [integrate(lambda x: kernel(x, xi) * g(x)) for xi in xs] # b_0 = double integral of k(x, x') g(x) g(x') [§3] b_0 = integrate2(lambda x, y: kernel(x, y) * g(x) * g(y)) # Posterior mean and variance of rho [§3] mean = rho_0 + b @ C @ (ys - f_bar(xs)) # scalar var = b_0 - b @ C @ b # scalar return mean, var ``` Notes for implementers: - The Fisher matrix is assumed known in the algorithm. In practice it is estimated, either from the sampled trajectories or from a fitted transition model [arxiv:2604.27563 §4.3]. - The text does not give the noise matrix for noise-free rewards. For Model 2 the Gram matrix has rank at most the number of policy parameters, so it is singular once M exceeds that number unless a noise term or nugget is added. This is an inference from the kernel form [arxiv:2604.27563 §4.2]. - The LQR is written as a cost to minimise, while the update rules are written as ascent. Pick one sign convention and apply it throughout [arxiv:2604.27563 §6.2] [alg. 2]. ## Contradictions None across papers: this is the only card that lists the slug. Inside the one paper, two points need care and are not resolved here. - The text says BPG is similar to BAC at M = 100 [arxiv:2604.27563 §8.1]. The random-walk policy runs use only M = 1, 25, 50 and 75 [arxiv:2604.27563 §8.1] [tab. 4], so that claim is not tested in the reported experiments. ## Papers - [Bayesian policy gradient and actor-critic algorithms](/papers/2604.27563): Bayesian quadrature over policy gradients (BPG, Models 1 and 2, sparse variant) and a Bayesian actor-critic (BAC) with a GPTD critic.