Neural operator

Overview

A neural operator learns a map between functions rather than between fixed vectors. For a partial differential equation, the input is a function such as an initial state or a material coefficient, and the output is the solution function 2109.13459 §2.1. Data are point values of both on a grid 2109.13459 §3. In the compressor card the input is a blade shape and the output is a set of flow fields 2609.16066 §3.3.

Earlier networks for this task were tied to one grid resolution or one PDE instance, and had to be retrained when either changed 2109.13459 §1. Neural operators avoid this by learning the map itself, without fixing the grid. The shared invariant is that the learned weights describe the map, not the values at particular grid points, so the same model can be queried at a different resolution 2109.13459 §1 2109.13459 §2.3. The authors argue that existing neural operators still do not exploit the structure of the operator's kernel 2109.13459 §2.4.

Three cards in this wiki carry this slug: the multiwavelet card, the transformer-based compressor card and the uncertainty card, which wraps a DeepONet rather than defining a new operator. Other neural operators appear only as baselines, with the numbers the cards report.

Variants

Multiwavelet operator (MWT)

The input is split into scales with a fixed multiwavelet decomposition built from orthogonal polynomials. The filters are computed, not learned 2109.13459 §2.2 2109.13459 app. B. A small learned kernel acts on the detail and coarse terms at each scale, and the same kernels are reused at every scale 2109.13459 §2.3. There are two polynomial choices. The Legendre basis is the headline variant. The Chebyshev basis needs an extra correction matrix in reconstruction 2109.13459 app. C.2 2109.13459 app. C.3.

The paper's premise is that the operator kernel is sparse in this basis when it is smooth away from its singularities. The argument covers smooth kernels only 2109.13459 §2.4. Card: Multiwavelet-based Operator Learning for Differential Equations.

Branch-trunk operator with Galerkin attention (TNO)

This variant follows the branch-trunk framework of deepONet, which the card credits to Lu et al. 2609.16066 §1 2609.16066 §3.1. A branch network encodes the design variables, here 28 blade-geometry values. A trunk network encodes each query point (r, z) on a meridional plane. The two feature vectors are combined and passed to a shared output network that returns five flow fields at that point 2609.16066 eq. 1 2609.16066 §3.1. The trunk is a stack of attention blocks in which the attention is Galerkin-type rather than softmax. Keys and values are layer-normalised, and the keys and values are summarised over all points before the queries are applied 2609.16066 §3.1 2609.16066 eq. 3. The card gives the recurrence of these blocks 2609.16066 eq. 2.

Because the trunk takes query coordinates as input, the output is evaluated point by point. The card trains and tests on 4096 points per field and does not test a different point count 2609.16066 §3.3. Card: Compressor-cascade performance from a transformer-based operator.

Performance numbers are computed from the predicted fields rather than learned separately. The card averages the predicted field over the span at two axial stations and applies the metric formulas to the averages 2609.16066 §3.2 2609.16066 tab. 6.

Uncertainty for DeepONets (PA-BNN-FP, PA-GAN-FP)

This is not a new operator layer. It wraps a DeepONet, which has a branch network on the sampled input function and a trunk network on the query point, and whose output is their inner product 2201.07766 §2.2. The predictive variance adds a noise term to the spread over posterior samples 2201.07766 §3.1.

For operator learning the DeepONet is pre-trained on clean data. At inference it is given a few noisy points, and a prior is placed over it. The physics-agnostic variants use a Bayesian neural network prior (PA-BNN-FP) or a GAN generator prior (PA-GAN-FP) 2201.07766 §4.3 2201.07766 §4.5.2. With a GAN prior, inference runs on a low-dimensional latent input 2201.07766 §4.3.

The Darcy test uses 20 noisy λ points and 10 noisy u points at inference 2201.07766 §6.5. In the paper's ×10² units, the GAN prior is more accurate and the BNN prior is better calibrated. For u in distribution the relative L2 error is 7.7 for the GAN prior and 14.6 for the BNN prior, and the RMSCE is 2 for the BNN and 8.6 for the GAN. Out of distribution the errors are 35.5 and 48, and the RMSCE is 9.7 for the BNN and 20.3 for the GAN 2201.07766 §6.5. The card says the BNN variant costs nothing extra 2201.07766 §6.5. These are results for one Darcy problem.

Post-training calibration is a separate step. The card's example, on a mixed PDE rather than an operator, cuts RMSCE by about a factor of 3 with 2 to 10 left-out realisations in one case 2201.07766 §6.3. Calibration uses training-distribution data and may not help out of distribution 2201.07766 §5.2. Card: Uncertainty Quantification in Scientific Machine Learning.

Baselines named in the cards

The MWT card's tables also report four other neural operators: GNO, MGNO, LNO and FNO 2109.13459 §3. LNO is described there as a low-rank kernel, similar to an unstacked DeepONet 2109.13459 §3. The card gives no description of the others. Their papers have no card in this wiki, so this page uses only the numbers the MWT card reports for them. FNO's own layer is covered on Fourier neural operator.

The TNO card compares against MLP, UNet, deepONet and FNO, all trained on the same fields 2609.16066 §4.1 2609.16066 tab. 4. Those baselines have no card in this wiki either, so this page cites only the TNO card's numbers for them.

When to use which

Implementation: the MWT code has no license file, so copy nothing from it. The method can be written from the paper's equations and pseudocode 2109.13459 §2.2 2109.13459 §2.3. The TNO card also records no code, so write from its equations. Several TNO details are not stated, listed below.

Implementation sketch

The MWT and FNO layers stack operator layers between a lift and a projection. The TNO variant does not. It has a branch network on the design variables and a trunk network on the query points, and combines them. The training loss for MWT is the relative L2 error 2109.13459 §3. The TNO card uses a mean squared error on normalised fields 2609.16066 §3.3.

# Neural operator, stacked-layer outline  [arxiv:2109.13459 §2.3] [§3]
# Input:  samples a_i = a(x_i) on an N-point grid  [arxiv:2109.13459 §3]
# Output: predicted solution u on the same grid

def neural_operator(a):
    # Map the input to a wider channel space; the paper does not name this step
    x = lift(a)
    for layer in range(num_layers):
        # Variant-specific: MWT splits scales, applies learned kernels
        # per scale and reconstructs  [arxiv:2109.13459 §2.3]
        x = operator_layer(x)
        # Non-linearity between layers  [arxiv:2109.13459 §2.3]
        if layer < num_layers - 1:
            x = relu(x)
    # Map back to one value per grid point
    return project(x)

def relative_l2(u_pred, u_true):
    # Loss and metric  [arxiv:2109.13459 §3]
    return norm(u_pred - u_true) / norm(u_true)

Training settings reported for the MWT runs: Adam, learning rate 1e-3, 500 epochs 2109.13459 §3. Settings reported for the TNO runs: Adam, 900 epochs, learning rate 1e-3 for the first 800 and 1e-4 for the last 100 2609.16066 §3.3.

Where the variants differ:

• Uncertainty. The PA variants keep a DeepONet and add a prior over it. The Darcy runs use HMC for both PA variants and deep ensembles for U-DeepONet 2201.07766 §6.5. The card gives no pseudocode, so write this part from its method section 2201.07766 §4.3.
• Scale handling. MWT replaces the layer with a decomposition, per-scale kernels and a reconstruction 2109.13459 §2.3. For the spectral alternative, see the Fourier neural operator page.
• Branch and trunk. TNO has no stacked layers. Its branch and trunk are combined per query point, and the card's equation 1 passes the products of the branch and trunk features into the output network 2609.16066 eq. 1. The text calls the combination a dot product and also describes a cross multiplication 2609.16066 §3.1. See Contradictions.
• Not stated for TNO. The card does not give the trunk's number of blocks, the embedding, the feed-forward width, the branch depth and width, the feature length p, or the number of attention heads. Its 154 thousand parameter count can serve as a check on a reimplementation 2609.16066 tab. 3 2609.16066 §3.1.
• Grid size. The card's Pitfalls section records that the released MWT code needs grids whose size is a power of two. Check this before relying on resolution independence 2109.13459 §2.3. Card: Multiwavelet-based Operator Learning for Differential Equations.
• Filter precision. The paper recommends k between 1 and 6, and reports numerical drift in the filters above k = 20 2109.13459 app. C.4.

Contradictions

• Size of the gain over FNO. The paper calls the improvement "nearly an order of magnitude". The card's tables show 3.7 times on KdV at s=64, 1.7 times on Burgers at s=256 and 1.2 times on Navier-Stokes against FNO-3D 2109.13459 §3.1 2109.13459 tab. 1 2109.13459 tab. 7 2109.13459 tab. 3. The order-of-magnitude figure matches the gap to GNO, MGNO and LNO, not to FNO 2109.13459 §3.3.
• Abstract against tables. The abstract quotes 0.0033 for KdV and 0.0018 for Burgers. The tables give 0.00338 to 0.00418 and 0.00178 to 0.00199 2109.13459 tab. 1 2109.13459 tab. 7.
• Kernel type. The paper describes the kernels as single-layer convolutions 2109.13459 §2.3. The released 1D code uses Fourier-domain maps for them. The card's Paper vs code section records this 2109.13459 §2.3.
• Baseline numbers. The Burgers and Navier-Stokes FNO and other baseline numbers are taken from earlier published results, not re-run 2109.13459 §3.3 2109.13459 app. D.1. Those comparisons are not like-for-like. The FNO numbers in the TNO card are for a compressor dataset and are not comparable with the MWT card's PDE numbers 2609.16066 tab. 4 2109.13459 tab. 1.
• Speed-up over CFD. The TNO paper states a speed-up of five orders of magnitude. Its own figures, about 800 s per CFD run against about 0.9 ms per prediction, give a ratio near 8.7 × 10^5, which is closer to six orders 2609.16066 §2.2 2609.16066 tab. 3 2609.16066 §4.1.1. The prediction timing's hardware is not stated.
• Branch and trunk combination. Equation 1 passes the p products of branch and trunk features into the output network. The text writes a dot product and also a cross multiplication 2609.16066 eq. 1 2609.16066 §3.1. The card's Pitfalls section records this 2609.16066 §3.1.
• Surrogate comparison. In the TNO card's comparison with conventional surrogates, the neural networks predict fields and derive metrics, while the conventional methods predict the metrics directly. The paper states this asymmetry 2609.16066 §4.1.3 2609.16066 tab. 8.

Papers

• Multiwavelet-based Operator Learning for Differential Equations: a neural operator with a fixed multiwavelet split and learned kernels per scale, compared against GNO, MGNO, LNO and FNO on KdV, Burgers, Darcy and Navier-Stokes.
• Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons: a framework that adds Bayesian methods, ensembles and GAN priors to PINNs and DeepONets, with several calibration-aware metrics and post-training calibration; the Darcy test uses noisy inference data.
• Compressor-cascade performance from a transformer-based operator: a branch-trunk operator with Galerkin attention that predicts five flow fields for one transonic compressor rotor and computes performance metrics from them, compared with MLP, UNet, deepONet, FNO and conventional surrogates.