Physics-Informed Deep Neural Operator Networks

In plain words

Engineers often need to know how a physical system responds to many different conditions: different loads, boundary values or material layouts. Solving the governing equations from scratch for each case is slow. A neural operator is a network that learns the whole mapping from an input field, such as a material map or a boundary condition, to the full solution field. Once trained, it can answer a new condition in one quick pass, without a new solver run.

This chapter reviews three designs. DeepONet combines the outputs of two small networks: one reads the input field at fixed sensor points, the other reads the location where the answer is wanted. The Fourier neural operator does its mixing in frequency space, which suits data on a regular grid. The graph and kernel operators treat the domain as a set of points that exchange messages with their neighbours, which suits irregular meshes. The chapter also shows how to add the governing equation to the training loss, so that less labelled data is needed, and it covers examples in porous flow, cavity flow, cracking and soft tissue.

The chapter argues that these operators can serve as fast surrogates for design, uncertainty analysis and real-time use. Results depend on the example and on how well the training data covers the conditions of interest. Several settings needed to reproduce the examples are missing from the text.

TL;DR

• DeepONet predicts an output at a query point as the dot product of a branch-net vector (from sampled input values) and a trunk-net vector (from the query coordinates) §2 eq. 1.
• Its physics-informed form adds initial, boundary and PDE residual terms to the data loss §2.3 eq. 8. The FNO, IFNO and NKN are Fourier or integral layers on a grid or graph, with features for unstructured geometries §3 §4.2.
• The review's examples report relative L2 errors from about 0.33% (POD-DeepONet, cavity flow) to 16.8% (out-of-distribution tissue) §6. These numbers are adopted from the cited papers, not from new runs here.
• Many training details (learning rates, epochs, loss weights, several network sizes) are not stated in the review, so the examples cannot be reproduced from this card alone §6.

Problem

Physics-informed neural networks (PINNs) are trained for specific boundary and initial conditions and loads. They need costly optimisation at inference, so they generalise poorly to new conditions without further training §1. Reduced-order models are restricted to a small set of conditions §1. The review wants a model that answers many conditions in real time, with little or no retraining §1.

Neural operators address this by learning a map between function spaces. The review stresses three points: DeepONet has a universal approximation result for operators §2, Thm., FNO is universal and resolution independent on regular grids §3 §5, and GNO-type kernel operators can handle unstructured domains §4.2. The review also notes that data-driven operators need large paired data sets, which are often unavailable in engineering, so physics-informed losses are proposed §2.3 §3.3.

Method

DeepONet and extensions §2

Forward pass §2 eq. 1. The input function v is sampled at m fixed sensor points. The branch net maps these values to p coefficients. The trunk net maps the query point xi to p basis values. The output is their dot product:

\mathcal G_\theta(v)(\xi) = \sum_{i=1}^{p} br_i\big(v(\eta_1),\dots,v(\eta_m)\big)\, tr_i(\xi) eq. 1

1. The branch and trunk outputs can be fully connected, residual or convolutional networks, by the operator approximation theorem §2, Thm. (the theorem is cited there to [lu2021_deeponet] and [chen1995universal]).
2. The review says the last layer of each branch net is bias-free in eq. 1, and that adding a bias may lower generalisation error §2.
3. The data loss is a weighted mean of squared errors over N training pairs. The weights are 1 in the simplest case §2 eq. 2.

Self-adaptive point weights §2 eq. 3–eq. 5. Each evaluation point gets a weight λ_j. The loss uses the squared weight:

\mathcal L(\theta,\lambda) = \frac{1}{N}\sum_{i=1}^{N} \lambda_i^{2}\,\big|u_i(\xi) - \mathcal G_\theta(v_i)(\xi)\big|^2 eq. 3

4. The objective is min over θ and max over λ: θ is trained by descent and λ by ascent §2 eq. 4.
5. The weights are updated by gradient ascent with rate η_λ §2 eq. 5. The text describes the weights as increasing and positive, but no projection or clipping rule is given §2.

Feature expansion §2.1. Known structure is put into the trunk or branch net. The review gives four routes: trigonometric features of time in the trunk (for a decaying reaction-diffusion system), an extra input function in the branch (periodic boundary terms in a cavity flow), attention-style averaging over query locations, and a POD basis.

• POD variant §2.1 eq. 6: the trunk is replaced by precomputed POD modes φ_1..φ_p of the mean-removed training outputs, plus the mean function φ_0. The branch learns the coefficients:

\mathcal G(v)(\xi) = \sum_{i=1}^{p} br_i(\eta)\,\phi_i(\xi) + \phi_0(\xi) eq. 6

The review notes that POD bases are hard to use for discontinuous outputs, similar to global spectral methods §2.1.

Multiple inputs §2.2 eq. 7. For several input functions (for example an initial field and boundary values), each has its own branch. Their outputs are multiplied together and then with the trunk output:

\mathcal G_\theta(v,w) = \sum_{i=1}^{p} br^{v}_i\, br^{w}_i\, tr_i eq. 7

Physics-informed DeepONet §2.3 eq. 8. The loss is a data term plus physics terms for the initial condition, the boundary and the PDE residual. The residual or variational form f(G) of the governing law is used. Derivatives come from automatic differentiation, or from finite differences or a Sobel filter for higher-dimensional problems §2.3:

\mathcal L = \mathcal L_{data} + \mathcal L_{init} + \mathcal L_{bound} + \mathcal L_{pde} \mathcal L_{pde} = \frac{1}{N_f}\sum_{i=1}^{N_f}\big[f(\mathcal G_\theta)\big]^2 eq. 8

The initial and boundary terms are squared errors at their own point sets, each averaged over its count §2.3 eq. 8.

Fourier neural operator and extensions §3

Layer §3 eq. 9. The input v is lifted by a shallow map P to a higher-dimensional state h(x,0). Each layer then applies a global Fourier convolution plus a local linear term and a bias, followed by an activation σ. The output is read by a projection Q:

h(x,j+1) = \sigma\!\left( W_j h(x,j) + \mathcal F^{-1}\!\big[R_j \cdot \mathcal F[h(\cdot,j)]\big](x) + c_j \right) eq. 9

1. Input and output must be on an equally spaced lattice, since the FFT needs one §3.
2. W_j, c_j and R_j are learned, and each layer has its own set §3.
3. The review says the lifting P is a linear or shallow map, and Q is a local map §3.

Feature expansion for unstructured or non-grid data §3.1. Three routes are given:

• dFNO+: when input and output domains differ (e.g. an initial condition maps to a space-time field), the input is extended with a time coordinate, or the operator is applied time step by time step, or the boundary input is zero-padded to the whole domain §3.1.
• gFNO+: for non-Cartesian domains, a bounding box is used and the input and output are projected by nearest neighbour; for unstructured meshes, interpolation to a lattice is used §3.1.
• Wavelet neural operators replace the Fourier integral by wavelet integrals §3.1.

Implicit FNO (IFNO) §3.2 eq. 11. Layers are read as time steps of an implicit (autonomous) update, with the same weights in every layer:

h(x,t+\Delta t) = h(x,t) + \Delta t\,\sigma\!\left( W h(x,t) + \mathcal F^{-1}\!\big[R\cdot\mathcal F[h(\cdot,t)]\big](x) + c \right) eq. 11

The shared weights keep the parameter count fixed as depth grows. Shallow-to-deep initialisation is possible because the optimal shallow weights can be reused as the starting point for a deeper network §3.2.

Physics-informed FNO (PINO) §3.3. Derivatives of a grid output cannot come from autodiff of the inverse FFT, so the review lists three options: finite differences or Fourier-spectral gradients, autodiff of the Fourier coefficients at each point with interpolation, or analytic gradients in Fourier space and the chain rule. The review says [li2021physics] computed exact gradients in Fourier space §3.3.

Partial physics for IFNO §3.3. When only partial physics is known, the review adds a penalty term: the operator's output for the zero input is pushed to zero, with a tunable weight λ > 0. This encodes a no-permanent-set assumption for a tissue model §3.3 §6.2.3.

Graph and kernel neural operators §4

Message passing §4.1 eq. 12. A graph layer updates each node from its own state and from aggregated neighbour messages. The review names mean, max, normalised sum, MLP and RNN aggregators, and gives GCN and GAT as examples §4.1:

h_i^{(k+1)} = \mathrm{update}\big(h_i^{(k)}, \mathrm{aggregate}(\{h_j^{(k)} : j \in \mathcal N(i)\})\big) eq. 12

Graph kernel network (GKN) §4.2 eq. 13–eq. 15. The solution operator is written as an integral against a learned kernel κ(x, y, a(x), a(y)), which is taken as the Green's function kernel §4.2 eq. 13. A GKN layer is:

h(x,j+1) = \sigma\!\left( W h(x,j) + \int_\Omega \kappa(x,y,a(x),a(y);\theta)\, h(y,j)\,dy + c \right) eq. 14

On points, the integral becomes an average over neighbours N(x) §4.2 eq. 15:

h(x,j+1) = \sigma\!\left( W h(x,j) + \frac{1}{|N(x)|}\sum_{y\in N(x)} \kappa(x,y,a(x),a(y);\theta)\, h(y,j) + c \right) eq. 15

The text says a full-domain interaction gives O(J²) edges, so cheaper variants use a ball B_r(x), Nyström sampling, multipole or multiwavelet approximations §4.2.

Non-local kernel network (NKN) §4.2 eq. 16. The layer index is treated as time, and each step is a forward step of a non-local diffusion-reaction equation. The kernel gives the diffusion, and a learned reaction β(x) acts as a decay term:

h(x,t+\Delta t) = h(x,t) + \Delta t\Big( \int_\Omega \kappa(x,y)\big(h(y,t)-h(x,t)\big)\,dy - \beta(x)\,h(x,t) + c \Big) eq. 16

The review says the NKN is stable in deep layers when κ is square integrable and non-negative and β is positive and bounded §4.2. It reports that the GKN can lose stability as depth grows §4.2 fig. 6.

Pseudocode

The forward pass of one DeepONet output, following eq. 1:

# DeepONet forward pass for one input function  [§2] [eq. 1]
# Input:  sensors (m,)  values of v at the fixed sensor points  [§2]
#         query (d,)    coordinates xi of the output point  [§2]
# Output: scalar G(v)(xi)

def deeponet_forward(sensors, query, branch, trunk):
    # Branch net gives p coefficients from the sensor values  [§2] [eq. 1]
    coefficients = branch(sensors)                    # (p,)
    # Trunk net gives p basis values at the query point  [§2] [eq. 1]
    basis = trunk(query)                              # (p,)
    # Output is the dot product of the two  [§2] [eq. 1]
    return dot(coefficients, basis)

The self-adaptive weighted loss and its training step, following eq. 3–eq. 5:

# Self-adaptive weighted data loss  [§2] [eq. 3]
# Input:  preds (N,), targets (N,), lambdas (N,) one weight per point

def weighted_loss(preds, targets, lambdas):
    # Squared error at each training point  [§2] [eq. 3]
    errors = (targets - preds) ** 2                   # (N,)
    # Weight each error by the square of its point weight  [§2] [eq. 3]
    return mean(lambdas ** 2 * errors)

# One training step: descent on theta, ascent on lambda  [§2] [eq. 4] [eq. 5]
# The step size for theta is not stated in the review.

def training_step(net, lambdas, batch, lr_theta, eta_lambda):
    # Loss at the current parameters and weights  [§2] [eq. 3]
    loss = weighted_loss(net(batch.inputs), batch.targets, lambdas)
    # Descent step on the network parameters  [§2] [eq. 4]
    net.theta = net.theta - lr_theta * grad(loss, net.theta)
    # Ascent step on the point weights  [§2] [eq. 5]
    lambdas = lambdas + eta_lambda * grad(loss, lambdas)
    return net, lambdas

One FNO layer on a uniform grid, following eq. 9:

# One Fourier layer on a uniform grid  [§3] [eq. 9]
# Input:  hidden (N, c), N grid points, c channels
# Output: hidden (N, c)

def fourier_layer(hidden, W, R, bias):
    # Fourier coefficients of each channel  [§3] [eq. 9]
    spectrum = fft(hidden)                            # (N_k, c), complex
    # Apply R to retained modes; R's shape and cutoff are not stated  [§3]
    spectrum_out = apply_R_to_modes(spectrum, R)      # (N_k, c), complex
    # Back to physical space  [§3] [eq. 9]
    global_part = ifft(spectrum_out)                  # (N, c)
    # Add the pointwise linear term and the bias, then activate  [§3] [eq. 9]
    return sigma(hidden @ W + global_part + bias)     # (N, c)

One explicit step of the NKN layer, following eq. 16:

# One NKN layer step, an explicit Euler step in layer index  [§4.2] [eq. 16]
# Input:  hidden (M, c) at M points, positions (M, d)
# Output: hidden (M, c) at the next layer index

def nkn_step(hidden, positions, dt, kernel, reaction, bias):
    # Nonlocal Laplacian: integral of kappa times (h(y) - h(x))  [§4.2] [eq. 16]
    diffusion = nonlocal_laplacian(hidden, positions, kernel)   # (M, c)
    # Reaction term beta(x) h(x), with beta a learned network  [§4.2] [eq. 16]
    decay = apply(reaction(positions), hidden)                  # (M, c)
    # Explicit update with step dt  [§4.2] [eq. 16]
    return hidden + dt * (diffusion - decay + bias)             # (M, c)

Hyperparameters

The review gives setup details per example in §6, and most are for one example only. Values not listed are not stated. Learning rate, number of epochs, batch size and loss weights are not stated for any example except where noted.

Results

Author-reported, as adopted by the review from the cited papers. The review does not run these experiments; each example's caption says it is adapted from the source named. Setup is in the Hyperparameters table.

The review draws qualitative points from these numbers: GKN loses accuracy near material interfaces and FNO errors cover larger regions §6.1.1 fig. 7; the data-driven DeepONet fails on fracture with 11 samples but is close with 43 §6.2.1; out-of-distribution tissue prediction is about as poor as the constitutive model §6.2.3.

Paper vs code

No official code.

The paper links one code repository: zongyi-li/graph-pde, cited for the Darcy data set §6.1.1. It is a third-party repository and not used here. A GitHub repository search for "DeepONet" in the organisation PredictiveIntelligenceLab found three physics-informed DeepONet repositories. The paper does not link them, and the name matches the authors' lab only by name, which this card did not verify. The one checked, Physics-informed-DeepONets (last push 2022-12-03), has no license on GitHub. Its code is therefore not recorded as code and was not read or quoted. The review's examples come from the cited papers, and no code comparison is made for this chapter.

Pitfalls

• Adopted results. The numbers in §6 are taken from the cited papers, and the captions say so. Use the source for reproduction, not this chapter.
• Missing training details. Learning rate, epochs, batch size, initialisation, input normalisation and loss weights are not stated for any example, except that Adam is named for the Darcy complex DeepONet §6.1.2. The self-adaptive rate η_λ is named but not given a value §2.
• Relative L2 norm. The review says it reports relative L2 error §6 but writes no formula. Whether it averages per sample or divides total norms is not stated. Compare numbers only within one example.
• Notation inconsistencies. The text writes the λ learning rate as η_λ in one place and n_λ in another §2 eq. 5. The FNO feature expansions are named dFNO+, gFNO+ and dgFNO+ in different places §3.1 §6.1.2.
• Self-adaptive weights. The text says λ must increase and stay positive §2, but eq. 3 uses λ² and the update has no clipping or projection. Choose a rule and state it.
• Branch bias. Eq. 1 has no bias term in the last branch layer, while the text says a bias may improve accuracy §2. The review does not say which form each example used.
• Eq. 24 typo. The second line of the hard-constraint formula for the fracture problem is labelled G^u on the left, but its right side is the v-field, so it must mean G^v §6.2.1 eq. 24. The stated boundary data also set v on the bottom edge only, while the hard constraint zeroes v on all four edges, so the top and side conditions for v are not stated §6.2.1 eq. 22.
• Cavity constants. U is named as a constant but never given a value, and the viscosity-to-Re mapping is not stated §6.1.3 eq. 21.
• Boundary GP. The Gaussian process is written over two-dimensional points (x, y), but the kernel is given with x and x' in [0, 1]. The text does not say whether the process runs over one boundary coordinate or over (x, y). Confirm this before implementing §6.1.2 eq. 18.
• Vague setup. "Discontinuity of 5×10⁻³ mm" for the porous medium is not explained. The conductivity truncation and the fracture length scale l₀ are not stated §6.2.1 §6.2.2.
• Mesh and grid limits. FNO needs a regular grid, and unstructured meshes need the projections in §3.1 or a switch to graph operators. The review does not give the error cost of the nearest-neighbour projection.
• Graph cost. A full-domain kernel gives O(J²) edges, and Nyström sampling or multipole approximations trade accuracy for speed. The chosen interaction range for the examples is not stated §4.2.
• Stability claims. The NKN stability argument assumes κ is square integrable and non-negative and β is positive and bounded §4.2. Check these in your own kernel before relying on the result.
• Physics-loss derivatives. For higher dimensions the review suggests finite differences or a Sobel filter §2.3. The review says finite differences need a fine grid, or the numerical error is magnified §3.3. For FNO outputs, autodiff through the grid is not direct §3.3.
• Partial-physics unit. The gain from partial physics is written as "around 1.5%" without saying whether that is percentage points or relative §6.2.3.
• Scope. This is a review. It presents no new method or code, and several claims are the authors' own opinion (for example on digital twins and spiking hardware) §7.

Lineage

• Builds on: DeepONet [lu2021_deeponet] (Lu et al., 2021); the graph kernel network (Li et al., arXiv:2003.03485); the Fourier neural operator (Li et al., arXiv:2010.08895); the non-local kernel network (You et al., arXiv:2201.02217); the implicit FNO [you2022learning]; physics-informed DeepONet [goswami2022physics]; physics-informed FNO (Li et al., arXiv:2111.03794). The arXiv ids are those in the paper's bibliography. None is in this wiki yet.
• Compared against, within the review: GKN, FNO and NKN on Darcy flow §6.1.1; DeepONet, POD-DeepONet and dgFNO+ on the notched domain §6.1.2; DeepONet, POD-DeepONet and dFNO+ on cavity flow §6.1.3; data-driven DeepONet on fracture §6.2.1; a Fung-type constitutive model on tissue §6.2.3.
• Superseded by: not stated in the review.