Automatic differentiation

Overview

Automatic differentiation (AD) gives the derivative of a computed quantity with respect to its inputs, using the same computation graph that produces the quantity 2110.15832 §1. Two compiled papers use it in different roles.

In the first, AD is part of the model: a network predicts a scalar free energy, and the stress and the dissipation are taken as its partial derivatives 2005.12183 §3.3 2005.12183 eq. 13. Because the outputs are derivatives of one potential, the thermodynamic identity holds for any weights 2005.12183 §2 2005.12183 §5.1.2.

In the second, AD is part of the loss: a physics-informed network penalises the residual of a flow equation at sample points, and the derivatives in that residual come from AD 2110.15832 §2.1. Here AD is exact only at the point itself. The paper shows that, with few sample points, the network can satisfy the residual at every point and still be far from the true flow 2110.15832 §1 2110.15832 §3.2.1.

Both papers also show a cost. In the first, the second derivatives of the activations enter the training when the loss compares derivatives with data, so an activation whose second derivative vanishes over part of its range can train badly 2005.12183 §3.2 2005.12183 tab. 2. In the second, each residual calls the network at several neighbours, which makes an iteration slower 2110.15832 §2.2 2110.15832 fig. 11c.

Variants

Derivative of a learned energy potential

The network outputs a scalar potential F from the state. Derived quantities are obtained by differentiating F with respect to the state variables. Stress is the derivative with respect to strain, and dissipation is minus the sum of the derivatives with respect to internal variables, multiplied by their rates 2005.12183 eq. 13 2005.12183 §3.3, steps 5–6. The variant is used for constitutive models of strain-rate-independent materials at the material-point level 2005.12183 §2.3. Card: Thermodynamics-based Artificial Neural Networks for constitutive modeling.

Activation choice matters here because the loss depends on the derivatives. The paper's energy network uses ELU with the squared argument, whose second derivative is nonzero everywhere 2005.12183 tab. 1 2005.12183 §5.1.1. In the paper's toy test, the squared-argument variants converged in far fewer epochs than the plain ReLU and ELU versions 2005.12183 tab. 2.

Coupled AD and neighbour differences in a PINN loss

Here the derivative at a collocation point is a mix of two estimates. The first is the exact AD slope at the point. The second is a finite-difference style estimate from neighbouring network outputs at x ± Δx and x ± 2Δx 2110.15832 §2.2 2110.15832 eq. 5b. The face value is a linear combination of the network output and its AD slope, with coefficients fixed by a Taylor expansion 2110.15832 eq. 6a 2110.15832 eq. 7a–b. For the second-order upwind stencil, with positive flow, the east face value is the network output plus half the spacing times the AD slope 2110.15832 eq. 8a. The paper reports second-order accuracy for this scheme 2110.15832 §2.3.1. Pressure gradients use a central variant that corrects the face value with the difference of AD slopes at neighbouring nodes 2110.15832 eq. 12a. Card: CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method.

The authors' results are on fluid problems. On the lid-driven cavity at Re = 400 with 2,601 points, they report an MSE about two orders of magnitude lower than the AD-only network, read from a figure 2110.15832 §3.3.1 2110.15832 fig. 10a. With 40,401 points the AD-only network reaches about 2e-4, and the coupled network goes below 1e-5 2110.15832 §3.3.2 2110.15832 fig. 11a. On the backward-facing step, the reattachment length is 2.604 against a reference of 2.608, versus 4.761 for the AD-only network 2110.15832 tab. 3. These are the authors' numbers.

When to use which

Implementation sketch

Two patterns, each with its own core. The first derives the stress and the dissipation from the energy network. The second builds a face value from the network output and its AD slope.

The derivative of the potential is taken with respect to the state. In the energy paper's setting, the derivative is partial with respect to the strain and internal-variable inputs, not total. A total derivative would also pass through the network that predicts the internal-variable update, and the paper does not say which form it uses 2005.12183 §3.3. The sketch below uses the partial form.

# Derivative-of-potential pattern  [arxiv:2005.12183 §3.3]
# energy_net maps (strain, internal variables) to a scalar energy
# autodiff_grad returns d(energy)/d(input) for each input, by AD

def tann_update(strain, strain_increment, stress, zetas, dt, nets):
    # 1. Updated strain  [§3.3, step 1]
    strain_next = strain + strain_increment
    # 2. Predict internal-variable increments with sNN_zeta  [§3.3, step 2]
    zeta_increments = nets.zeta(strain_next, strain_increment, stress, zetas)
    # 3. Updated internal variables and their rates  [§3.3, step 3]
    zetas_next = zetas + zeta_increments
    zeta_rates = zeta_increments / dt
    # 4. Predict the energy potential with sNN_F  [§3.3, step 4]
    energy = nets.F(strain_next, zetas_next)
    # 5. Dissipation from the energy potential  [§3.3, step 5] [eq. 13]
    dissipation = -dot(partial(energy, zetas_next), zeta_rates)
    # 6. Stress from the energy potential, and its increment  [§3.3, step 6]
    stress_next = partial(energy, strain_next)
    stress_increment = stress_next - stress
    return stress_increment, dissipation, energy

# Coupled face value, second-order upwind, positive flow  [arxiv:2110.15832 eq. 8a]
# Inputs: u_hat and its AD slope u_x at the node x; dx is the grid spacing
# Output: u_e, the value at the east face

def coupled_east_face(u_hat, u_x, dx):
    # Network output at the node, with coefficient a1 = 1  [eq. 6a] [eq. 7a]
    value = u_hat
    # Add the AD slope with coefficient a2 = dx / 2  [eq. 7b] [eq. 8a]
    value = value + (dx / 2.0) * u_x
    return value

# The derivative is then (u_e - u_w) / dx  [arxiv:2110.15832 eq. 9]
# The paper derives only the positive-flow branch; a negative face velocity
# needs a mirror formula that the paper does not state  [eq. 8a] [eq. 8b]

Contradictions

Within the energy paper, the text calls ReLU and ELU with a linear argument "fail", while the table shows that they converge, with larger errors 2005.12183 §3.2.1 2005.12183 tab. 2. This page uses the table values. Across the two cards, none known.

Papers

• Thermodynamics-based Artificial Neural Networks for constitutive modeling: derives stress and dissipation as derivatives of a learned free-energy network, with activation constraints from the second-order derivative in the loss.
• CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method: mixes AD slopes with neighbour-point differences in a PINN residual, so sparse collocation points constrain the flow; the AD-only loss can be misleading.