---
kind: concept
slug: automatic-differentiation
title: Automatic differentiation
summary: >-
  Taking exact derivatives of a network's outputs with automatic differentiation (AD) and building them into the model or its loss. Two patterns: a learned free-energy potential whose derivatives give stress, so the first law holds by construction; and PINN residuals where AD slopes are mixed with neighbour-point differences so sparse samples still constrain the flow.
tags: [differentiation, physics-constraints, constitutive-modeling, pde]
papers: ["2005.12183", "2110.15832"]
related: [physics-informed-neural-network, surrogate-modeling]
status: draft
updated_at: 2026-11-01
---

## Overview

Automatic differentiation (AD) gives the derivative of a computed quantity with
respect to its inputs, using the same computation graph that produces the
quantity [arxiv:2110.15832 §1]. Two compiled papers use it in different roles.

In the first, AD is part of the model: a network predicts a scalar free energy,
and the stress and the dissipation are taken as its partial derivatives
[arxiv:2005.12183 §3.3] [arxiv:2005.12183 eq. 13]. Because the outputs are
derivatives of one potential, the thermodynamic identity holds for any weights
[arxiv:2005.12183 §2] [arxiv:2005.12183 §5.1.2].

In the second, AD is part of the loss: a physics-informed network penalises the
residual of a flow equation at sample points, and the derivatives in that
residual come from AD [arxiv:2110.15832 §2.1]. Here AD is exact only at the
point itself. The paper shows that, with few sample points, the network can
satisfy the residual at every point and still be far from the true flow
[arxiv:2110.15832 §1] [arxiv:2110.15832 §3.2.1].

Both papers also show a cost. In the first, the second derivatives of the
activations enter the training when the loss compares derivatives with data,
so an activation whose second derivative vanishes over part of its range can
train badly [arxiv:2005.12183 §3.2] [arxiv:2005.12183 tab. 2]. In the second,
each residual calls the network at several neighbours, which makes an iteration
slower [arxiv:2110.15832 §2.2] [arxiv:2110.15832 fig. 11c].

## Variants

### Derivative of a learned energy potential

The network outputs a scalar potential F from the state. Derived quantities are
obtained by differentiating F with respect to the state variables. Stress is
the derivative with respect to strain, and dissipation is minus the sum of the
derivatives with respect to internal variables, multiplied by their rates
[arxiv:2005.12183 eq. 13] [arxiv:2005.12183 §3.3, steps 5–6]. The variant is
used for constitutive models of strain-rate-independent materials at the
material-point level [arxiv:2005.12183 §2.3]. Card:
[Thermodynamics-based Artificial Neural Networks for constitutive modeling](/papers/2005.12183).

Activation choice matters here because the loss depends on the derivatives. The
paper's energy network uses ELU with the squared argument, whose second
derivative is nonzero everywhere [arxiv:2005.12183 tab. 1]
[arxiv:2005.12183 §5.1.1]. In the paper's toy test, the squared-argument
variants converged in far fewer epochs than the plain ReLU and ELU versions
[arxiv:2005.12183 tab. 2].

### Coupled AD and neighbour differences in a PINN loss

Here the derivative at a collocation point is a mix of two estimates. The
first is the exact AD slope at the point. The second is a finite-difference
style estimate from neighbouring network outputs at x ± Δx and x ± 2Δx
[arxiv:2110.15832 §2.2] [arxiv:2110.15832 eq. 5b]. The face value is a linear
combination of the network output and its AD slope, with coefficients fixed by
a Taylor expansion [arxiv:2110.15832 eq. 6a] [arxiv:2110.15832 eq. 7a–b]. For
the second-order upwind stencil, with positive flow, the east face value is the
network output plus half the spacing times the AD slope [arxiv:2110.15832 eq. 8a].
The paper reports second-order accuracy for this scheme [arxiv:2110.15832 §2.3.1].
Pressure gradients use a central variant that corrects the face value with the
difference of AD slopes at neighbouring nodes [arxiv:2110.15832 eq. 12a]. Card:
[CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method](/papers/2110.15832).

The authors' results are on fluid problems. On the lid-driven cavity at
Re = 400 with 2,601 points, they report an MSE about two orders of magnitude
lower than the AD-only network, read from a figure [arxiv:2110.15832 §3.3.1]
[arxiv:2110.15832 fig. 10a]. With 40,401 points the AD-only network reaches
about 2e-4, and the coupled network goes below 1e-5 [arxiv:2110.15832 §3.3.2]
[arxiv:2110.15832 fig. 11a]. On the backward-facing step, the reattachment
length is 2.604 against a reference of 2.608, versus 4.761 for the AD-only
network [arxiv:2110.15832 tab. 3]. These are the authors' numbers.

## When to use which

| Situation | Use | Why |
|---|---|---|
| A learned quantity must obey an identity that links it to a derivative (energy to stress) | Derivative of a learned potential | The identity holds for any weights, not only for the training data [arxiv:2005.12183 §5.1.2] |
| The loss compares network derivatives with data | Activation with a nonzero second derivative everywhere | Piecewise-linear or partly flat activations did worse in the paper's toy test [arxiv:2005.12183 tab. 2] |
| A PINN has few collocation points and an AD-only residual reaches a small value | Coupled AD and neighbour differences | The AD-only loss can be small while the solution is wrong; the coupled form ties neighbouring points together [arxiv:2110.15832 §1] [arxiv:2110.15832 §3.3.2] |
| The coupled PINN is limited by iteration time with few CPUs | Consider the cost before choosing it | The paper reports the coupled variant as the slowest per iteration with 4 CPUs per network, and similar cost with over 20 CPUs [arxiv:2110.15832 fig. 11c] |
| Dense sampling is affordable | The AD-only residual as the baseline; check the coupled variant at the same density | The coupled variant still had lower error at 40,401 points (below 1e-5, against about 2e-4 for AD-only), but the paper gives no cost-benefit verdict [arxiv:2110.15832 §3.3.2] [arxiv:2110.15832 fig. 11a] |
| Predictions are needed for inputs beyond the training range | Check the tangent as well as the energy | Accuracy drops out of range, but the authors report the tangent is still well predicted, from plots only [arxiv:2005.12183 §5.1.3] |
| Non-isothermal or finite-strain problems | Not covered by these cards | The energy paper's examples are isothermal, and finite strain was not developed [arxiv:2005.12183 §2.3] [arxiv:2005.12183 §5 intro] |

## Implementation sketch

Two patterns, each with its own core. The first derives the stress and the
dissipation from the energy network. The second builds a face value from the
network output and its AD slope.

The derivative of the potential is taken with respect to the state. In the
energy paper's setting, the derivative is partial with respect to the strain
and internal-variable inputs, not total. A total derivative would also pass
through the network that predicts the internal-variable update, and the paper
does not say which form it uses [arxiv:2005.12183 §3.3]. The sketch below uses
the partial form.

```pseudocode
# Derivative-of-potential pattern  [arxiv:2005.12183 §3.3]
# energy_net maps (strain, internal variables) to a scalar energy
# autodiff_grad returns d(energy)/d(input) for each input, by AD

def tann_update(strain, strain_increment, stress, zetas, dt, nets):
    # 1. Updated strain  [§3.3, step 1]
    strain_next = strain + strain_increment
    # 2. Predict internal-variable increments with sNN_zeta  [§3.3, step 2]
    zeta_increments = nets.zeta(strain_next, strain_increment, stress, zetas)
    # 3. Updated internal variables and their rates  [§3.3, step 3]
    zetas_next = zetas + zeta_increments
    zeta_rates = zeta_increments / dt
    # 4. Predict the energy potential with sNN_F  [§3.3, step 4]
    energy = nets.F(strain_next, zetas_next)
    # 5. Dissipation from the energy potential  [§3.3, step 5] [eq. 13]
    dissipation = -dot(partial(energy, zetas_next), zeta_rates)
    # 6. Stress from the energy potential, and its increment  [§3.3, step 6]
    stress_next = partial(energy, strain_next)
    stress_increment = stress_next - stress
    return stress_increment, dissipation, energy
```

```pseudocode
# Coupled face value, second-order upwind, positive flow  [arxiv:2110.15832 eq. 8a]
# Inputs: u_hat and its AD slope u_x at the node x; dx is the grid spacing
# Output: u_e, the value at the east face

def coupled_east_face(u_hat, u_x, dx):
    # Network output at the node, with coefficient a1 = 1  [eq. 6a] [eq. 7a]
    value = u_hat
    # Add the AD slope with coefficient a2 = dx / 2  [eq. 7b] [eq. 8a]
    value = value + (dx / 2.0) * u_x
    return value

# The derivative is then (u_e - u_w) / dx  [arxiv:2110.15832 eq. 9]
# The paper derives only the positive-flow branch; a negative face velocity
# needs a mirror formula that the paper does not state  [eq. 8a] [eq. 8b]
```

## Contradictions

Within the energy paper, the text calls ReLU and ELU with a linear argument
"fail", while the table shows that they converge, with larger errors
[arxiv:2005.12183 §3.2.1] [arxiv:2005.12183 tab. 2]. This page uses the table
values. Across the two cards, none known.

## Papers

- [Thermodynamics-based Artificial Neural Networks for constitutive modeling](/papers/2005.12183): derives stress and dissipation as derivatives of a learned free-energy network, with activation constraints from the second-order derivative in the loss.
- [CAN-PINN: A Fast Physics-Informed Neural Network Based on Coupled-Automatic-Numerical Differentiation Method](/papers/2110.15832): mixes AD slopes with neighbour-point differences in a PINN residual, so sparse collocation points constrain the flow; the AD-only loss can be misleading.