Energy-based models

Overview

An energy-based model gives each pair of inputs one scalar, the energy. Low energy means the pair fits together. The paper's energy takes an image graph and a candidate scene graph and returns one number for the pair 2103.02221 eq. 6 2103.02221 §3.4.

The motivation is a limit of per-label training. Cross-entropy splits the likelihood of a scene graph into one term per object and one per relation, so each relation is scored without the others. An impossible combination is penalised no more than a plausible wrong label 2103.02221 eq. 1 2103.02221 §1.

The training signal compares whole outputs. It lowers the energy of the true output and compares it with the lowest energy among competing outputs 2103.02221 eq. 2 2103.02221 §3.3. The minimum over all scene graphs cannot be enumerated, so it is approximated by sampling. The paper does this by noisy gradient steps over the object and relation labels, starting from the base model's own prediction 2103.02221 eq. 3 2103.02221 §3.3.

Variants

Energy loss against a sampled competitor (paper form)

The energy is computed by two graph networks, one over the scene graph and one over the image graph. Each runs three rounds of message passing, is pooled with gates, and the two pooled vectors go through a small MLP 2103.02221 §3.4 2103.02221 eq. 6. The loss is the ground-truth energy minus the lowest energy found, plus a squared-energy term that stops energies from growing without bound 2103.02221 eq. 2 2103.02221 eq. 4. The paper adds the base model's own task loss with a weight 2103.02221 eq. 5.

The sampler is stochastic gradient Langevin dynamics. The paper reports 20 iterations, a step size of 1, gradients clipped to [-0.01, 0.01], and node and edge states mapped back to [0, 1] after each update 2103.02221 §4.3. Its update rule uses one λ for both the step and the noise variance, and the paper does not give λ 2103.02221 eq. 3.

Card: Energy-Based Learning for Scene Graph Generation.

Contrastive-divergence form (released code default)

The released code's default loss does not take a minimum. It uses the batch mean of the positive energies minus the batch mean of the energies of one sampled negative, plus the sum of squared energies. The code also has a soft-plus and a log-sum-exp loss that the paper does not describe 2103.02221 § Paper vs code. The paper does not say which loss produced its results 2103.02221 § Pitfalls.

Card: Energy-Based Learning for Scene Graph Generation.

When to use which

The comparisons below use the paper's own tables and text. The paper reports one model family and one sampler setting per table, so they do not show how the method behaves elsewhere.

Implementation sketch

The core is one training step. The sampler is a plain loop, detailed below.

# Energy-based training step, paper form  [arxiv:2103.02221 §3.3] [eq. 2]
# Input:  image I, ground-truth scene graph sg_plus
# Output: scalar loss

def energy_loss_step(image, sg_plus, base, energy):
    # Ground-truth image graph from its object boxes  [§3.3]
    g_plus = region_graph(image, sg_plus.boxes)
    # Base model's own prediction and task loss  [§3.1]
    sg_0, task_loss = base(image)
    g_pred = region_graph(image, sg_0.boxes)
    # Competitor: sampled approximation of the minimum in eq. 2  [eq. 3]
    # Paper reports 20 iterations  [§4.3]
    sg_low = sgld(energy, g_pred, sg_0, steps=20)
    e_plus = energy(g_plus, sg_plus)
    e_low = energy(g_pred, sg_low)
    # Push the true graph's energy below the competitor's  [eq. 2]
    loss_e = e_plus - e_low
    # Keep energies bounded  [eq. 4]
    loss_r = e_plus**2 + e_low**2
    # Weighted total; the weights are not stated  [eq. 5]
    return lam_e * loss_e + lam_r * loss_r + lam_t * task_loss

# Sampler: stochastic gradient Langevin dynamics over labels  [§4.3] [eq. 3]
# Input:  energy, image graph, starting scene graph, iterations
# Output: scene graph with lower energy, states in [0, 1]

def sgld(energy, g_img, sg, steps):
    for _ in range(steps):
        # Gradient of the energy with respect to node and edge states
        g_nodes, g_edges = energy_gradients(energy, g_img, sg)
        # Clip each entry to [-0.01, 0.01]  [§4.3]
        g_nodes = clip(g_nodes, -0.01, 0.01)
        g_edges = clip(g_edges, -0.01, 0.01)
        # Gradient step of size 1 plus Gaussian noise  [§4.3]
        sg.nodes = sg.nodes - g_nodes + gaussian_noise()
        sg.edges = sg.edges - g_edges + gaussian_noise()
        # Map states back to [0, 1]  [§4.3]
        sg.nodes = min_max(sg.nodes)
        sg.edges = min_max(sg.edges)
    return sg

Notes on the sketch. The paper's update uses λ/2 for the step and N(0, λ) for the noise, with λ never given. Section 4.3 gives a step of 1 instead, so the two settings are not reconciled in the paper 2103.02221 eq. 3 2103.02221 §4.3. The min-max step divides by the row maximum after subtracting the minimum, so a constant row gives a zero divisor. Add a guard if you reimplement it 2103.02221 § Pitfalls.

Contradictions

• Text versus table, on the headline gains. The text gives VCTree a 12.7% gain on PredCls mR@20 2103.02221 §5. From the table the same cell goes from 13.07 to 14.20, which is +8.6% 2103.02221 tab. 1.
• Abstract maxima versus table cells. The abstract says "up to 21%" on Visual Genome and "27%" on GQA 2103.02221 § Abstract. No single cell matches these numbers. Computed from 2103.02221 tab. 1, the largest gain on Visual Genome is +21.9% (VCTree-TDE, PredCls mR@20), and on GQA it is +37.2% (Transformer, SGCls mR@100). Quote a figure with its cell.
• Image graph ablation. The text says removing the image graph lowers recall 2103.02221 §5. At mR@20 the gap is 0.01, 14.20 against 14.19. At mR@50 and mR@100 it is 0.14 and 0.17 2103.02221 tab. 5. The paper gives one run per setting and no variance, so the size of the effect is not established.
• Paper form versus code default. The loss and learning rate differ between the equations and the released code [arxiv:2103.02221 § Paper vs code]. This page does not decide which one the paper's numbers came from.

Papers

• Energy-Based Learning for Scene Graph Generation: trains scene graph generators with an energy loss. A graph network scores each image and scene graph pair, and Langevin-style sampling finds the lowest-energy competitor.