---
kind: concept
slug: energy-based-models
title: Energy-based models
summary: >-
  Scores a whole structured output with one energy, low for compatible outputs, and trains by pushing the true output's energy below a competitor's. One paper so far: an energy loss on scene graph generators, with Langevin-style sampling for the competitor. Author-reported gains in mean Recall@K on Visual Genome and GQA, at 20 extra passes per training step.
tags: [energy-based-learning, structured-prediction, sampling]
papers: ["2103.02221"]
related: [langevin-dynamics, scene-graph-generation, graph-neural-network, constrained-decoding]
status: draft
updated_at: 2026-11-09
---

## Overview

An energy-based model gives each pair of inputs one scalar, the energy. Low
energy means the pair fits together. The paper's energy takes an image graph
and a candidate scene graph and returns one number for the pair
[arxiv:2103.02221 eq. 6] [arxiv:2103.02221 §3.4].

The motivation is a limit of per-label training. Cross-entropy splits the
likelihood of a scene graph into one term per object and one per relation,
so each relation is scored without the others. An impossible combination is
penalised no more than a plausible wrong label [arxiv:2103.02221 eq. 1]
[arxiv:2103.02221 §1].

The training signal compares whole outputs. It lowers the energy of the true
output and compares it with the lowest energy among competing outputs
[arxiv:2103.02221 eq. 2] [arxiv:2103.02221 §3.3]. The minimum over all scene
graphs cannot be enumerated, so it is approximated by sampling. The paper
does this by noisy gradient steps over the object and relation labels,
starting from the base model's own prediction [arxiv:2103.02221 eq. 3]
[arxiv:2103.02221 §3.3].

## Variants

### Energy loss against a sampled competitor (paper form)

The energy is computed by two graph networks, one over the scene graph and
one over the image graph. Each runs three rounds of message passing, is
pooled with gates, and the two pooled vectors go through a small MLP
[arxiv:2103.02221 §3.4] [arxiv:2103.02221 eq. 6]. The loss is the
ground-truth energy minus the lowest energy found, plus a squared-energy term
that stops energies from growing without bound [arxiv:2103.02221 eq. 2]
[arxiv:2103.02221 eq. 4]. The paper adds the base model's own task loss with
a weight [arxiv:2103.02221 eq. 5].

The sampler is stochastic gradient Langevin dynamics. The paper reports 20
iterations, a step size of 1, gradients clipped to [-0.01, 0.01], and node
and edge states mapped back to [0, 1] after each update
[arxiv:2103.02221 §4.3]. Its update rule uses one λ for both the step and the
noise variance, and the paper does not give λ [arxiv:2103.02221 eq. 3].

Card: [Energy-Based Learning for Scene Graph Generation](/papers/2103.02221).

### Contrastive-divergence form (released code default)

The released code's default loss does not take a minimum. It uses the batch
mean of the positive energies minus the batch mean of the energies of one
sampled negative, plus the sum of squared energies. The code also has a
soft-plus and a log-sum-exp loss that the paper does not describe
[arxiv:2103.02221 § Paper vs code]. The paper does not say which loss
produced its results [arxiv:2103.02221 § Pitfalls].

Card: [Energy-Based Learning for Scene Graph Generation](/papers/2103.02221).

## When to use which

The comparisons below use the paper's own tables and text. The paper reports
one model family and one sampler setting per table, so they do not show how
the method behaves elsewhere.

| Situation | Use | Why |
|---|---|---|
| Scene graph model with per-label losses, and inconsistent graphs must be penalised | Paper-form energy loss added to the base model's task loss | The motivation is stated in [arxiv:2103.02221 §1]. Mean Recall@20 rises for all four base models on Visual Genome in PredCls, e.g. VCTree 13.07 to 14.20 and VCTree-TDE 16.30 to 19.87 [arxiv:2103.02221 tab. 1] |
| Few training examples per triplet | Paper-form energy loss | Gains are largest in the lowest bucket: 16.9 to 18.55 Recall@20 for 1-5 triplets, and 0.26 for 20-25 [arxiv:2103.02221 tab. 3]. The paper's few-shot setup is underspecified [arxiv:2103.02221 § Pitfalls] |
| Tight training budget | Energy loss with fewer sampler steps, or the base model alone | Each training step runs 20 forward and backward passes of the energy model [arxiv:2103.02221 §4.3]. With 0 sampler steps VCTree gets 14.18 mR@20, against 14.20 with 20 steps [arxiv:2103.02221 tab. 5], a single run per setting |
| Reproducing a reported number | Choose one variant and state which one | The paper form and the code default differ, and the paper does not say which produced Table 1 [arxiv:2103.02221 § Paper vs code] [arxiv:2103.02221 § Pitfalls] |
| Using the released code | Implement from the paper and card, and read the code licence first | The code's licence is unresolved: the LICENSE file has no full text. The card quotes no code [arxiv:2103.02221 § Pitfalls] |

## Implementation sketch

The core is one training step. The sampler is a plain loop, detailed below.

```pseudocode
# Energy-based training step, paper form  [arxiv:2103.02221 §3.3] [eq. 2]
# Input:  image I, ground-truth scene graph sg_plus
# Output: scalar loss

def energy_loss_step(image, sg_plus, base, energy):
    # Ground-truth image graph from its object boxes  [§3.3]
    g_plus = region_graph(image, sg_plus.boxes)
    # Base model's own prediction and task loss  [§3.1]
    sg_0, task_loss = base(image)
    g_pred = region_graph(image, sg_0.boxes)
    # Competitor: sampled approximation of the minimum in eq. 2  [eq. 3]
    # Paper reports 20 iterations  [§4.3]
    sg_low = sgld(energy, g_pred, sg_0, steps=20)
    e_plus = energy(g_plus, sg_plus)
    e_low = energy(g_pred, sg_low)
    # Push the true graph's energy below the competitor's  [eq. 2]
    loss_e = e_plus - e_low
    # Keep energies bounded  [eq. 4]
    loss_r = e_plus**2 + e_low**2
    # Weighted total; the weights are not stated  [eq. 5]
    return lam_e * loss_e + lam_r * loss_r + lam_t * task_loss
```

```pseudocode
# Sampler: stochastic gradient Langevin dynamics over labels  [§4.3] [eq. 3]
# Input:  energy, image graph, starting scene graph, iterations
# Output: scene graph with lower energy, states in [0, 1]

def sgld(energy, g_img, sg, steps):
    for _ in range(steps):
        # Gradient of the energy with respect to node and edge states
        g_nodes, g_edges = energy_gradients(energy, g_img, sg)
        # Clip each entry to [-0.01, 0.01]  [§4.3]
        g_nodes = clip(g_nodes, -0.01, 0.01)
        g_edges = clip(g_edges, -0.01, 0.01)
        # Gradient step of size 1 plus Gaussian noise  [§4.3]
        sg.nodes = sg.nodes - g_nodes + gaussian_noise()
        sg.edges = sg.edges - g_edges + gaussian_noise()
        # Map states back to [0, 1]  [§4.3]
        sg.nodes = min_max(sg.nodes)
        sg.edges = min_max(sg.edges)
    return sg
```

Notes on the sketch. The paper's update uses λ/2 for the step and N(0, λ)
for the noise, with λ never given. Section 4.3 gives a step of 1 instead, so
the two settings are not reconciled in the paper [arxiv:2103.02221 eq. 3]
[arxiv:2103.02221 §4.3]. The min-max step divides by the row maximum after
subtracting the minimum, so a constant row gives a zero divisor. Add a guard
if you reimplement it [arxiv:2103.02221 § Pitfalls].

## Contradictions

- **Text versus table, on the headline gains.** The text gives VCTree a
  12.7% gain on PredCls mR@20 [arxiv:2103.02221 §5]. From the table the same
  cell goes from 13.07 to 14.20, which is +8.6% [arxiv:2103.02221 tab. 1].
- **Abstract maxima versus table cells.** The abstract says "up to 21%" on
  Visual Genome and "27%" on GQA [arxiv:2103.02221 § Abstract]. No single
  cell matches these numbers. Computed from [arxiv:2103.02221 tab. 1], the
  largest gain on Visual Genome is +21.9% (VCTree-TDE, PredCls mR@20), and on
  GQA it is +37.2% (Transformer, SGCls mR@100). Quote a figure with its cell.
- **Image graph ablation.** The text says removing the image graph lowers
  recall [arxiv:2103.02221 §5]. At mR@20 the gap is 0.01, 14.20 against
  14.19. At mR@50 and mR@100 it is 0.14 and 0.17 [arxiv:2103.02221 tab. 5].
  The paper gives one run per setting and no variance, so the size of the
  effect is not established.
- **Paper form versus code default.** The loss and learning rate differ
  between the equations and the released code [arxiv:2103.02221 § Paper vs
  code]. This page does not decide which one the paper's numbers came from.

## Papers

- [Energy-Based Learning for Scene Graph Generation](/papers/2103.02221): trains scene graph generators with an energy loss. A graph network scores each image and scene graph pair, and Langevin-style sampling finds the lowest-energy competitor.