Scene graph generation

Overview

A scene graph is a graph built from an image. Each detected object is a node with a category, and each relation is a directed edge that becomes a subject-predicate-object triplet 2104.01111 §1.1 2104.01111 eq. 1. Scene graph generation (SGG) is the task of producing that graph from pixels.

The invariant every image-based variant keeps: objects are found first, then a predicate is decided for each ordered pair of objects, and pairs without a real relation are dropped 2104.01111 §1.2. The variants differ in how object and relation decisions share information. Most recent methods model the two jointly through context rather than independently 2104.01111 §2.6.

A second kind of variant does not classify predicates at all. It builds an interaction graph from tracked hand and object positions in human video, with edges set by thresholds on signal statistics 2508.05342 §3.2. See the last variant below.

Relations are unevenly distributed. Common predicates dominate the annotations, and rare ones are many and hard to learn, so a model that only fits frequent predicates scores well on plain recall and poorly on the rare ones 2104.01111 §4 2104.01111 §6.2.

Variants

Bottom-up, two-stage pipeline

Detect boxes, label the objects, then predict predicates for the detected pairs. The joint probability factorises into a box term, an object term given the boxes, and a relation term given boxes and objects 2104.01111 §1.2 2104.01111 eq. 7. Card: A Comprehensive Survey of Scene Graphs: Generation and Application.

Top-down, joint pipeline

Detect boxes, then infer objects and relations together from the box proposals 2104.01111 §1.2 2104.01111 eq. 8. Card: A Comprehensive Survey of Scene Graphs: Generation and Application.

One-stage message passing with priority-aware loss (GPS-Net)

Faster R-CNN proposals and their features form the nodes of a graph. Direction- aware message passing scores each ordered pair with a tri-linear model over the subject, object and union features, and sends each neighbour's feature along both directions of the pair 2003.12962 §3.1 2003.12962 eq. 3-4. A node-priority loss gives each object a focusing parameter from its share of the ground-truth triplets, capped at 2 2003.12962 §3.2 2003.12962 eq. 6-8. The predicate head adds a softened class-pair frequency prior, gated per pair by the union feature 2003.12962 §3.3 2003.12962 eq. 9-10. The paper describes the detector and the graph as one network trained jointly 2003.12962 §3. Card: GPS-Net: Graph Property Sensing Network for Scene Graph Generation. Its own ablation on VG adds each part to a baseline; the message-passing part gives the largest single gain in SGCls R@100 2003.12962 tab. 5.

Dense graph over detector queries (DETR-style)

Keep a DETR-style detector, then take its N object queries as graph nodes. A dense graph transformer updates node and edge features over all N x N pairs, and the edge read-out gives one relation logit per ordered pair 2409.02035 §4.1 2409.02035 §4.2. Training matches queries to ground-truth objects with Hungarian matching, then applies a binary cross-entropy loss to the relations of the matched queries only 2409.02035 §4.3 2409.02035 eq. 5-9. The target is dependency between objects in a bin: which one must come out first. Compared with object-pair pooling, which crops the feature map over the union of two boxes, this variant pairs query embeddings instead 2409.02035 §1. Card: A Modern Take on Visual Relationship Reasoning for Grasp Planning.

Model families

The survey describes five families. It does not rank them.

• CRF-based: detection and predicate prediction as a conditional random field 2104.01111 §2.1. See conditional random field.
• TransE-based: each relation is treated as a translation between object embeddings 2104.01111 §2.2.
• CNN-based: convolutional features over object pairs, with context and zoom modules 2104.01111 §2.3.
• RNN/LSTM-based: sequential context over the objects, with chain, fully connected, subgraph and tree orderings 2104.01111 §2.4.
• GNN-based: message passing over the object and relation graph 2104.01111 §2.5. See graph neural network.

The survey does not cover the one-stage message-passing variant above, so it has no family label for it here.

Add-on: prior knowledge

Language priors, statistical priors from how annotated objects co-occur, and knowledge graphs. The survey suggests these help rare relations most 2104.01111 §3 2104.01111 §3.1 2104.01111 §3.2 2104.01111 §3.3. The frequency prior in the one-stage variant above is of the statistical kind: class-pair counts from the training annotations, softened and then gated per pair 2003.12962 §3.3.

Add-on: long-tailed predicates

Zero-, one- and few-shot learning, causal debiasing, and an energy-based loss over joint object and relation likelihood 2104.01111 §4. See long-tailed recognition. A per-node focusing term is a related idea: a focal-style loss whose focusing parameter depends on the node's role in the graph 2003.12962 §3.2. See focal loss.

Interaction graph from human video (rule-based edges)

Nodes are the two hands and the tracked objects. Each hand-object edge is labelled Coupled-Motion (the hand moves the object) or Docked, and each object-object edge is labelled E-OO (task-relevant) or T-OO (transitory). The labels come from distance gates and from the entropy and mutual information of the position signals, not from a learned predicate classifier 2508.05342 §3.2.1 2508.05342 §3.2.2 2508.05342 eq. 1 2508.05342 eq. 3. The edge thresholds are only partly stated: the distance gates are 0.15 m and 0.2 m, and the two MI thresholds are not 2508.05342 §5.1. The graph feeds a vision-language-action planner, see Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control and vision-language-action model. Its graph accuracy is measured against the authors' own annotations on their own block-assembly video, not on a scene-graph benchmark 2508.05342 §6.1 2508.05342 eq. 9.

When to use which

The survey gives no single recommendation. The table follows the stated constraints and reported results of the cited papers.

Reported numbers from the survey, author-reported on VG150 with graph constraints. Units are not stated in the table. Dashes mean no value in the source table 2104.01111 tab. 2:

The energy-based loss row also lists SGCls mR@100 of 20.45 and SGDet mR@100 of 11.6 2104.01111 tab. 2.

GPS-Net's own VG table compares it with methods that use the Neural Motifs detector (marked ◇ there), and with RelDN's backbone (‡). Its "Mean" column is an average of recall values, not of mean recall; the same column in the GPS-Net table therefore does not measure mR. Values below are author-reported 2003.12962 tab. 1 2003.12962 tab. 2:

Other results in the same paper, also against RelDN with the same detector: on OpenImages the score_wtd is 47.03 against 44.61 2003.12962 tab. 3, and on VRD with a COCO backbone the relation R@50 is 27.8 against 25.3 2003.12962 tab. 4. Its ablation on VG reports PredCls R@100 of 67.3 for the baseline and 68.8 for the full model, and SGCls R@100 of 36.3 and 40.1 2003.12962 tab. 5.

Implementation sketch

The bottom-up pipeline is the common core. Each variant changes the boxes, the features or the predicate head. The sketch shows only that core.

# Bottom-up scene graph generation  [arxiv:2104.01111 §1.2] [eq. 7]
# Input:  image I
# Output: boxes, object labels, and a predicate for each kept pair

def scene_graph(image):
    # 1. Propose boxes and per-box features  [arxiv:2104.01111 §1.2]
    boxes, feats = detector(image)                # feats: (n, d)

    # 2. Label each object  [arxiv:2104.01111 §1.2] [eq. 7]
    labels = classify_objects(feats)              # one class per box

    # 3. Score a predicate for every ordered pair  [arxiv:2104.01111 §1.2]
    for i, j in ordered_pairs(boxes):             # i != j
        rel[i, j] = predicate_head(feats[i], feats[j], labels[i], labels[j])

    # 4. Keep pairs that carry a real relation  [arxiv:2104.01111 §1.1]
    return boxes, labels, drop_background(rel)

Where variants differ: a top-down model replaces steps 2 and 3 with one joint step over the proposals 2104.01111 eq. 8. A dense query-graph model keeps the detector heads, feeds its N query embeddings into a graph transformer, and reads one relation logit per ordered pair 2409.02035 §4.2. A GNN or RNN variant replaces the independent pair scores with messages passed along the graph 2104.01111 §2.4 2104.01111 §2.5. The one-stage message-passing variant adds direction- aware messages before the pair head and a per-node focusing term to the object loss 2003.12962 §3.1 2003.12962 §3.2. Prior-knowledge and long-tail methods add terms to the predicate head 2104.01111 §3 2104.01111 §4. The interaction-graph variant has no predicate head: its edges come from threshold rules on hand and object signals 2508.05342 §3.2.1.

Contradictions

Numbers in the tables above come from different papers, not from one controlled experiment. Each row reflects its own backbone, training split and evaluation details, and the survey says its table is not a ranking 2104.01111 §6.2 2104.01111 tab. 2. Other disagreements matter when comparing papers:

• The same baseline has different numbers in two papers. The survey lists IMP with PredCls R@100 53.0, SGCls R@100 24.4 and SGDet R@100 4.2 2104.01111 tab. 2. GPS-Net lists its IMP row, marked as using the Neural Motifs detector, with 61.3, 35.4 and 24.5 2003.12962 tab. 1. Neither card says which protocol the other paper used, so the two rows are not interchangeable.
• Recall and mean recall can rank methods differently. A method can have high R@K and low mR@K, because frequent predicates dominate R@K 2104.01111 §6.2. GPS-Net's "Mean" column averages recall values, so it is not a mean-recall measure 2003.12962 tab. 1.
• Graph constraint changes R@K. Some papers allow one predicate per object pair, others allow several 2104.01111 §6.2. The two sets of numbers are not comparable.
• Interaction-graph and D3GD numbers are not comparable to VG150. The GF-VLA graph scores (GRA 98.5%, 97.2% and 96.8%) are on the authors' own videos and annotations, and no VG150 predicate metric is reported for them 2508.05342 §6.1 2104.01111 tab. 2. The D3GD AP_rel values are on MetaGraspNetV2 bin scenes with dependency labels 2409.02035 §3 2409.02035 §5.
• The Docked rule is stated two ways. The text gives both "Docked if MI is falling and below a threshold, or already Docked" and "Docked may only follow Coupled-Motion" 2508.05342 §3.2.1. No statement of which reading the reported results used was found in the card.
• Dataset size disagrees within the survey. The text says VrR-VG keeps the top 500 relationships, while its dataset table lists 117 relationship categories 2104.01111 §6.1 2104.01111 tab. 1. Check the original release before relying on either number.
• Loss scale: paper and code differ. Eq. 9 sums the BCE over all P x P pairs. The released code takes a mean and rescales it by the number of matched nodes, so its magnitude differs from the equation 2409.02035 eq. 9 2409.02035 §4.3.
• Softmax axis: paper and code differ. The paper says column-wise softmax. The code takes softmax over the first node index 2409.02035 §4.2.
• One Easy number, two values. The D3G (DETR) synthetic-easy AP_rel is 76.27 in the main table and 76.77 in the design-choice table for the same configuration 2409.02035 tab. 3 2409.02035 tab. 5. Treat it as uncertain.

Papers

• GPS-Net: Graph Property Sensing Network for Scene Graph Generation: a one-stage generator with direction-aware message passing, a node-priority loss and a gated frequency prior; reports mean-recall gains on VG and OpenImages (author-reported).
• A Comprehensive Survey of Scene Graphs: Generation and Application: a survey that groups SGG methods into families and add-ons, reviews datasets and metrics, and tabulates author-reported VG150 results. It has no method or experiments of its own.
• A Modern Take on Visual Relationship Reasoning for Grasp Planning: a DETR-style detector with a dense graph transformer over object queries, which predicts dependencies for bin picking; it also introduces the D3GD benchmark. Results are author-reported.
• Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control: builds hand-object and object-object graphs from RGB-D human demonstrations with entropy and mutual information, and feeds them to a VLA planner for a dual-arm robot. Its results are on its own tasks.