--- kind: concept slug: scene-graph-generation title: Scene graph generation summary: >- Detect the objects in an image and predict a relation for each pair: a subject-predicate-object triplet, or a blocking dependency for bin picking. Image methods fall into CRF, TransE, CNN, RNN/LSTM, GNN, one-stage message-passing and query-graph families. Predicates are long-tailed, so compare mean recall per predicate. Robot papers use their own tasks and cannot be ranked against VG150. tags: [scene-graph, visual-relationship-detection, interaction-graph, grasp-planning] papers: ["2003.12962", "2104.01111", "2409.02035", "2508.05342"] related: [object-detection, long-tailed-recognition, graph-neural-network, graph-transformer, bipartite-matching, object-queries, conditional-random-field, focal-loss, vision-language-action-model, learning-from-demonstration] status: draft updated_at: 2026-11-01 --- ## Overview A scene graph is a graph built from an image. Each detected object is a node with a category, and each relation is a directed edge that becomes a subject-predicate-object triplet [arxiv:2104.01111 §1.1] [arxiv:2104.01111 eq. 1]. Scene graph generation (SGG) is the task of producing that graph from pixels. The invariant every image-based variant keeps: objects are found first, then a predicate is decided for each ordered pair of objects, and pairs without a real relation are dropped [arxiv:2104.01111 §1.2]. The variants differ in how object and relation decisions share information. Most recent methods model the two jointly through context rather than independently [arxiv:2104.01111 §2.6]. A second kind of variant does not classify predicates at all. It builds an interaction graph from tracked hand and object positions in human video, with edges set by thresholds on signal statistics [arxiv:2508.05342 §3.2]. See the last variant below. Relations are unevenly distributed. Common predicates dominate the annotations, and rare ones are many and hard to learn, so a model that only fits frequent predicates scores well on plain recall and poorly on the rare ones [arxiv:2104.01111 §4] [arxiv:2104.01111 §6.2]. ## Variants ### Bottom-up, two-stage pipeline Detect boxes, label the objects, then predict predicates for the detected pairs. The joint probability factorises into a box term, an object term given the boxes, and a relation term given boxes and objects [arxiv:2104.01111 §1.2] [arxiv:2104.01111 eq. 7]. Card: [A Comprehensive Survey of Scene Graphs: Generation and Application](/papers/2104.01111). ### Top-down, joint pipeline Detect boxes, then infer objects and relations together from the box proposals [arxiv:2104.01111 §1.2] [arxiv:2104.01111 eq. 8]. Card: [A Comprehensive Survey of Scene Graphs: Generation and Application](/papers/2104.01111). ### One-stage message passing with priority-aware loss (GPS-Net) Faster R-CNN proposals and their features form the nodes of a graph. Direction- aware message passing scores each ordered pair with a tri-linear model over the subject, object and union features, and sends each neighbour's feature along both directions of the pair [arxiv:2003.12962 §3.1] [arxiv:2003.12962 eq. 3-4]. A node-priority loss gives each object a focusing parameter from its share of the ground-truth triplets, capped at 2 [arxiv:2003.12962 §3.2] [arxiv:2003.12962 eq. 6-8]. The predicate head adds a softened class-pair frequency prior, gated per pair by the union feature [arxiv:2003.12962 §3.3] [arxiv:2003.12962 eq. 9-10]. The paper describes the detector and the graph as one network trained jointly [arxiv:2003.12962 §3]. Card: [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](/papers/2003.12962). Its own ablation on VG adds each part to a baseline; the message-passing part gives the largest single gain in SGCls R@100 [arxiv:2003.12962 tab. 5]. ### Dense graph over detector queries (DETR-style) Keep a DETR-style detector, then take its N object queries as graph nodes. A dense graph transformer updates node and edge features over all N x N pairs, and the edge read-out gives one relation logit per ordered pair [arxiv:2409.02035 §4.1] [arxiv:2409.02035 §4.2]. Training matches queries to ground-truth objects with Hungarian matching, then applies a binary cross-entropy loss to the relations of the matched queries only [arxiv:2409.02035 §4.3] [arxiv:2409.02035 eq. 5-9]. The target is dependency between objects in a bin: which one must come out first. Compared with object-pair pooling, which crops the feature map over the union of two boxes, this variant pairs query embeddings instead [arxiv:2409.02035 §1]. Card: [A Modern Take on Visual Relationship Reasoning for Grasp Planning](/papers/2409.02035). ### Model families The survey describes five families. It does not rank them. - **CRF-based**: detection and predicate prediction as a conditional random field [arxiv:2104.01111 §2.1]. See [conditional random field](/concepts/conditional-random-field). - **TransE-based**: each relation is treated as a translation between object embeddings [arxiv:2104.01111 §2.2]. - **CNN-based**: convolutional features over object pairs, with context and zoom modules [arxiv:2104.01111 §2.3]. - **RNN/LSTM-based**: sequential context over the objects, with chain, fully connected, subgraph and tree orderings [arxiv:2104.01111 §2.4]. - **GNN-based**: message passing over the object and relation graph [arxiv:2104.01111 §2.5]. See [graph neural network](/concepts/graph-neural-network). The survey does not cover the one-stage message-passing variant above, so it has no family label for it here. ### Add-on: prior knowledge Language priors, statistical priors from how annotated objects co-occur, and knowledge graphs. The survey suggests these help rare relations most [arxiv:2104.01111 §3] [arxiv:2104.01111 §3.1] [arxiv:2104.01111 §3.2] [arxiv:2104.01111 §3.3]. The frequency prior in the one-stage variant above is of the statistical kind: class-pair counts from the training annotations, softened and then gated per pair [arxiv:2003.12962 §3.3]. ### Add-on: long-tailed predicates Zero-, one- and few-shot learning, causal debiasing, and an energy-based loss over joint object and relation likelihood [arxiv:2104.01111 §4]. See [long-tailed recognition](/concepts/long-tailed-recognition). A per-node focusing term is a related idea: a focal-style loss whose focusing parameter depends on the node's role in the graph [arxiv:2003.12962 §3.2]. See [focal loss](/concepts/focal-loss). ### Interaction graph from human video (rule-based edges) Nodes are the two hands and the tracked objects. Each hand-object edge is labelled `Coupled-Motion` (the hand moves the object) or `Docked`, and each object-object edge is labelled `E-OO` (task-relevant) or `T-OO` (transitory). The labels come from distance gates and from the entropy and mutual information of the position signals, not from a learned predicate classifier [arxiv:2508.05342 §3.2.1] [arxiv:2508.05342 §3.2.2] [arxiv:2508.05342 eq. 1] [arxiv:2508.05342 eq. 3]. The edge thresholds are only partly stated: the distance gates are 0.15 m and 0.2 m, and the two MI thresholds are not [arxiv:2508.05342 §5.1]. The graph feeds a vision-language-action planner, see [Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control](/papers/2508.05342) and [vision-language-action model](/concepts/vision-language-action-model). Its graph accuracy is measured against the authors' own annotations on their own block-assembly video, not on a scene-graph benchmark [arxiv:2508.05342 §6.1] [arxiv:2508.05342 eq. 9]. ## When to use which The survey gives no single recommendation. The table follows the stated constraints and reported results of the cited papers. | Situation | Use | Why | |---|---|---| | Rare relations matter for your task, not only "on" or "has" | Add prior knowledge or a long-tail method, and report mR@K | Prior knowledge is said to help rare relations most [arxiv:2104.01111 §3]. Among the methods the survey investigated, the energy-based loss gives the best mR@K [arxiv:2104.01111 §6.2]. The survey does not include GPS-Net, so that ranking does not cover it. | | Single-stage training with the detector, and you want a predicate-level mean recall gain on VG150 | One-stage message passing with priority-aware loss (GPS-Net) | Its own table gives 22.8 PredCls mR@100 against 19.4 for VCTree-HL, with the same detector family [arxiv:2003.12962 tab. 2]. Author-reported, and its ablation covers only its own setup [arxiv:2003.12962 tab. 5]. The [GPS-Net card](/papers/2003.12962) records that the released code is partial. | | Few labelled examples of rare predicates | Zero-, one- or few-shot approaches | Named as the way to handle rare object-predicate combinations [arxiv:2104.01111 §4]. The survey reports no numbers for them. | | You want to test relation reasoning without detection errors | Predicate classification (PredCls) or scene graph classification (SGCls) | In SGCls the boxes are given and only labels are predicted; PredCls gives boxes and labels [arxiv:2104.01111 §6.2]. | | You need a system that starts from pixels | Scene graph detection (SGDet) | Nothing is given. In the survey's table, every method that gives both numbers has lower SGDet R@100 than SGCls R@100 [arxiv:2104.01111 tab. 2]. | | Input is RGB-D video of hands manipulating objects, and you need who touches what over time | Interaction graph with rule-based edges | It labels hand-object and object-object links per frame without a predicate classifier [arxiv:2508.05342 §3.2]. Its MI thresholds are not stated, and it was tested on one block-assembly setup [arxiv:2508.05342 §5.1] [arxiv:2508.05342 §6.1]. | | Dense clutter where dependencies set the picking order, and you can train a detector with query outputs | Dense graph over detector queries | Reports the best AP_rel on each synthetic split and on synth-to-real, against the object-pair baselines it compares with [arxiv:2409.02035 tab. 3]. Its synth-to-real OR and OP are unstable, since no validation set exists to tune the score threshold [arxiv:2409.02035 §6.3]. Memory grows with N squared per graph layer; the paper's efficiency claim is about compute [arxiv:2409.02035 §4.2]. | | Downstream use is captioning, retrieval, question answering or 3D rooms | Pick the family by your constraints, then check its evaluation setup | The survey lists these applications [arxiv:2104.01111 §5] but does not rank families for them. | | You need a reproducible implementation | Start from the original paper of your chosen family | The survey links no official code [arxiv:2104.01111 §1] [arxiv:2104.01111 §6]. Only GPS-Net and the two robot papers have cards here; the original works of the other families do not. | Reported numbers from the survey, author-reported on VG150 with graph constraints. Units are not stated in the table. Dashes mean no value in the source table [arxiv:2104.01111 tab. 2]: | Method (family or add-on) | PredCls mR@100 | SGCls R@100 | SGDet R@100 | |---|---|---|---| | IMP (RNN/LSTM) | 8.0 | 24.4 | 4.2 | | MotifNet (RNN/LSTM, statistical prior) | 15.3 | 36.5 | 30.3 | | VCTree-HL (RNN/LSTM) | 19.4 | 38.8 | 31.3 | | GB-Net (knowledge graph) | 20.9 | 38.8 | 30.0 | | ResCAGCN + PUM (statistical prior) | 22.0 | 39.0 | 31.3 | | VCTree-TDE, energy-based loss | 29.97 | - | - | | BGT-Net (RNN/LSTM) | - | 47.1 | 37.3 | | DG-PGNN (knowledge graph) | - | 40.8 | 33.1 | The energy-based loss row also lists SGCls mR@100 of 20.45 and SGDet mR@100 of 11.6 [arxiv:2104.01111 tab. 2]. GPS-Net's own VG table compares it with methods that use the Neural Motifs detector (marked ◇ there), and with RelDN's backbone (‡). Its "Mean" column is an average of recall values, not of mean recall; the same column in the GPS-Net table therefore does not measure mR. Values below are author-reported [arxiv:2003.12962 tab. 1] [arxiv:2003.12962 tab. 2]: | Method | Detector | PredCls R@100 | SGCls R@100 | SGDet R@100 | PredCls mR@100 | SGCls mR@100 | SGDet mR@100 | |---|---|---|---|---|---|---|---| | IMP ◇ | Motifs | 61.3 | 35.4 | 24.5 | 10.5 | 6.0 | 4.8 | | MOTIFS ◇ | Motifs | 67.1 | 36.5 | 30.3 | 15.3 | 8.2 | 6.6 | | KERN ◇ | Motifs | 67.6 | 37.4 | 29.8 | 19.2 | 10.0 | 7.3 | | VCTREE-HL ◇ | Motifs | 68.1 | 38.8 | 31.3 | 19.4 | 10.8 | 8.0 | | RelDN ‡ | RelDN | 68.4 | 36.8 | 32.7 | - | - | - | | GPS-Net ◇ | Motifs | 68.8 | 40.1 | 31.7 | 22.8 | 12.6 | 9.8 | | GPS-Net ‡ | RelDN | 69.7 | 42.3 | 33.2 | - | - | - | Other results in the same paper, also against RelDN with the same detector: on OpenImages the score_wtd is 47.03 against 44.61 [arxiv:2003.12962 tab. 3], and on VRD with a COCO backbone the relation R@50 is 27.8 against 25.3 [arxiv:2003.12962 tab. 4]. Its ablation on VG reports PredCls R@100 of 67.3 for the baseline and 68.8 for the full model, and SGCls R@100 of 36.3 and 40.1 [arxiv:2003.12962 tab. 5]. ## Implementation sketch The bottom-up pipeline is the common core. Each variant changes the boxes, the features or the predicate head. The sketch shows only that core. ```pseudocode # Bottom-up scene graph generation [arxiv:2104.01111 §1.2] [eq. 7] # Input: image I # Output: boxes, object labels, and a predicate for each kept pair def scene_graph(image): # 1. Propose boxes and per-box features [arxiv:2104.01111 §1.2] boxes, feats = detector(image) # feats: (n, d) # 2. Label each object [arxiv:2104.01111 §1.2] [eq. 7] labels = classify_objects(feats) # one class per box # 3. Score a predicate for every ordered pair [arxiv:2104.01111 §1.2] for i, j in ordered_pairs(boxes): # i != j rel[i, j] = predicate_head(feats[i], feats[j], labels[i], labels[j]) # 4. Keep pairs that carry a real relation [arxiv:2104.01111 §1.1] return boxes, labels, drop_background(rel) ``` Where variants differ: a top-down model replaces steps 2 and 3 with one joint step over the proposals [arxiv:2104.01111 eq. 8]. A dense query-graph model keeps the detector heads, feeds its N query embeddings into a graph transformer, and reads one relation logit per ordered pair [arxiv:2409.02035 §4.2]. A GNN or RNN variant replaces the independent pair scores with messages passed along the graph [arxiv:2104.01111 §2.4] [arxiv:2104.01111 §2.5]. The one-stage message-passing variant adds direction- aware messages before the pair head and a per-node focusing term to the object loss [arxiv:2003.12962 §3.1] [arxiv:2003.12962 §3.2]. Prior-knowledge and long-tail methods add terms to the predicate head [arxiv:2104.01111 §3] [arxiv:2104.01111 §4]. The interaction-graph variant has no predicate head: its edges come from threshold rules on hand and object signals [arxiv:2508.05342 §3.2.1]. ## Contradictions Numbers in the tables above come from different papers, not from one controlled experiment. Each row reflects its own backbone, training split and evaluation details, and the survey says its table is not a ranking [arxiv:2104.01111 §6.2] [arxiv:2104.01111 tab. 2]. Other disagreements matter when comparing papers: - **The same baseline has different numbers in two papers.** The survey lists IMP with PredCls R@100 53.0, SGCls R@100 24.4 and SGDet R@100 4.2 [arxiv:2104.01111 tab. 2]. GPS-Net lists its IMP row, marked as using the Neural Motifs detector, with 61.3, 35.4 and 24.5 [arxiv:2003.12962 tab. 1]. Neither card says which protocol the other paper used, so the two rows are not interchangeable. - **Recall and mean recall can rank methods differently.** A method can have high R@K and low mR@K, because frequent predicates dominate R@K [arxiv:2104.01111 §6.2]. GPS-Net's "Mean" column averages recall values, so it is not a mean-recall measure [arxiv:2003.12962 tab. 1]. - **Graph constraint changes R@K.** Some papers allow one predicate per object pair, others allow several [arxiv:2104.01111 §6.2]. The two sets of numbers are not comparable. - **Interaction-graph and D3GD numbers are not comparable to VG150.** The GF-VLA graph scores (GRA 98.5%, 97.2% and 96.8%) are on the authors' own videos and annotations, and no VG150 predicate metric is reported for them [arxiv:2508.05342 §6.1] [arxiv:2104.01111 tab. 2]. The D3GD AP_rel values are on MetaGraspNetV2 bin scenes with dependency labels [arxiv:2409.02035 §3] [arxiv:2409.02035 §5]. - **The Docked rule is stated two ways.** The text gives both "Docked if MI is falling and below a threshold, or already Docked" and "Docked may only follow Coupled-Motion" [arxiv:2508.05342 §3.2.1]. No statement of which reading the reported results used was found in the card. - **Dataset size disagrees within the survey.** The text says VrR-VG keeps the top 500 relationships, while its dataset table lists 117 relationship categories [arxiv:2104.01111 §6.1] [arxiv:2104.01111 tab. 1]. Check the original release before relying on either number. - **Loss scale: paper and code differ.** Eq. 9 sums the BCE over all P x P pairs. The released code takes a mean and rescales it by the number of matched nodes, so its magnitude differs from the equation [arxiv:2409.02035 eq. 9] [arxiv:2409.02035 §4.3]. - **Softmax axis: paper and code differ.** The paper says column-wise softmax. The code takes softmax over the first node index [arxiv:2409.02035 §4.2]. - **One Easy number, two values.** The D3G (DETR) synthetic-easy AP_rel is 76.27 in the main table and 76.77 in the design-choice table for the same configuration [arxiv:2409.02035 tab. 3] [arxiv:2409.02035 tab. 5]. Treat it as uncertain. ## Papers - [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](/papers/2003.12962): a one-stage generator with direction-aware message passing, a node-priority loss and a gated frequency prior; reports mean-recall gains on VG and OpenImages (author-reported). - [A Comprehensive Survey of Scene Graphs: Generation and Application](/papers/2104.01111): a survey that groups SGG methods into families and add-ons, reviews datasets and metrics, and tabulates author-reported VG150 results. It has no method or experiments of its own. - [A Modern Take on Visual Relationship Reasoning for Grasp Planning](/papers/2409.02035): a DETR-style detector with a dense graph transformer over object queries, which predicts dependencies for bin picking; it also introduces the D3GD benchmark. Results are author-reported. - [Information-Theoretic Graph Fusion with Vision-Language-Action Model for Policy Reasoning and Dual Robotic Control](/papers/2508.05342): builds hand-object and object-object graphs from RGB-D human demonstrations with entropy and mutual information, and feeds them to a VLA planner for a dual-arm robot. Its results are on its own tasks.