A Comprehensive Survey of Scene Graphs: Generation and Application

In plain words

A scene graph is a simple map of a picture: it lists the things in the picture and how they relate, for example "girl - swinging - racket" or "cone - in front of - girl". Once a picture is turned into such a map, a program can reason about it rather than only look at pixels.

This paper is a survey. It does not propose a new model. It sorts the many ways researchers have tried to build these maps from images into families, explains how each family works, and lists the data sets and scoring rules people use. It also covers what scene graphs are good for, such as answering questions about images, retrieving images from a description, and understanding 3D rooms.

The hard part the survey keeps coming back to is that some relations are very common ("on", "has") while many useful ones are rare. A model that simply learns the common ones looks good on average but misses the interesting cases.

TL;DR

Read this card if you want an overview of scene graph generation and a table of reported results to compare against. It is a literature review, so it gives you no algorithm to reimplement and no hyperparameters. Its numbers are copied from the original papers under differing setups, so use them to orient, not to rank methods. The strongest recurring advice is to measure the mean recall per predicate, because plain recall is dominated by frequent relations §6.2.

Problem

Object detection says what and where things are. Scene graph generation (SGG) also asks how the detected objects relate. The output for an image is a graph: objects are nodes with a category and attributes, and each relation is a directed edge that becomes a subject-predicate-object triplet §1.1 eq. 1. The survey treats the relation step as the key challenge: most work focuses on inferring relations once objects are found §1.2.

Method

The survey has no method of its own. Its taxonomy has two axes.

Pipeline view §1.2 eq. 7 eq. 8. The survey describes two ways to structure the work:

• Bottom-up (two stages): first detect boxes, then objects, then predicates for the detected pairs. The probability factorises into a box term, an object term given the boxes, and a relation term given boxes and objects eq. 7.
• Top-down (joint): detect boxes, then infer objects and relations jointly from the box proposals eq. 8.

It also breaks a general pipeline into links from the image to object features, object classes, and pairwise features, with a cross-entropy loss on object and predicate labels as the default training objective §1.2. It groups the work into feature extraction, contextualisation, and graph construction and inference §1.2.

Model families §2. Each family is described in prose with the named models it covers. No family is ranked by the survey itself.

• CRF-based §2.1 fig. 4: object detection and relation prediction as a conditional random field.
• TransE-based §2.2 fig. 6 fig. 7: relations treated as translations between object embeddings.
• CNN-based §2.3 fig. 8 fig. 9 fig. 10 fig. 11: convolutional features over object pairs, including context and zoom modules.
• RNN/LSTM-based §2.4 fig. 12 fig. 13: sequential context over objects, including chain, fully connected, subgraph and tree orderings.
• GNN-based §2.5 fig. 14: message passing over the object and relation graph.
• Discussion §2.6: nearly all recent methods model objects and relations jointly through context rather than independently.

Prior knowledge §3 fig. 16. Language priors §3.1, statistical priors from annotation co-occurrence §3.2, and knowledge graphs §3.3. The survey suggests these help rare relations most.

Long-tailed predicates §4 fig. 17. Predicate frequencies are heavily skewed, and the number of possible object-predicate combinations makes rare combinations unavoidable. Approaches named include zero-, one- and few-shot learning, causal debiasing, and an energy-based loss for joint object and relation likelihood §4. The energy-based loss's objective is written as a log-likelihood sum over objects and relations §4.

Applications §5: visual-textual transformation such as image generation and captioning §5.1; cross-modal retrieval §5.2; visual question answering §5.3; image understanding and reasoning §5.4; 3D scene graphs §5.5; and human-object and human-human interaction §5.6. fig. 18 shows example scenarios.

# Bottom-up scene graph generation  [§1.2] [eq. 7]
# Input:  image I
# Output: scene graph SG = (objects, relations, edges)

def generate_scene_graph(image):
    # 1. Detect candidate boxes and their features  [§1.2]
    boxes, features = detector(image)              # features (m, d)

    # 2. Classify each detected object  [§1.2] [eq. 7]
    labels = object_classifier(features)           # per box category

    # 3. Predict a predicate for each ordered pair of objects  [§1.2]
    for each pair (i, j) with i != j:
        predicates[i, j] = predicate_model(
            features[i], features[j], labels[i], labels[j])

    # 4. Keep the pairs with a non-background relation  [§1.1]
    return build_graph(boxes, labels, predicates)

Datasets

The survey reviews 14 datasets: 10 static-image, 2 video and 2 3D §6.1. tab. 1 lists their size, object and relation counts, and predicate counts. Prose descriptions are in §6.1. The commonly used benchmark for relation detection is VG150, a filtered subset of Visual Genome with 150 object categories §6.1 tab. 1.

Evaluation

§6.2 fig. 19. Three tasks: predicate detection (boxes and labels given), phrase detection (triplets with a localised union box), and relationship detection (full graph from pixels). Under the diagnosis protocol the survey describes, relation retrieval has three subtasks, predicate classification (PredCls), scene graph classification (SGCls, boxes given but no labels) and scene graph detection (SGDet, nothing given). Zero-shot retrieval and sentence-to-graph retrieval are also defined §6.2.

Recall at K (R@K) is the usual metric. Because it is biased toward frequent predicates, mean recall at K (mR@K) averages recall over predicates and is usually lower §6.2. Whether one predicate or several are allowed per object pair changes R@K, so check the constraint before comparing numbers §6.2.

Hyperparameters

Not applicable: this is a survey with no training procedure of its own §1. Hyperparameters appear only inside the cited works, which this card does not reproduce.

Results

Author-reported, not re-run by the survey. Setup: VG150 with graph constraints, the protocol in tab. 2. Values are as printed in the table; the table does not state units. A subset of rows, chosen from tab. 2 and each row from the cited original paper:

Source for all rows: tab. 2. The survey notes that the energy-based loss [suhail2021energy] gives the best mR@K among the methods it investigated §6.2.

Paper vs code

No official code. The paper links no repository or project page §1 §6. The arXiv record exposes no comment field with a code link. A GitHub repository search for the title returned one result, under the organisation SZU-AdvTech-2022, which is not an author or an author's lab, so it was not used. No code-based checks were made.

Pitfalls

• Numbers come from different papers. Each row of tab. 2 reflects its own backbone, training split, and evaluation details. Rows marked with an asterisk are evaluated on other datasets; rows marked with a dagger are re-implementations from another paper. Do not treat the table as a controlled ranking.
• Mean recall and recall answer different questions. R@K can look high while mR@K is low, because frequent predicates dominate. The survey itself says mR@K is the fairer measure on long-tailed data §6.2.
• Graph constraint changes R@K. Some papers allow one predicate per object pair, others allow several. Compare only rows evaluated under the same constraint §6.2.
• Text and table disagree on one dataset. The text says VrR-VG keeps the top 500 relationships §6.1; tab. 1 lists 117 relationship categories for it. Check the original dataset release before relying on either number.
• Broken equation reference. The energy-based loss section refers to a label that is not defined in the source, so the referenced equation is not reproduced §4. The equation shown there is the one labelled as the energy-based objective.
• Inconsistent metric name. Sentence-to-graph retrieval is abbreviated S2GR in one place and S2RR in the next §6.2.
• Date of coverage. The arXiv version used here is v5 (updated 2022-01-07). Its references stop around 2021, so later methods are not covered.
• Third-party figure. The long-tail figure is credited in its caption to a different paper fig. 17. Do not reuse it; this wiki uses no images anyway.

Lineage

• Builds on: scene graphs as a data structure were introduced by Johnson et al. §1, named in the paper but not summarised here, since no card exists for that work.
• Compared against: the survey's own tab. 2 compares the surveyed methods; it is a comparison of published numbers, not a benchmark it ran.
• Superseded by: not known from content in this wiki.