Paper · 2022-07-22 · cs.CV, cs.AI, cs.CL, cs.LG, cs.MM
Panoptic Scene Graph Generation
Predicts scene-graph relations between panoptic segmentation masks instead of bounding boxes, with a new 49k-image dataset built from COCO and Visual Genome (133 object classes, 56 predicates). Benchmarks two-stage baselines and two DETR-style one-stage models, PSGTR and PSGFormer. Author-reported best SGDet R@100 is 36.3 (PSGTR, 60 epochs, ResNet-50).
Scene graph generationPanoptic segmentationBipartite matching for set-prediction lossesObject queries (learned query slots in transformer decoders)scene-graphpanoptic-segmentationdetrtransformerdatasetbenchmark
arXivPDFCodeMarkdown16 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
A scene graph describes a picture as a set of things and the relations between them, such as "person holding phone". Most existing work describes each thing with a rectangle around it. A rectangle is a rough fit: it catches pixels that belong to something else, misses the background, and the labels that go with it are often redundant, such as "hair" as a thing a person has.
This paper proposes to describe things with exact outlines instead: every pixel is assigned to one thing or background region, and relations are predicted between those regions. It also builds a dataset for this task by merging two existing image collections, cleaning the result by hand, and fixing a short list of relation words.
The paper then tests several ways to predict these graphs. Some first find the objects and then predict relations between them. Others predict whole graphs in one pass, using a transformer that proposes a fixed number of candidate relations at once. The authors report that the one-pass models trail the two-stage ones on the main recall score, that PSGFormer is ahead of the two-stage models on mean recall per relation at a short training budget, and that PSGTR needs much longer training to catch up.
TL;DR
- Replaces bounding-box grounding of scene-graph nodes with panoptic masks, so every node is a non-overlapping region that includes background ("stuff") §3.
- Releases a dataset of 48,749 COCO and Visual Genome images with 133 object classes and 56 predicates, built by automatic merging and then hand correction §3 tab. 1 app. A.2.
- Benchmarks four two-stage baselines (Panoptic FPN features with IMP, MOTIFS, VCTree or GPSNet relation heads) and two DETR-style one-stage models, PSGTR and PSGFormer §4 tab. 2.
- Author-reported: one-stage models need 60 epochs to reach the best recall; PSGFormer has the best mean recall at 12 epochs §5.1.
- The code is MIT-licensed; several hyperparameters and loss terms differ from the paper, see Paper vs code.
Problem
Scene graph generation (SGG) is usually defined with bounding boxes for objects, class labels, and a set of relations §3 eq. 1. The authors argue this has four practical problems: boxes include pixels of other objects, miss the background that gives context, invite redundant or trivial relations, and label the same object several times §1.
PSG keeps the same structure but grounds each object with a mask from a panoptic segmentation, so masks do not overlap §3 eq. 2. Evaluation then requires the mask of each predicted subject and object to overlap the ground truth mask with IoU above 0.5, in addition to the correct triplet labels §3.
The paper also notes a cost of the mask-based setting: PSGTR's object masks are poor for panoptic segmentation, because the same physical object is often segmented by several triplets, and merging them needs a re-identification step that the paper does not solve §6 app. C.
Method
PSG dataset §3 tab. 1 app. A
Construction has three steps §3:
- Coarse fusion. The authors start from the 48,749 images that appear in both COCO and Visual Genome. COCO masks are matched to Visual Genome boxes by category and by overlap, and Visual Genome relations are transferred to the matched COCO masks app. A.3. Category similarity uses fastText word embeddings and cosine similarity; instances are matched greedily in order of box overlap, and a pair is accepted when the category similarity is above a threshold the paper does not state app. A.3.
- Predicate dictionary. 56 predicates were chosen from the noisy predicates and from VG-150, VrR-VG and GQA. The selection keeps predicates that are representative, practical and well covered, drops one direction of opposite pairs (for example keeps "in-front-of" and not "behind"), and drops most positional predicates from GQA §3 app. A.4.
- Annotation. Annotators removed wrong triplets and added missing relations for thing-thing, thing-background and background-background pairs. They were told to use the most specific predicate that fits and not a general one such as "on" §3 app. A.3.
Quality control: the main authors annotated 1,000 images to write a guide; a contracted company annotated the training set; two workers labelled each image and a head worker checked it. The test images were annotated by the authors §3. The paper reports US$11K spent on training annotation §3.
The 133 object classes are COCO's 80 thing and 53 stuff classes §1 app. A.2. Averages per image: 11.0 instances, 5.6 relations, of which 34% are thing-thing, 21% stuff-stuff and 45% thing-stuff app. A.2.
tab. 1 compares the new dataset with the classic VG-based sets. Only GQA and PSG are duplicate-free, and PSG is the only set grounded by segmentation.
| Dataset | Images | Object classes | Relation classes | Predicates per image | Duplicate-free | Grounding | Source |
|---|---|---|---|---|---|---|---|
| VG tab. 1 | 108K | 34K | 40K | 21.4 | no | bounding box | COCO and Flickr |
| VG-150 tab. 1 | 88K | 150 | 50 | 5.7 | no | bounding box | clean VG |
| VrR-VG tab. 1 | 59K | 1,600 | 117 | 3.4 | no | bounding box | clean VG |
| GQA tab. 1 | 85K | 1,703 | 310 | 50.6 | yes | bounding box | re-annotated VG |
| PSG tab. 1 | 49K | 133 | 56 | 5.7 | yes | segmentation | annotated COCO |
Evaluation §3
- PredCls (predicate classification): ground-truth objects and masks are given; only two-stage models can use it.
- SGDet (scene graph generation, main result): the model starts from the image alone.
- SGCls is not applicable: panoptic models cannot classify ground-truth masks directly.
- Metrics are R@K (triplet recall) and mR@K (mean recall over predicate classes) for K = 20, 50 and 100.
- PQ (panoptic quality) is used for diagnosis only §3.
Two-stage baselines §4.1 eq. 3 fig. 3
The image is first processed by a Panoptic FPN, fine-tuned on the PSG annotations app. B.1. Each detected thing or stuff region gives a mask, a class and a feature. Region features are pooled with RoIAlign over the tightest box around the mask, and union features are formed for object pairs app. B.1 fig. 3. A relation head from IMP, MOTIFS, VCTree or GPSNet then predicts the predicates §4.1. The joint probability is factorised into masks, then objects given masks, then relations given objects and masks eq. 3.
PSGTR §4.2 eq. 4 eq. 5 eq. 6 fig. 4
PSGTR adapts DETR. A CNN and a transformer encoder-decoder produce a fixed set of 100 triplet queries app. B.2 §4.2. Each query gives a subject class, a predicate class and an object class through three FFNs, and two panoptic heads give subject and object masks §4.2 fig. 4.
Training uses a triplet Hungarian matcher §4.2 eq. 4 eq. 5. The matching cost for a query and a ground-truth triplet adds the mask cost of the subject and object with the class costs of the subject, predicate and object. The objective selects the assignment of queries to ground-truth triplets that maximises the total cost eq. 5. The loss on the matched pairs is cross-entropy for the three classes and a DICE (F-1) loss for the two masks eq. 6.
PSGFormer §4.3 eq. 7 eq. 8 fig. 5
PSGFormer keeps the DETR-style encoder and separates objects from relations:
- The encoder is shared. Two decoders run on the same features: one for 100 object queries and one for 100 relation queries fig. 5 app. B.3 psgformer_r50.py.
- Each object query gives a class, a box and a mask; the mask head follows DETR's segmentation head §4.3 app. B.3.
- Two extra FFNs, f_S and f_O, turn each object query into a subject candidate and an object candidate §4.3.
- Each relation query selects its subject and object by the highest cosine similarity to these candidates §4.3 eq. 7. The i-th subject, relation and object form one triplet eq. 8.
- A standard DETR-style panoptic segmentation loss on the object decoder strengthens it and avoids duplicate object groundings §4.3 app. B.3.
The paper describes the selection as a prompt-like matching, in which the relation query acts as a prompt and the candidates act as fill-in words, and also as cross-attention whose keys are the candidates §4.3.
Pseudocode
Inference, PSGFormer triplet assembly §4.3 eq. 7 eq. 8:
# PSGFormer triplet assembly [§4.3] [eq. 7] [eq. 8]
# object_queries: (N_obj, d) with N_obj = 100 [app. B.3]
# relation_queries: (N_rel, d) with N_rel = 100 [app. B.3]
def assemble_triplets(object_queries, relation_queries):
# Subject and object candidates come from two separate FFNs [§4.3]
subject_candidates = f_S(object_queries) # (N_obj, d)
object_candidates = f_O(object_queries) # (N_obj, d)
# Cosine similarity between each relation query and each candidate;
# the best-matching candidate is that relation's subject [eq. 7]
subject_ids = argmax_cosine(relation_queries, subject_candidates)
# The same rule selects the object [eq. 7]
object_ids = argmax_cosine(relation_queries, object_candidates)
# Query i gives the i-th triplet (subject, relation, object) [eq. 8]
triplets = []
for i in range(N_rel):
triplet = (subject_ids[i], relation_queries[i], object_ids[i])
triplets.append(triplet)
return tripletsTraining, paper form of the triplet loss §4.2 eq. 4 eq. 5 eq. 6:
# Triplet matching and loss, paper form [§4.2] [eq. 4] [eq. 5] [eq. 6]
# queries: T triplet predictions; gt_triplets: ground-truth triplets
def triplet_loss(queries, gt_triplets):
# Hungarian matching; the cost sums mask and class terms for the
# subject, predicate and object [§4.2] [eq. 4] [eq. 5]
matches = hungarian_match(queries, gt_triplets)
loss = 0.0
for query_index, gt_index in matches:
# DICE (F-1) loss on the subject and object masks [§4.2] [eq. 6]
for role in ["subject", "object"]:
loss += dice_loss(
queries[query_index].mask[role],
gt_triplets[gt_index].mask[role],
)
# Cross-entropy on the subject, predicate and object classes
# [§4.2] [eq. 6]
for role in ["subject", "predicate", "object"]:
loss += cross_entropy(
queries[query_index].label[role],
gt_triplets[gt_index].label[role],
)
return lossThe paper does not say what the target of an unmatched query is; see Pitfalls.
Hyperparameters
Values are from the appendix and the code. "Not stated" means the paper does not
give the value. Code values are from the code_commit above.
| Name | Value | Where | Notes |
|---|---|---|---|
| Two-stage: Panoptic FPN fine-tune | |||
| optimizer | SGD, momentum 0.9, weight decay 1e-4 | app. B.1 [code: configs/base/models/…fpn_psg.py#L44] | |
| learning rate | 0.02 | app. B.1 [code: configs/base/models/…fpn_psg.py#L44] | |
| batch size | 8 | app. B.1 [code: configs/base/models/…fpn_psg.py#L40] | per GPU in code; total GPU count not stated in paper |
| epochs | 12 | app. B.1 | |
| warm-up | linear, 500 iterations, from lr/3 | app. B.1 [code: configs/base/models/…fpn_psg.py#L49-L51] | |
| decay | ×0.1 at epochs 8 and 11 | app. B.1 [code: configs/base/models/…fpn_psg.py#L52] | |
| gradient clip | max L2 norm 35 | app. B.1 [code: configs/base/models/…fpn_psg.py#L46] | |
| Two-stage: relation head (MOTIFS, VCTree, GPSNet) | |||
| optimizer | SGD, momentum 0.9, weight decay 1e-4 | app. B.1 | |
| learning rate | 0.03 | app. B.1 …predcls_psg.py | IMP config uses 0.001, see Paper vs code |
| batch size | 16 | app. B.1 …predcls_psg.py | |
| epochs, warm-up, decay | 12; 500 iterations from lr/3; ×0.1 at epochs 7 and 10 | app. B.1 …predcls_psg.py | |
| gradient clip | max L2 norm 35 | app. B.1 …predcls_psg.py | |
| other relation-head settings | as in each original paper | app. B.1 | the paper gives no values |
| PSGTR and PSGFormer (shared settings) | |||
| backbone | ResNet-50 or ResNet-101 | tab. 2 | |
| queries | 100 triplet queries (PSGTR); 100 object and 100 relation queries (PSGFormer) | app. B.2 app. B.3 psgformer_r50_psg.py | |
| FFN | 3-layer MLP | app. B.2 app. B.3 | |
| transformer depth | not stated in the paper; panoptic head is a 6-layer FPN-like CNN app. B.3 | psgformer_r50.py psgformer_r50.py psgformer_r50.py | code: 6 encoder layers, 6 layers in each decoder |
| init | COCO-pretrained DETR | app. B.2 | code loads a file named detr_pan_r50.pth (PSGTR) or detr4psgformer_r50.pth (PSGFormer); files are not in the repo |
| batch size | 1 per GPU, 8 V100 GPUs | app. B.2 app. B.3 | |
| training time | about 2 days (PSGTR), 2.5 days (PSGFormer) | app. B.2 app. B.3 | hardware as stated |
| epochs | 60; learning rate step at epoch 40 | app. B.2 app. B.3 psgtr_r50_psg.py psgformer_r50_psg.py | 12-epoch rows in tab. 2 are not in the repo |
| optimizer | AdamW, weight decay 1e-4 | app. B.2 app. B.3 psgtr_r50_psg.py | |
| learning rate, PSGTR | 1e-4 (backbone 1e-5) | app. B.2 psgtr_r50_psg.py | |
| learning rate, PSGFormer | "same as PSGTR" | app. B.3 | code uses 1e-3 with multipliers, see Paper vs code |
| gradient clip | not stated | app. B.2 app. B.3 | code uses max norm 0.1 psgtr_r50_psg.py psgformer_r50_psg.py |
| input size | shortest side 480 to 800 (multi-scale), longest side at most 1333 | app. B.2 psgtr_r50_psg.py | |
| extra augmentation | not stated | app. B.2 | code adds a resize to 400–600, a crop of 384×600, then a resize psgtr_r50_psg.py |
| auxiliary panoptic loss | ratio 5:1 (triplet task to panoptic task) | app. B.3 | not located in the code read |
| matcher cost | not stated | code: class 4, L1 3, GIoU 2 for objects psgformer_r50.py | |
| Evaluation | |||
| mask IoU threshold | 0.5 | §3 psgtr_r50_psg.py | |
| K | 20, 50, 100 | §3 tab. 2 |
Results
Author-reported. Setup: PSG dataset, MMDetection-based code, COCO-pretrained DETR initialisation for one-stage models. No variance or number of runs is reported §5.1 tab. 2. Rows marked † are trained for 60 epochs; other one-stage rows are trained for 12 epochs by default tab. 2.
| Backbone | Method | PredCls R/mR@20 | PredCls R/mR@50 | PredCls R/mR@100 | SGDet R/mR@20 | SGDet R/mR@50 | SGDet R/mR@100 |
|---|---|---|---|---|---|---|---|
| ResNet-50 | IMP | 31.9 / 9.55 | 36.8 / 10.9 | 38.9 / 11.6 | 16.5 / 6.52 | 18.2 / 7.05 | 18.6 / 7.23 |
| ResNet-50 | MOTIFS | 44.9 / 20.2 | 50.4 / 22.1 | 52.4 / 22.9 | 20.0 / 9.10 | 21.7 / 9.57 | 22.0 / 9.69 |
| ResNet-50 | VCTree | 45.3 / 20.5 | 50.8 / 22.6 | 52.7 / 23.3 | 20.6 / 9.70 | 22.1 / 10.2 | 22.5 / 10.2 |
| ResNet-50 | GPSNet | 31.5 / 13.2 | 39.9 / 16.4 | 44.7 / 18.3 | 17.8 / 7.03 | 19.6 / 7.49 | 20.1 / 7.67 |
| ResNet-50 | PSGTR | – | – | – | 3.82 / 1.29 | 4.16 / 1.54 | 4.27 / 1.57 |
| ResNet-50 | PSGFormer | – | – | – | 16.8 / 14.5 | 19.2 / 17.4 | 20.2 / 18.7 |
| ResNet-50 | PSGTR† | – | – | – | 28.4 / 16.6 | 34.4 / 20.8 | 36.3 / 22.1 |
| ResNet-50 | PSGFormer† | – | – | – | 18.0 / 14.8 | 19.6 / 17.0 | 20.1 / 17.6 |
| ResNet-101 | IMP | 30.5 / 8.97 | 35.9 / 10.5 | 38.3 / 11.3 | 17.9 / 7.35 | 19.5 / 7.88 | 20.1 / 8.02 |
| ResNet-101 | MOTIFS | 45.1 / 19.9 | 50.5 / 21.5 | 52.5 / 22.2 | 20.9 / 9.60 | 22.5 / 10.1 | 23.1 / 10.3 |
| ResNet-101 | VCTree | 45.9 / 21.4 | 51.2 / 23.1 | 53.1 / 23.8 | 21.7 / 9.68 | 23.3 / 10.2 | 23.7 / 10.3 |
| ResNet-101 | GPSNet | 38.8 / 17.1 | 46.6 / 20.2 | 50.0 / 21.3 | 18.4 / 6.52 | 20.0 / 6.97 | 20.6 / 7.17 |
| ResNet-101 | PSGTR | – | – | – | 3.47 / 1.18 | 3.88 / 1.56 | 4.00 / 1.64 |
| ResNet-101 | PSGFormer | – | – | – | 18.0 / 14.2 | 20.1 / 18.3 | 21.0 / 19.8 |
| ResNet-101 | PSGTR† | – | – | – | 28.2 / 15.4 | 32.1 / 20.3 | 35.3 / 21.5 |
| ResNet-101 | PSGFormer† | – | – | – | 18.6 / 16.7 | 20.4 / 19.3 | 20.7 / 19.7 |
tab. 2. "–" means the metric does not apply to one-stage models.
Reading the table as the paper does §5.1:
- Two-stage results depend on the first stage. IMP's R@20 is about 32% when the ground-truth segmentation is given (PredCls) and about 17% in SGDet with ResNet-50 §5.1 tab. 2.
- At 12 epochs, PSGFormer exceeds the best two-stage model, VCTree, by 4.8 points on mR@20 and by 8.5 points on mR@100 (ResNet-50) §5.1 tab. 2. PSGFormer does not beat the two-stage models on R@K, which the paper reads as a less biased model §5.1.
- PSGTR is very weak at 12 epochs but gives the best SGDet R@K at 60 epochs, with R@100 of 36.3 for ResNet-50 §5.1 tab. 2.
- Panoptic quality is low for PSGTR even when its PSG scores are high, because triplet queries produce object masks independently §6 app. C fig. 6.
Figure 6 shows segmentation output and the top 20 triplets for five methods fig. 6.
Paper vs code
Official repository: OpenPSG, MIT License, copyright held by the first author,
at commit 34b2a892f7441966265e3d60ad01ee8eeae89041. The paper links it in
its experiments section §5. Where a config file name is long, the middle of
the path is abbreviated with …; the full names are in the repository at that
commit. Discrepancies:
- Loss terms. The paper's total loss has cross-entropy and DICE terms only, without weights §4.2 eq. 6. PSGTR's code also uses an L1 box loss (weight 5), a GIoU loss (weight 2) and a BCE focal loss (weight 2) for each of the subject and object, and a relation cross-entropy with weight 2 [code: configs/psgtr/psgtr_r50.py#L52-L70]. PSGFormer's config uses object cross-entropy 4, L1 3, GIoU 2, focal 1, DICE 1, relation cross-entropy 2, and subject and object identity losses 2 each [code: configs/psgformer/psgformer_r50.py#L68-L79].
- Matcher. The paper's matching cost has mask and class terms only §4.2 eq. 4. The code's PSGTR matcher adds box L1 (weight 5) and GIoU (weight 2) terms psgtr_r50.py. PSGFormer's object matcher uses the same weights as its object loss (class 4, L1 3, GIoU 2; the head asserts this) plus an identity matcher for subjects and objects [code: configs/psgformer/psgformer_r50.py#L81-L96] [code: openpsg/models/relation_heads/psgformer_head.py#L125-L132].
- Subject and object selection. The paper gives cosine similarity and an argmax §4.3 eq. 7. The code divides the similarity by a temperature of 0.1 psgformer_head.py psgformer_head.py. The argmax is unchanged by this, but the identity losses use the scaled scores.
- Background weight. The code gives the no-object class a weight of 0.02 in the object and relation classification losses [code: openpsg/models/relation_heads/psgformer_head.py#L45] psgformer_head.py. The paper says only that it follows DETR's training recipe app. B.3.
- Learning rate for PSGFormer. The paper says PSGFormer uses the PSGTR learning rate, 1e-4 with 1e-5 for the backbone app. B.3. The code sets the base rate to 1e-3 and multiplies it by 0.1 only for listed modules: the backbone, the encoder, decoder1, object queries, input projection, class and box heads, bbox attention and mask head [code: configs/psgformer/psgformer_r50_psg.py#L203-L216]. Decoder2, the relation queries, the subject, object and relation update layers and the relation classifier keep 1e-3 psgformer_head.py.
- IMP learning rate. The IMP configs use SGD with learning rate 0.001 …predcls_psg.py. The other relation heads use 0.03 app. B.1. The paper's sentence on hyperparameters names only MOTIFS, VCTree and GPSNet.
- Augmentation. The code's training pipeline includes a branch that resizes to 400–600, takes a random crop of 384×600 (the config comment says it must leave relations, i.e. no empty crops), and resizes again [code: configs/psgtr/psgtr_r50_psg.py#L137-L150]. The paper describes DETR's augmentation and says cropped relations are filtered app. B.2.
- Frozen layers. The backbone freezes the first stage and uses frozen batch-norm statistics psgformer_r50.py. The paper does not mention this.
- Schedules for the 12-epoch rows. The shipped PSGTR and PSGFormer ResNet-50 configs train for 60 epochs psgtr_r50_psg.py psgformer_r50_psg.py. I found no 12-epoch config for either model, so the 12-epoch rows in tab. 2 cannot be reproduced from the repository as read.
- Post-processing. Masks are binarised at a sigmoid threshold of 0.85, and object masks that overlap with IoU above 0.5 are de-duplicated during inference psgformer_head.py psgformer_head.py. The paper does not describe this.
- Auxiliary panoptic ratio. The paper gives a 5:1 ratio between the triplet loss and the panoptic loss app. B.3. I did not find this ratio in the loss code read for this card.
- Predicate names. The paper's list uses "on the back of", "picking (grabbing)" and "cleaning (washing, brushing)". The config uses "on back of", "picking" and "cleaning" app. A.4 [code: configs/psgformer/psgformer_r50_psg.py#L47-L104]. The config list has 56 entries, as the paper states.
No discrepancies were found for: the 56 predicates (count), 100 queries per decoder, the PSGTR learning rate and backbone multiplier, the two-stage Panoptic FPN schedule, the two-stage relation-head batch size, learning rate, warm-up and steps, the input size range, and the mask IoU threshold of 0.5.
Pitfalls
- Dataset merge threshold. The category-similarity threshold used when matching COCO and Visual Genome objects is not stated app. A.3. Re-creating the merge needs this value from the authors.
- Dataset provenance. The annotation handbook is said to be on the project page, not in the paper app. A.4. The paper does not state a license for the dataset; this card covers the paper's text and code only.
- Unmatched queries in the loss. The paper's loss sums over every query but defines targets only for matched pairs §4.2 eq. 6. The code gives unmatched queries a no-object class with weight 0.02 [code: openpsg/models/relation_heads/psgformer_head.py#L45]. Use the code's rule if you need to reproduce results.
- Subject and object may coincide. Subject and object are chosen by separate argmax operations, and the code does not forbid the same object from being both psgformer_head.py. The paper does not discuss this case.
- Duplicate object masks. PSGTR produces object masks per triplet, so the same object can appear several times. The paper proposes a pixel-wise argmax merge and reports that this still splits some objects app. C §6. PQ stays low even when recall is high §6.
- Mean recall versus recall. PSGFormer is better on mR at 12 epochs and PSGTR is better on R at 60 epochs tab. 2. Choose the metric that matches your use; a model's ranking can change with the metric.
- Results are on the PSG dataset only. The table has no results on Visual Genome or other SGG test sets tab. 2.
- Training cost. One-stage training takes about 2 to 2.5 days on eight V100 GPUs at batch 1 per GPU app. B.2 app. B.3.
- Hidden learning-rate multipliers. Reproducing PSGFormer needs the code's per-module multipliers. A copy of the paper's stated rate will train the decoder2 and relation branches at a different rate than the released code psgformer_r50_psg.py.
- Weights not in the repo. Both one-stage models start from DETR weights whose files are referenced but not included [code: configs/psgtr/psgtr_r50_psg.py#L233] [code: configs/psgformer/psgformer_r50_psg.py#L244].
- Two-stage features. The relation heads are trained on features from a frozen Panoptic FPN app. B.1. The paper attributes IMP's drop from PredCls to SGDet to segmentation quality §5.1, so the two-stage numbers depend on the segmentation model.
Lineage
- Builds on: classic SGG relation heads IMP, MOTIFS, VCTree and GPSNet, which the paper adapts to masks §4.1. Panoptic FPN is the first stage §4.1. DETR is the basis of PSGTR and PSGFormer §4.2 §4.3.
- Compared against: the four two-stage relation heads tab. 2; PSGFormer and PSGTR are compared at 12 and 60 epochs tab. 2.
- Datasets used as sources: COCO and Visual Genome; VG-150, VrR-VG and GQA are used for predicate selection §3 tab. 1.
- Superseded by: not known from content already in this wiki.
Source and license
Adapted from “Panoptic Scene Graph Generation” by Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, Ziwei Liu (arXiv:2207.11247), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/Jingkang50/OpenPSG at commit 34b2a892f744 (MIT). Any quoted code is shown under that license.