GPS-Net: Graph Property Sensing Network for Scene Graph Generation

In plain words

A scene graph lists the things in a picture and how they relate, as triplets of subject, relation and object. Most earlier systems treat every object as equally important and read the relations as if they had no direction. They also learn from a label distribution where a few relations are very common and many are rare, so they tend to fall back on the common ones.

GPS-Net targets three properties of these graphs. First, direction: which object is the subject changes the meaning, so each object's context is built with that direction in mind. Second, importance: an object that takes part in many relations matters more for the graph, so the training loss gives it more weight. Third, rare labels: the frequency of each relation is used as a starting guess, but it is softened, and for each pair of objects the image decides how much of that guess to keep.

TL;DR

• One-stage scene graph generator: a Faster R-CNN detector feeds a graph with three additions, all trained jointly §3.
• Message passing uses a tri-linear attention that reads subject and object roles, plus a transformer-style refinement §3.1.
• A node-priority loss adapts the focal-loss focusing parameter per node §3.2.
• The frequency prior is softened and gated per pair before relation scoring §3.3.
• Author-reported state of the art on VG, OpenImages and VRD §4.3; code is partial and not a full training pipeline (see Paper vs code).

Problem

Scene graph generation has to find the objects and then classify a relation for each pair. Three properties of the graph are under-explored: edge direction, the different priority of nodes, and the long-tailed distribution of relations §1. Attention that ignores direction gives every node the same context §3.1. A frequency prior is useful, but on a long-tailed label set it fails on rare relations §1, §3.3. Two-stage methods that model priorities use losses that are not differentiable and need policy-gradient training §2, §3.2.

Method

The pipeline has three added parts. Object proposals and their features come from Faster R-CNN. For proposal i the paper concatenates a 2048-d appearance feature, the O-dimensional class scores and a 4-d spatial feature, and projects the result to a 512-d node feature x_i. The union box of a pair gives a 2048-d feature u_ij §3.

Direction-aware message passing (DMP). The paper first recalls a global context layer with a softmax over neighbours eq. 1 and a simplified variant eq. 2. Both lose node-specific context, because the concatenation lets the node feature be ignored. DMP instead scores each ordered pair with a tri-linear model over subject, object and union features eq. 3, normalises the scores, and sends each neighbour's projected feature along both directions of the pair eq. 4. A two-layer refinement with layer norm and a residual finishes the module eq. 5.

# Direction-aware message passing, one image  [§3.1] [eq. 3-5]
# Input:  nodes (N, 512) = x_i; union (P, 512) for candidate pairs
# Output: refined nodes (N, 512) = z_i

def dmp(nodes, pairs, union):
    # Tri-linear score for each candidate ordered pair  [eq. 3]
    # pairs holds (i, j, p): p indexes the union feature of i and j
    scores = zeros(N, N)
    for (i, j, p) in pairs:
        scores[i, j] = w_e @ (
            (W_s @ nodes[i]) * (W_o @ nodes[j]) * (W_u @ union[p])
        )
    # Normalise to alpha; the normalisation is not stated  [§3.1]
    alpha = normalise(scores)                   # (N, N), no self terms
    # Stack both directions and project each neighbour  [eq. 4]
    context = zeros(N, 512)
    for i in range(N):
        for j in range(N):
            if j == i:
                continue                        # neighbours exclude i  [§3.1]
            pair = [alpha[i, j], alpha[j, i]]   # (2,)
            context[i] += kron(pair, W_t3 @ nodes[j])   # (512,)
    # Refine with FC, layer norm, ReLU, FC, then add the residual  [eq. 5]
    refined = W_t1 @ relu(layer_norm(W_t2 @ context))   # (N, 512)
    return nodes + refined

Node priority sensitive loss (NPS). Each node gets a priority equal to the share of graph triplets that contain it eq. 6. That share feeds a mapping that gives the focusing parameter, capped at 2 eq. 7; the exponent mu controls how strongly priority matters. The per-node loss is a focal-style term with this per-node focusing parameter eq. 8. The paper says it is differentiable and convex in the probability, unlike the earlier non-differentiable loss §3.2.

# Node-priority-sensitive loss  [§3.2] [eq. 6-8]
# Input:  logits (N, O); gt_classes (N,); triplets (N,) = t_i; total = ||T||
# Output: scalar loss

def nps_loss(logits, gt_classes, triplets, total, mu=4):
    # Priority: share of the graph's triplets that contain node i  [eq. 6]
    theta = triplets / total                       # (N,)
    # Focusing parameter, capped at 2  [eq. 7]
    gamma = minimum(2, -((1 - theta) ** mu) * log(theta))   # (N,)
    # Softmax probability of the ground-truth class  [eq. 8]
    probs = softmax(logits)                        # (N, O)
    p = probs[arange(N), gt_classes]               # (N,)
    # Focal-style term with a per-node focusing parameter  [eq. 8]
    loss = -((1 - p) ** gamma) * log(p)            # (N,)
    # The reduction over nodes is not stated in the paper  [§3.2]
    return mean(loss)

Adaptive reasoning module (ARM). For relations, the frequency distribution of the subject and object classes is softened with a log-softmax eq. 9. A sigmoid gate computed from each pair's union feature rescales this prior per pair §3.3. The gated prior is added to the logits of a classifier that reads a fused feature of the two nodes and the union feature eq. 10. Testing takes the argmax over predicates eq. 11.

# Adaptive reasoning module for predicates  [§3.3] [eq. 9-10]
# Input:  subj_cls, obj_cls (ints); z_s, z_o (512,) node features
#         u (2048,) union feature of the pair
# Output: predicate probabilities (R,)

def arm_probs(subj_cls, obj_cls, z_s, z_o, u):
    # Frequency prior for the class pair, softened by log-softmax  [eq. 9]
    prior = log_softmax(freq[subj_cls, obj_cls])    # (R,)
    # Per-pair gate in (0, 1) from the union feature  [§3.3]
    gate = sigmoid(W_p @ u)                         # (R,), W_p is R x 2048
    # Fusion '*' is defined in §3.3: x * y = ReLU(Wx x + Wy y)
    #   - (Wx x - Wy y) * (Wx x - Wy y), with 1024-d projections
    fused = fuse(fuse(z_s, z_o), u)                 # (1024,)
    # Classifier logits plus the gated prior  [eq. 10]
    logits = W_r @ fused + gate * prior             # (R,), W_r is R x 1024
    return softmax(logits)

Inference. The predicted predicate for a pair is the argmax of the ARM probabilities eq. 11. In SGDET the paper scores only pairs whose boxes overlap §4.2.

Hyperparameters

Results

Author-reported. Setup: Faster R-CNN detector, VG150 with 50 predicates, three protocols SGDET, SGCLS and PREDCLS §4.1. Rows marked ◇ use the same detector and metric as Neural Motifs; ‡ uses the RelDN backbone tab. 1 caption. Mean is over R@50 and R@100 of the three protocols tab. 1 caption.

Visual Genome, recall tab. 1

Visual Genome, mean recall per predicate, mR@100 tab. 2

Appendix A reports mR@K for all three protocols: GPS-Net ◇ gets 6.9 / 8.7 / 9.8 (SGDET, R@20 / 50 / 100), 10.0 / 11.8 / 12.6 (SGCLS) and 17.4 / 21.3 / 22.8 (PREDCLS), mean 14.5 app. A. VCTREE-HL ◇ is 8.2 / 10.1 / 10.8 for SGCLS and 14.0 / 17.9 / 19.4 for PREDCLS app. A.

OpenImages tab. 3 (RelDN's metric; the same detector as RelDN)

Per-predicate AP_rel for GPS-Net, in the order at, on, holds, plays, interacts with, wears, hits, inside of, under: 35.10, 38.90, 51.47, 45.66, 44.58, 32.35, 71.71, 47.21, 57.28 tab. 3. The RelDN row for the same nine predicates is 32.90, 37.00, 43.09, 41.04, 44.16, 7.83, 51.04, 44.72, 50.00 tab. 3.

VRD, relation/phrase detection tab. 4

Appendix A also gives the k=70 recall for GPS-Net: relation R@50 / R@100 is 23.6 / 28.9 with the ImageNet backbone and 30.6 / 37.0 with COCO; phrase R@50 / R@100 is 30.4 / 38.2 and 36.8 / 44.5 app. A.

Ablations on VG tab. 5 (Exp 1 is the baseline built on the Neural Motifs no-context model with the paper's feature construction §4.4)

Ablations on the message-passing module, stacking, and mu tab. 6

The stacking rows have swapped-looking labels; see Pitfalls.

Paper vs code

The code is the lib/ folder (DMP, NPS loss, frequency softening) and gpsnet-detectron/ (a RelDN-based detector with the full relation head). The README says the full code is still "coming soon" for the first, and points to the second as the complete implementation README.md. Code is at commit dfbe63a793026b231b3cd60073aaa91a2ad4d06a.

• Priority (θ). Paper: θ_i = t_i / ||T||, the share of ground-truth triplets that contain node i eq. 6 §3.2. lib/nps_loss.py counts each node's distinct related neighbours over positive ground-truth relations, normalises by the image total, and uses 2θ inside the formula nps_loss.py. With simple graphs this equals the paper's θ, but a pair with several predicates counts once in the code and once per triplet in the paper.
• Priority in the detector build. gpsnet-detectron does not count triplets. It counts, for each proposal, how many ground-truth subject and object boxes it overlaps at IoU above 0.5, and normalises over all proposals in the batch model_builder_rel_gpsnet.py. This is a different priority from the paper's.
• Exponent. Paper's best mu is 4 §4.4. Both code paths use exponent 5 nps_loss.py model_builder_rel_gpsnet.py.
• Focusing weight gradient. The paper's equation does not say whether the focusing weight is differentiated. Both code paths detach the probability used in the weight, so only the -log p term gets gradient nps_loss.py refine_obj_feats.py.
• Reduction. The paper does not state how node losses are reduced. Both paths take the mean nps_loss.py refine_obj_feats.py.
• Attention for non-candidate pairs. The paper defines e_ij for pairs but does not say what happens to pairs without a union box. The code fills a zero tensor, adds scores at candidate pairs, then applies a sigmoid to the whole tensor, so every other pair gets sigmoid(0) = 0.5, and only the diagonal is masked refine_obj_feats.py.
• Attention normalisation. The paper says α is normalised §3.1 but gives no formula. The code divides atten_tensor by atten_tensor.sum(1) [code: gpsnet-detectron/lib/modeling_rel/refine_obj_feats.py#L120]. Broadcasting divides entry (i, j) by the sum of row j, not of row i, so the result is not normalised over the subject index. The released lib/direction_aware_context.py has the same expression direction_aware_context.py. Whether this is intended is not clear from the code.
• Refinement block. Paper's eq. 5 has no ReLU after the projection and no ReLU on the output eq. 5. The code puts a ReLU after W_t3 [code: gpsnet-detectron/lib/modeling_rel/refine_obj_feats.py#L152-L159] and a ReLU on the output refine_obj_feats.py. The layer-norm and FC sizes (512 to 128 to 512) match the paper.
• Stacked message passing. The paper describes one DMP module §3.1. The detector stacks two message-passing layers for objects [code: gpsnet-detectron/lib/modeling_rel/model_builder_rel_gpsnet.py#L517-L518]. The paper does not state the number of layers.
• Frequency softening. Paper: log-softmax of the frequency vector eq. 9. The code first normalises class-pair counts to probabilities, adds an epsilon, then applies log-softmax to those probabilities, with background counts in index 0 sparse_targets_rel.py. The paper does not say whether its p is a count or a probability.
• Adaptive gate. The paper's gate d = sigmoid(W_p u) and its use as d ⊙ p̃ eq. 10 are not in the released code. The detector adds the prior to the logits with no gate model_builder_rel_gpsnet.py.
• Fusion. The paper's fusion is the ReLU-based function in §3.3. The detector uses a product of two learned projections of node features, word embeddings and a 5-d position, multiplied by the union feature [code: gpsnet-detectron/lib/modeling_rel/model_builder_rel_gpsnet.py#L157-L161] model_builder_rel_gpsnet.py.
• Relation loss. The relation loss is plain cross-entropy [code: gpsnet-detectron/lib/modeling_rel/reldn_heads.py#L162]. The paper does not describe a relation loss beyond the classifier in eq. 10.
• Learning rate and batch. Paper: learning rate 1e-3, batch 6 §4.2. Config: base LR 0.01 with one image per GPU on 8 GPUs [code: gpsnet-detectron/configs/vg/e2e_faster_rcnn_VGG16_8_epochs_vg_v3_default_sgcls.yaml#L14-L18] e2e_faster_rcnn_VGG16_8_epochs_vg_v3_default_sgcls.yaml. The library default is 1e-3 config.py.
• NMS. Paper: per-class NMS at IoU 0.3 §4.2. The VG config sets 0.5 [code: gpsnet-detectron/configs/vg/e2e_faster_rcnn_VGG16_8_epochs_vg_v3_default_sgcls.yaml#L43]; the library default is 0.3 config.py.
• Background sampling. Paper: 3 : 1 background to foreground §4.2. The code fills to a fixed 64 pairs with background after keeping foreground [code: gpsnet-detectron/lib/modeling_rel/relpn_heads.py#L324]. The 3 : 1 ratio is not checked.
• Completeness. lib/nps_loss.py calls np.zeros but imports only torch and Variable nps_loss.py, so it does not run as shipped. The detector code is written for PyTorch 0.4.x README.md and uses old-style Variable and F.sigmoid calls.

Pitfalls

• Dimension conflict. The paper gives u_ij as 2048-d §3 but W_u as 512 x 512 in the tri-linear score §3.1. The released code multiplies the union feature with 512-d products refine_obj_feats.py, so a projection must happen before the module. Where it happens is not in the released files.
• Stacking table labels. In tab. 6 (left) the bold values sit in the column labelled "w.o. stack", while the text says stacking helps §4.4. The printed numbers are lower in the "w. stack" column. Treat the direction as unresolved; the code always stacks the two directions refine_obj_feats.py.
• Missing pairs. Non-candidate pairs get weight 0.5 in the code (see Paper vs code). An implementation that uses 0 there will not match.
• Gradient through the focusing weight. The code detaches it. A standard focal-loss implementation does not. The results depend on which the authors used.
• Priority definition. The triplet-share priority of the paper and the proposal-overlap priority of the detector are different quantities. Ablation numbers from the released code therefore do not test the paper's definition.
• Gamma edge cases. A node with no triplets has theta = 0 and -log(theta) is infinite. The min(2, ...) cap sends its focusing parameter to 2. The code does the same through clamp(max=2) or np.minimum.
• Not stated. Training length, learning-rate schedule (in the paper), weight decay, momentum, the loss weights between object and relation losses, and the number of message-passing layers are not stated in the paper.
• Cost. Attention is computed for N x N pairs per image, and the code builds an N x N x p tensor refine_obj_feats.py. The paper does not discuss compute or memory.
• Numbers are author-reported. Comparisons across detectors (◇ vs ‡) and across datasets use different backbones and splits. The ‡ rows use RelDN's detector, so compare within a row group.
• Code state. The released code is partial and needs PyTorch 0.3 or 0.4 and CUDA. Reproducing the table needs the full detector build, which is only in gpsnet-detectron/.
• Licensing of the detector folder. The root LICENSE is MIT LICENSE. gpsnet-detectron/ adapts code from RelDN and Detectron.pytorch. Its README refers to a "LICENSES" file that is not in the repository, and the Detectron.pytorch submodule directory is empty in this clone. Cite its code by path only.

Lineage

• Builds on: Neural Motifs (Zellers et al., CVPR 2018), the baseline and source of the feature construction and frequency prior §3, §4.4. Also GCNet for the global-context simplification §3.1, graph attention networks for the neighbourhood notation §3.1, focal loss for the focusing parameter §3.2, and Tucker-style tri-linear fusion §3.1. The paper gives no arXiv identifiers for these; no card exists in this wiki.
• Compared against: VCTREE, CMAT, KERN, RelDN, Graph-RCNN, IMP, FREQ, MOTIFS and GPI tab. 1; RelDN, ViP-CNN, VRL, KL distillation, MF-URLN, Zoom-Net and CAI + SCA-M tab. 4.
• Superseded by: not known from wiki content.