A Modern Take on Visual Relationship Reasoning for Grasp Planning

In plain words

When a robot takes objects out of a cluttered bin, some items sit on top of others. The top ones have to come out first, or the pile collapses. Knowing which object is blocking which lets a robot plan a safe picking order. Older methods first find the objects and then guess the blocking relations between each pair, using crops of the picture around each pair. In a crowded bin these crops often contain the wrong things.

This card's paper, D3G, does both steps in one network. It looks at the whole picture, proposes a fixed set of object slots, and then fills in a table that says, for every pair of slots, whether one blocks the other. Small graph layers let each pair's answer take the other objects into account. The authors also built a bin-picking test set with many objects per scene and used a relation-level average precision to compare methods. The model has the best relation AP on their synthetic splits and on the real-image split, though not every other metric is best.

TL;DR

D3G is a DETR-style detector whose decoder output is turned into an N x N dependency graph by a dense graph transformer; it is trained with Hungarian matching for the objects and a BCE loss on the matched object pairs. The paper adds D3GD, a bin-picking benchmark built on MetaGraspNetV2 with difficulty splits, and uses Average Precision of Relationships (AP_rel) as the main metric. Relevant if you need object-dependency graphs for picking order in dense clutter. Results are author-reported. Code is MIT-licensed.

Problem

Robot picking in clutter needs a dependency graph: object A must be removed before object B if A rests on B §1 fig. 2. Prior visual manipulation relationship methods (VMRN, GVMRN, GGNN) detect objects first and then encode each pair with object-pair pooling, which crops the feature map on the union of the two boxes §1 §6.1. In dense clutter the crop can be dominated by one object or contain unrelated ones, which causes redundant or missing edges §1. Evaluation in prior work relies on mAP, OR and OP, and OR/OP depend on an object-score threshold that is usually tuned on a validation set §5. The authors also note that existing datasets (VMRD, REGRAD) are simpler than bin scenes: up to 35 objects from 97 classes in MetaGraspNetV2 §3. The paper notes that relations in its data can be annotated exhaustively, unlike Open Images §5.

Method

D3G has three parts. Detection follows DETR §4.1. A graph transformer then works on the N decoder outputs ("object queries") §4.2. A loss matches the queries to ground-truth objects and supervises both the detections and the relations §4.3. Symbols below follow the paper: N queries, P ground-truth objects, F node dimension, S heads, z = F/S, L layers.

Detection (§4.1). An image of size 3 x H0 x W0 goes through a CNN backbone that outputs C = 2048 channels at stride 32. A 1 x 1 convolution reduces the channels to d, and the spatial map is flattened into a sequence. A transformer encoder with positional encodings processes it. A transformer decoder with N learned object queries produces N embeddings, which feed box and class heads §4.1. The paper does not state N or d; the code uses 100 queries and width 256 detr_base.yaml configs.py.

Edges (§4.2). Three learned maps M_h, M_e1, M_e2 turn the N node embeddings into h0 (N x F), e1 and e2. Pairwise concatenation of e1 and e2 gives an N x N x 2F tensor, which M_e halves to F; this gives e0 §4.2. The paper's alternative edge initializers (Hadamard product, sum, difference) are compared in tab. 4. Pairwise concatenation is kept.

Graph transformer layer (§4.2). Each layer has S heads and works like multi-head attention with edge features. For each head, query and key projections of the nodes form a pair score, which is multiplied by an edge projection, giving a pseudo-attention tensor per pair eq. 1. Summing over the z channels and applying a softmax across the first node index gives the attention matrix eq. 2. The new node features are the attention-weighted values of all heads, concatenated and projected eq. 3. The new edge features are the pseudo-attention of all heads, concatenated and projected eq. 4. Both paths then use residual connections, normalization and a feed-forward network. The paper argues that this dense form, with a fixed set of queries, is cheaper than message passing over sparse graphs §4.2.

# Dense graph transformer layer  [§4.2] [eq. 1-4]
# Input:  nodes (N, F), edges (N, N, F)
# Output: nodes (N, F), edges (N, N, F)

def graph_layer(nodes, edges):
    # Per-head pseudo-attention from node pairs and edge features  [eq. 1]
    for head in range(S):
        pair_scores = (
            queries[head](nodes) @ keys[head](nodes).T / sqrt(z)
        )                                             # (N, N, z) per pair
        pseudo_attn[head] = pair_scores * edge_proj[head](edges)
    # Normalise across the first node index  [eq. 2]
    for head in range(S):
        attn[head] = softmax(sum_over_z(pseudo_attn[head]), axis=0)
    # Node update: attention-weighted values, heads concatenated  [eq. 3]
    nodes = concat(attn[head] @ values[head](nodes)) @ O_h
    # Edge update: pseudo-attention, heads concatenated  [eq. 4]
    edges = concat(pseudo_attn) @ O_e
    # Residual, normalisation and feed-forward  [§4.2]
    nodes, edges = add_norm_ffn(nodes, edges)
    return nodes, edges

Edge read-out (§4.2). After the last layer only the edge output is kept. An MLP maps each edge to one logit, and the N x N logit matrix is the predicted adjacency matrix §4.2.

# D3G forward pass  [§4.1] [§4.2]
# Input:  image (3, H0, W0)
# Output: class scores and boxes per query; N x N relation logits

def d3g_forward(image):
    # Backbone features at stride 32, reduced to d channels  [§4.1]
    features = conv1x1(backbone(image))           # (d, H, W)
    # Encoder over the flattened map, decoder over N learned queries  [§4.1]
    memory = encoder(flatten(features))
    nodes = decoder(object_queries, memory)       # (N, F)
    class_scores, boxes = detection_heads(nodes)
    # Node and edge inputs, then L graph transformer layers  [§4.2]
    h = map_h(nodes)                              # (N, F)
    edges = map_e(pairwise_concat(map_e1(nodes), map_e2(nodes)))
    for layer in graph_layers:
        h, edges = layer(h, edges)
    # One logit per ordered pair (i, j)  [§4.2]
    relation_logits = edge_mlp(edges)             # (N, N)
    return class_scores, boxes, relation_logits

Loss (§4.3). Ground-truth objects are padded with the empty class to N. The Hungarian algorithm finds a one-to-one assignment sigma between queries and objects, minimising a matching cost made of the class probability and a box term that combines L1 and generalized IoU eq. 5 eq. 6. The node loss is the class negative log-likelihood, including the empty class, plus the box loss on matched objects eq. 7. The relation loss first builds a selection matrix T (N x P) that picks the matched queries §4.3. The predicted sub-graph is G-hat = T^T e_pred T eq. 8. The relation loss is a binary cross-entropy over all P x P ordered pairs against the ground-truth matrix G eq. 9. The total loss is node plus relation §4.3. Fig. 4 shows this procedure fig. 4.

# Set loss with a relation term  [§4.3] [eq. 5-9]
# Input:  class_scores (N, K+1), boxes (N, 4), relation_logits (N, N)
#         gt_classes (P,), gt_boxes (P, 4), gt_graph (P, P)
# Output: scalar loss

def d3g_loss(class_scores, boxes, relation_logits, targets):
    # One-to-one assignment of queries to objects  [eq. 5] [eq. 6]
    assignment = hungarian(match_cost(class_scores, boxes, targets))
    # Node term: class NLL (empty class included) plus box loss  [eq. 7]
    node_loss = node_criterion(class_scores, boxes, assignment, targets)
    # Selection matrix T (N, P) keeps the matched query pairs  [eq. 8]
    selection = one_hot_selection(assignment, num_queries, num_objects)
    predicted_graph = selection.T @ relation_logits @ selection
    # BCE over all P x P ordered pairs  [eq. 9]
    relation_loss = bce_with_logits(predicted_graph, gt_graph)
    return node_loss + relation_loss

Inference (§5, §4.2). The relation output is thresholded at 0.5 after the sigmoid, which yields the binary adjacency matrix used for OR and OP [code: models/detr_graph.py#L289]. AP_rel sweeps the detection confidence. A triplet counts as true positive when both boxes have IoU above 0.5 with an unmatched ground-truth object and all three labels match §5.

Hyperparameters

Source key: "paper" means stated in the paper; "code" means the official repository's config or code at code_commit. "not stated" means neither gives the value.

Results

Author-reported. Setup: the D3GD testbed built on MetaGraspNetV2 (synthetic scenes plus 743 real test images), compared against VMRN, GVMRN and GGNN (all Mask-RCNN ResNet-50) and EGTR (deformable DETR) §6.1 §6.3.

Benchmark splits and statistics

Splits are defined by object count and two clutter measures tab. 1:

The paper drops Trivial and evaluates Easy, Medium and Hard §3. Average statistics against VMRD and REGRAD tab. 2:

D3GD results (synthetic and synth-to-real)

Metrics: mAP (COCO-style, detection), AP_rel, OR and OP. Higher is better. Best per column is bold in the paper; here the paper's values are reproduced without emphasis tab. 3.

Headline numbers from the text §6.3: D3G (Def. DETR) reaches 76.96 AP_rel on Synth Easy and 46.45 on Synth Hard; D3G (DETR) reaches 24.44 AP_rel synth-to-real, against 13.36 for VMRN. The paper notes that the synth-to-real OR and OP numbers vary a lot because no validation set is available to tune the score threshold §6.3.

Design choices (synthetic, AP_rel)

Edge initialisation tab. 4:

Graph transformer size tab. 5:

The S=2, L=2 Easy value here (76.77) differs from the 76.27 in tab. 3 and tab. 4 for the same configuration. See Pitfalls.

VMRD (standard benchmark)

Results on VMRD with PascalVOC mAP tab. 6 §6.5. Time is in ms on a 2080Ti for D3G; competitor times are as claimed in their own papers §6.5.

IA is the fraction of dependency graphs reconstructed completely §6.5.

Paper vs code

Code: github.com/paolotron/D3G at commit dc21f2065c6a61cb5b41b9fb05acb90187f72871, MIT License, copyright Paolo Rabino (an author). The repository was cloned to raw/code/2409.02035. The discrepancies below were checked against that commit.

• Relation loss scale. Paper: sum over all P x P pairs of BCE eq. 9. Code: mean BCE over the matrix, multiplied by the number of matched GT nodes, which is sum divided by that number, per image and then averaged over the batch detr_graph.py detr_graph.py detr_graph.py. An implementation that follows eq. 9 literally will use a different loss scale than the released code.
• Relation loss on every decoder layer. Paper: L_tot = L_node + L_rel, one relation term §4.3. Code: with deep supervision the graph loss is also computed on each auxiliary decoder layer and stored under keys with a layer suffix detr.py. Those keys are not in the weight dictionary, so they are not scaled by the weight table [code: models/detr_graph.py#L226-L234]. The detectron2 trainer sums the returned values, which would make the relation loss a sum over all decoder layers. This is inferred from detectron2's usual behaviour; detectron2 is not in the repository, so it was not checked here.
• Loss weights. Paper: no weights given §4.3. Code: matching and classification weights 1 / 5 (L1) / 2 (GIoU), no-object weight 0.1, relation weight 1.0 detr_graph.py configs.py configs.py.
• Edge adapters. Paper: M_e halves the 2F concatenation to F §4.2. Code: the two edge projections map directly to F/2 each, and their concatenation is already F wide; there is no separate M_e detr_gheads.py detr_gheads.py.
• Attention details not in the paper. Code clamps the summed scores to [-5, 5] before the softmax graph_transformer_dense.py. Edge and node update use LayerNorm after the residual in the head [code: models/detr_gheads.py#L94] graph_transformer_dense.py.
• Training schedule. Paper: detection to convergence (about 18 epochs for DETR-based models), then relation training with detection fine-tuned at a lower learning rate for 6 epochs §6.2. Code: detection pretraining for 60,000 iterations at LR 6e-4 detr_pretrain.yaml; relation training for 30,000 iterations at LR 5e-4 from that checkpoint, with all parameters trainable detr_graphdense.yaml. The option to freeze detection and train only the graph head (FINETUNE_GHEAD) exists, but it defaults to False and is not set in the relation configs configs.py detr_graph.py. No two-phase schedule with its own epochs appears in the configs. The paper's epoch counts cannot be converted to iterations because the training-set size is not stated.
• Augmentation. Paper: random flip, brightness, contrast and saturation §6.2. Code (v3): the same four operations with ranges 0.9–1.1, followed by a resize to the transform's image size (default 512 x 512) [code: data/augmentations.py#L112-L119]. The base config also lists multi-scale training sizes and a crop detr_base.yaml; which of these applies to the relation runs was not traced.
• Ground-truth graph. The dense target is the transpose of the stored dependency matrix, multiplied by -1 metagraspnet_synth_mapper.py. The paper defines G_ij with direction (G_ij != G_ji) §4.3. The code convention is the one to follow; check it against the stored graphs before relying on it.
• Deformable variant loss. The deformable graph model divides the relation loss by the batch size without applying the graph weight [code: models/deformable_detr_graph.py#L499]. The DETR variant does apply it.
• README. The README example names configs/detr_graphdense_medium.yaml, which is not in configs/ README.md.

No discrepancies were found for the graph head width (256), heads (2) and layers (2) detr_graphdense.yaml, which match the paper's choices §6.4.

Pitfalls

• Loss scale is ambiguous. Eq. 9 and the released code give different relation-loss magnitudes (see Paper vs code). Choose one and document it.
• Edge direction and labels. The paper's OR and OP text describes three relation labels (in, out, no relation) §5, but the model predicts one logit per ordered pair. Direction comes from the (i, j) and (j, i) entries. Cross-check the sign and transpose convention of the ground truth against the code metagraspnet_synth_mapper.py.
• Relations are hard decisions. The relation output is thresholded at 0.5 after the sigmoid detr_graph.py. AP_rel then sweeps only the detection confidence, with a constant score for the relation [code: metrics/calculate_ap_results.py#L13]. The paper calls AP_rel "threshold-free" §5, which holds only for the detection-confidence sweep. The relation threshold stays fixed and affects OR and OP.
• Quadratic memory. Each graph layer holds an N x N x F edge tensor per sample. With the released settings (N = 100, F = 256, batch 32), that is about 8.2e7 values per tensor, roughly 330 MB in fp32 before intermediate copies for backward. The per-head pseudo-attention adds more [code: configs/detr_graphdense.yaml#L15-L21]. The paper's efficiency claim §4.2 is for compute, not memory; mixed precision is enabled in the config.
• Softmax axis. The paper says column-wise softmax §4.2. The code applies softmax over the first node index of the pair tensor and then contracts over the second index graph_transformer_dense.py. Implement it the way the code does, or verify the intended axis against the paper's figure.
• Unmatched queries. Training supervises only the P x P sub-matrix of matched queries §4.3. At inference, relations are read for all N queries, including those with low object scores detr_graph.py.
• Hidden choices. The paper does not state N, d, batch size, learning rates, loss weights, number of encoder and decoder layers, or the cost weights. The values in Hyperparameters come from the code. They are not the paper's stated values.
• Synth-to-real. No validation set exists in this setting, so the score threshold cannot be tuned for OR and OP; treat those numbers as unstable §6.3.
• Timing. Competitor inference times come from their own papers and were not measured on the same hardware §6.5.
• Table and equation numbers. Numbers were assigned in source order, not checked against the compiled PDF. Equation numbers were counted from the LaTeX source (nine numbered equations). Verify against the paper before citing elsewhere.
• Internal inconsistency. The S=2, L=2 Easy AP_rel is 76.77 in tab. 5 but 76.27 in tab. 3 and tab. 4. The other columns agree. Treat the Easy value as uncertain.

Lineage

• Builds on: DETR for the detection transformer and set prediction §4.1; the graph transformer of Dwivedi and Bresson, "A Generalization of Transformer Networks to Graphs" (arXiv id not given in the source) §4.2; Deformable DETR as an alternative detector §6.1.
• Compared against: VMRN, GVMRN and GGNN (object pair pooling with Mask-RCNN) and EGTR (scene-graph generation from a deformable DETR decoder) §6.1.
• Benchmark: built on MetaGraspNetV2 §3; the paper names VMRD and REGRAD as earlier datasets §3.
• Superseded by: not known from this wiki.