UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detection

In plain words

Finding "a person rides a horse" in a photo usually takes two steps. First a detector finds the people and the objects. Then a second network checks every person-object pair to decide whether they interact. Checking many pairs is the slow part.

UnionDet adds a second kind of detector that looks at the whole region covering a person and an object at once. It asks, in one pass, "where is the box around this interaction, and what is the action?" The ordinary detector for people and objects runs alongside it, and the two outputs are combined to score each candidate triplet. The authors report that this removes the per-pair network pass, so the extra time for interaction prediction is small. The difficulty is that a box around two things is harder to train than a box around one thing: the boxes are biased toward the person, and overlapping regions give conflicting labels. The paper adds three training changes for this.

TL;DR

UnionDet is a one-stage human-object interaction (HOI) detector built on RetinaNet with ResNet50-FPN §3. A union branch predicts action classes and boxes for each interacting pair; an instance branch predicts people, objects and their actions §3.2 §3.3. The authors report the top V-COCO and HICO-DET scores of the compared methods, with about 9 ms of added interaction time tab. 1 tab. 2. The results are author-reported, and no official code was found.

Problem

HOI detection outputs a ⟨human, verb, object⟩ triplet with boxes §1. Prior detectors first detect objects, then pair them and run a network per pair, so the pipeline cannot be trained end to end and the per-pair stage dominates inference time §1 §2. The paper argues that a detector predicting the union box of an interacting pair directly can remove this stage §1 §3.

Three problems come with union-level detection §3.1 fig. 1:

• Bias toward people. Every union box contains a person, so a plain detector puts its predictions densely around people.
• Standard IoU is a poor match score. A box can reach high IoU with a union region yet miss one of the two objects when they are small or far apart.
• Overlap. One union box can hold several interactions and more than two objects, and different pairs can share the same union box.

Method

The architecture is one meta-architecture with two parallel branches on a shared anchor set §3. The authors state that it works with any one-stage detector; they implemented RetinaNet with ResNet50-FPN for a fair comparison with Faster R-CNN based baselines §3. The shared backbone is a feature pyramid network §3.2. fig. 2 shows the pipeline.

Union branch (§3.2)

The union branch has three sub-branches on the shared pyramid: action classification, union box regression, and target-object classification. The first two produce detections; the third is used only in training §3.2. It is trained with three changes.

Union anchor labeling §3.2 eq. 1. An anchor is positive for a ground-truth union box only when three tests all pass: the anchor's IoU with the union box is above t_u, the anchor covers more than t_h of the human box, and more than t_o of the object box. The paper sets all three to 0.5 §3.2. When an anchor matches several ground truths, it keeps the one with the largest IoU §3.2.

Vanilla union loss §3.2 eq. 2. Positive anchors get focal loss on the action labels, a smooth L1 loss on the box, and no other term. Negative anchors get focal loss toward an all-zero action target. The paper notes that this vanilla form is biased toward people and suffers from overlapping union boxes.

Target object classification §3.2 eq. 3. A binary cross-entropy loss on the target object class is added for positive anchors, so the union branch learns to find the object as well. The score is not used at inference. The authors report that it improves union detection and the final result tab. 3.

Union foreground focal loss §3.2 eq. 4. The focal action loss is multiplied by the ground-truth action label, so only the positive action labels of a positive anchor are counted. The authors argue that the usual focal loss can push overlapping union actions toward negative loss. Their variant drops the negative-label terms for matched anchors; the background term for unmatched anchors is unchanged eq. 2 eq. 4.

Instance branch (§3.3)

The instance branch is an anchor-based single-stage detector for objects, with an extra action sub-branch. It predicts T_s + T_o action types per anchor, where subject-side and object-side actions are separate types, so that the direction of an interaction is kept §3.3.2. The action loss is computed only at anchors that contain an object §3.3.2. Object classification and box regression use fixed pre-trained detector weights, and only the action sub-branch is trained for that loss §3.3.3.

Training (§3.4)

The two branches are trained jointly. The total loss is the plain sum of the union loss and the instance loss, with no weights stated §3.4. Focal loss uses alpha = 0.25 and gamma = 2.0 §3.4. The optimiser is Adam with learning rate 1e-5 §3.4. The object detector is frozen on V-COCO and fine-tuned on HICO-DET §3.3.1.

Inference (§3.5)

Both branches run in parallel. Each branch's detections go through non-maximum suppression: the instance detections per class using object scores, and the union detections with all action classes pooled, not class-wise §3.5.

For each human-object pair, the method picks the union box with the highest union-instance matching score §3.5.1 eq. 5. This score averages two terms with weight 1/2 each: the IoU between the tightest box covering the pair and the union box, and the geometric mean of the fractions of each person and object box that the union box covers eq. 5. The HOI score for the pair multiplies the instance scores of the pair by one plus the matching score times the union action score §3.5.2 eq. 6. Without a target object, the score is the human score times the human action score. With no union box, it adds the object score §3.5.2.

The paper reports that the final scoring takes under 1 ms, and that the full model adds about 9 ms to the base detector §3.5 §4.

# Union anchor labeling for one ground-truth triplet  [§3.2] [eq. 1]
# Input:  anchors (N, 4); union box g, human box h, object box o
# Output: positive-anchor mask (N,) for this triplet

def union_anchor_mask(anchors, g, h, o):
    # 1. Anchors must overlap the union box enough  [eq. 1]
    overlap = iou(anchors, g) > 0.5
    # 2. Anchors must cover most of the human box  [eq. 1]
    covers_human = intersection(anchors, h) / area(h) > 0.5
    # 3. Anchors must cover most of the object box  [eq. 1]
    covers_object = intersection(anchors, o) / area(o) > 0.5
    # 4. An anchor is positive only if all three tests pass  [eq. 1]
    return overlap & covers_human & covers_object

# Pair scoring at inference  [§3.5] [eq. 5] [eq. 6]
# Input:  human and object detections, union detections with
#         action scores
# Output: one HOI score per (human, object) pair

def hoi_scores(humans, objects, unions):
    # 1. For each pair, find the best-matching union box  [eq. 5]
    for each (human, object) pair:
        best_union = argmax over unions of union_match(human, object, u)
        # 2. Combine instance scores with the union action score  [eq. 6]
        if best_union exists and involves a target object:
            score = (s_h * s_h_act + s_o * s_o_act) * (1 + mu * s_u_act)
        else if no union box:
            score = s_h * s_h_act + s_o * s_o_act
        else:
            score = s_h * s_h_act
    return scores

Hyperparameters

Results

Author-reported. V-COCO test set and HICO-DET test set, both evaluated with the official code; the authors report both V-COCO evaluation scenarios because prior papers do not always say which one they used §4.1 §4.

Table 1 also lists methods that use pose, keypoints or word embeddings. The authors say UnionDet is competitive with them but uses no external knowledge tab. 1 §4. On HICO-DET, the table lists methods with external features separately, and the paper states that those are outside its scope tab. 2.

Ablation on V-COCO, scenario 1 / scenario 2 tab. 3:

The paper's table caption describes the first row as the instance branch alone tab. 3.

Paper vs code

No official code. The paper and its LaTeX source name no repository or project page §4. A GitHub repository search for the title and for "UnionDet human object interaction" returned no results, and the author account pages I checked had no public repositories.

Pitfalls

• Which parts are trained is unclear. The instance section says the object classification and box regression sub-branches use pre-trained weights and only the action sub-branch is trained §3.3.3. The training section says both branches are trained jointly §3.4. The paper does not say whether the shared backbone is updated. Frozen versus fine-tuned differs by dataset §3.3.1.
• Equation 1 divides by ground-truth box areas. The human and object inclusion ratios are written as anchor overlap over the human or object box. This means the anchor must cover a large part of each person and object, not the other way round. The paper does not discuss anchors larger than the boxes.
• Positive definition. Positive anchors are written as those with exactly one match §3.2. The largest-IoU rule keeps the match count at one or zero, so the two statements agree.
• Negatives are dropped on matched anchors. Under the union focal loss, a positive anchor contributes nothing for action labels it does not match eq. 4. The paper gives no ablation of this effect beyond tab. 3.
• Matching score cost. The paper states the scoring is O(n^3) in the number of detections n §3.5. The authors report it takes under 1 ms in their setting. The detection cap and NMS thresholds that bound n are not stated.
• Loss weights and schedule are not given. The loss terms are summed with weight 1, the learning rate is 1e-5 with Adam, and no schedule, batch size or iteration count is stated §3.4. Expect to tune these when reimplementing.
• Timing is not comparable. The 9.06 ms figure is the added cost over the base detector. The end-to-end time (77.6 ms, one GTX 1080 Ti) depends on the backbone and the detector, and the authors warn against comparing it §4.
• Evaluation scenario. V-COCO has two AP_role evaluation scenarios, and the two columns in tab. 1 are not the same measurement. Compare numbers only within the same scenario.
• Supplement not available here. The text refers to a supplement for training details, end-to-end timing and qualitative results without a union branch §3.2 §4. It is not in the source archive used for this card.

Lineage

• Builds on: RetinaNet and focal loss for the one-stage detector §3.4 (Lin et al.); the union box as a visual relationship representation §3.1 (Lu et al., 2016).
• Compared against: VSRL tab. 2 (Gupta and Malik, 2015); InteractNet tab. 1 (Gkioxari et al., 2018); iCAN tab. 1 (Gao et al., 2018); TIN tab. 1 (Li et al., 2019); DCA tab. 1 (Wang et al., 2019); GPNN tab. 1 (Qi et al., 2018).
• Superseded by: not known from the content of this wiki.