Temporal action proposal generation

Overview

Given an untrimmed video, a temporal action proposal generator returns candidate time segments, each with a start and an end, that should cover the action instances in it 2102.01894 §12102.01894 §3.1. The proposals are then scored and labelled by a separate stage. The paper measures proposal quality as average recall at a given number of proposals (AR@AN), using tIoU thresholds from 0.5 to 1.0 in steps of 0.05 on THUMOS14 2102.01894 §4.2.

The shared requirement is a set of segments that covers every instance at a bounded output size. The variants differ in how candidates are generated and whether a clean-up step is needed afterwards.

Variants

Anchor-based dense windows

Many windows of several lengths are placed over the video and each is scored. The paper's view is that covering all instance durations this way is hard to do at a reasonable cost 2102.01894 §2. No card on this page covers this family yet.

Boundary scoring and pairing

Each frame gets a start and an end probability; starts are then matched with ends using local context. The paper says these methods are sensitive to noise, often produce incomplete proposals, and rely on hand-designed matching and non-maximum suppression (NMS) 2102.01894 §12102.01894 §2. No card on this page covers this family yet.

Direct set prediction with a transformer decoder

Card: Relaxed Transformer Decoders for Direct Action Proposal Generation (RTD-Net). Its mechanism, as the paper describes it:

• Boundary-attentive features. Features are multiplied by the predicted start and end scores, scaled by a factor of 2, and the two products are concatenated 2102.01894 §3.52102.01894 tab. 3.
• Parallel decoding of learned queries. A DETR-style decoder with six layers turns a fixed set of learned queries into segments in one pass 2102.01894 §3.32102.01894 app. A.2.
• Three heads per query: a boundary head, a foreground probability and a completeness estimate of how well the query overlaps a ground-truth instance 2102.01894 §3.32102.01894 §3.4.
• Training: Hungarian matching, then extra "relaxed" positives after the strict assignment, then a three-stage schedule 2102.01894 §3.42102.01894 app. A.2.
• Inference: every query yields a segment with no NMS, scored by the mean of the foreground and completeness outputs 2102.01894 §3.5.

Author-reported THUMOS14 test results: AR@50 41.52, AR@200 56.41 and AR@500 62.91, against BMN with soft-NMS at 39.36, 54.70 and 62.07 2102.01894 tab. 1. Relaxed matching adds 0.45 AR@50 over strict matching alone in the authors' ablation 2102.01894 tab. 5.

When to use which

The choice depends on the constraints below. Each row states only what the cited results support; all numbers are author-reported.

The paper's ablations also show that one decoder layer is far below six: AR@50 32.76 against 41.52 2102.01894 tab. F. Start from the paper's depth rather than a smaller one.

Implementation sketch

The core of the direct variant, in pseudocode. Names are ours; the paper gives the steps but not a reference implementation.

# Inputs: snippet features X (T, D); start and end scores s, e (T,)
# Q learned queries; no NMS at any stage
def propose(X, s, e, queries):
    # Scale each boundary score to [0, 1] in the window [arxiv:2102.01894 §3.5]
    s, e = normalise(s), normalise(e)
    # Re-weight features by twice each score, then join them
    # [arxiv:2102.01894 tab. 3]
    H = concat(X * (2 * s)[:, None], X * (2 * e)[:, None])
    # Encode with a three-layer MLP plus position [arxiv:2102.01894 §3.5]
    memory = mlp3(add_position(H))
    # Decode all queries in parallel, six layers [arxiv:2102.01894 app. A.2]
    hidden = decoder6(queries, memory)
    # Heads: segment, foreground, completeness [arxiv:2102.01894 §3.3]
    segments = boundary_head(hidden)
    p_fg, p_c = fg_head(hidden), completeness_head(hidden)
    # Score is the mean of the two confidences; keep every query
    # [arxiv:2102.01894 §3.5]
    return segments, (p_fg + p_c) / 2

# Training assignment, after the strict stage [arxiv:2102.01894 §3.4]
def assign(predictions, truths, strict):
    # Hungarian matching on the cost of eq. 1 [arxiv:2102.01894 eq. 1]
    pairs = hungarian(cost(predictions, truths))
    # Relaxed stage: add extra positives [arxiv:2102.01894 §3.4]
    return pairs + relaxed_positives(predictions, truths, pairs)

Contradictions

• Feature source of Table 1. Section 4.2 says the main experiments use I3D features 2102.01894 §4.2. The footnote to the RTD-Net row in Table 1 says its results use P-GCN I3D features 2102.01894 tab. 1.
• Timing. The text gives 0.114 s per sample 2102.01894 §4.2. The runtime breakdown for a three-minute video in Table H lists 49.29 ms for boundary prediction and re-weighting, 0.32 ms for the encoder, 8.97 ms for the decoder and 0.89 ms for the heads 2102.01894 tab. H. The paper does not reconcile the two.
• Paper and released code. The card for the paper lists where the released code differs from the text: matcher weights, loss forms, the stage that trains the boundary head, and checkpoint selection on the THUMOS14 test split. See the card's Paper vs code section before reimplementing from the text. This is not a claim the paper makes about itself.

Papers

• Relaxed Transformer Decoders for Direct Action Proposal Generation: a DETR-style decoder that outputs temporal segments directly, with boundary re-weighting, relaxed matching and a completeness head; no NMS.