Relaxed Transformer Decoders for Direct Action Proposal Generation

In plain words

Finding when actions happen in a long video means proposing candidate time segments. Older methods either place many fixed-length windows over the video or guess start and end points separately and then pair them up, and both need a clean-up step to remove duplicates. This paper reuses a transformer decoder, the design that finds objects in images without anchors: a fixed set of learned slots each output one segment, all at once.

Video needs three adjustments. Neighbouring frames look almost the same, so the model sharpens its features at likely start and end points. Labelled boundaries are fuzzy, so more than one slot can reasonably count as correct for the same action. And each guess gets a second score that estimates how complete it is. The result is a pipeline without the clean-up step, which the authors report as faster than their bottom-up baseline.

TL;DR

RTD-Net is a direct action proposal generator for untrimmed video, built by adapting the DETR decoder. It uses two-stream features (I3D in the main experiments), reweights them with predicted boundary scores, and decodes a fixed set of learned queries into segments with a three-branch head: boundary, foreground score and completeness. Training uses bipartite matching, a relaxed top-1 assignment, and a three-stage schedule. Author-reported THUMOS14 test results are the best in the authors' comparison at AR@50, AR@200 and AR@500. The released code differs from the paper in matching weights, loss forms and staging, so it is not a faithful copy of the text.

Problem

Temporal action proposal generation must return start and end times that cover each action instance §1. Anchor-based methods place dense, multi-scale windows; this is hard to make cover all instance durations at reasonable cost §2. Boundary-based methods score every frame's boundary probability and then match starts with ends using only local context. They are sensitive to noise, often produce incomplete proposals, and depend on hand-designed matching and non-maximum suppression (NMS) §1§2.

Two properties of video stop a direct DETR-style decoder from working as-is. Features change slowly over time, so self-attention in the encoder smooths out the boundaries that matter §1§3.3. Temporal labels are also ambiguous and sparse, so a strict one-to-one assignment can slow or destabilise convergence §1§3.3. The paper evaluates on THUMOS14 and ActivityNet-1.3 §4.1.

Method

Setup. A video with l_f frames should yield a set of proposals, each a start and end time; the ground truth is a set of N_g instances §3.1. The proposals come from a fixed number of learned queries, so the output is a set with a fixed cap §3.1§3.3.

1. Features. Each snippet's appearance and motion features are concatenated into one vector. Features are taken from a sliding window of length l_s with temporal stride τ §3.2. The paper uses I3D as its main feature extractor §2§4.2. The released loader asserts 2048-dimensional snippet features thumos14.py.

2. Boundary scores. A frame-level boundary predictor (three convolutional layers, following BSN) outputs start and end probabilities for every location §3.4. The released code does not contain this predictor; it reads precomputed start, end and frame scores from CSV files thumos14.py.

3. Normalise and scale. The window's start and end probabilities are normalised to [0, 1], then multiplied by a scale factor α_r. The paper picks α_r = 2 by ablation §3.5§4.3 tab. 3. The released model hard-codes 2 rtd.py.

4. Boundary-attentive representation. The features are multiplied separately by the start scores and by the end scores, and the two results are concatenated along the channel axis §3.5. Positional information is added and a three-layer MLP encoder compacts the result §3.5. The released code also applies a linear projection before the multiplication, and concatenates a sinusoidal position code before the MLP rtd.py transformer.py.

5. Decoder. A set of learned query embeddings is decoded in parallel. Each layer runs self-attention among the queries and cross-attention to the encoder output, so that every proposal is informed by the others §3.3. Six decoder layers work best in the ablation app. A.2 tab. F. Position information enters the cross-attention keys, not the input app. A.2 tab. E.

6. Three heads per query §3.3:

   • a boundary head that outputs the temporal box;
   • a binary classifier that outputs the foreground probability p_bc;
   • a completeness head that outputs p_c, an estimate of the query's tIoU with the best ground-truth instance §3.4.

7. Label assignment (training). Predictions are matched to ground truth with the Hungarian algorithm on the cost in eq. 1 §3.4. Then the relaxation step adds extra positives (step 9).

   C = \sum_{n:\sigma(n)\neq\emptyset} \left[ \alpha\,\ell_1(\psi_n, \hat\psi_{\sigma(n)}) - \beta\,\mathrm{tIoU}(\psi_n, \hat\psi_{\sigma(n)}) - \gamma\,p_{bc,n} \right] eq. 1

   With α = 1, β = 5 and γ = 2 §3.4.

8. Losses §3.4:

   L_{cls} = -\gamma\,\frac{1}{N}\sum_{n=1}^{N} \left( \hat p_n \log p_{bc,n} + (1-\hat p_n)\log(1-p_{bc,n}) \right) eq. 2

   L_{boundary} = \frac{1}{N_{pos}} \sum_{n:\sigma'(n)\neq\emptyset} \left( \alpha\,L_{loc,n} + \beta\,L_{overlap,n} \right) eq. 3

   L_{loc,n} = \lVert \hat t_s^{\sigma'(n)} - t_s(n) \rVert_1 + \lVert \hat t_e^{\sigma'(n)} - t_e(n) \rVert_1, \quad L_{overlap,n} = 1 - \mathrm{tIoU}(\psi_n, \hat\psi_{\sigma'(n)}) eq. 4

   L_{complete} = \frac{1}{N_{train}} \sum_{n=1}^{N_{train}} \left( p_{c,n} - \hat g_{\mathrm{tIoU},n} \right)^2 eq. 5

   Here ĝ is the largest tIoU of a training proposal with any ground-truth instance §3.4. The paper does not give numeric values for α and β in eq. 3, nor for γ in eq. 2 beyond the matcher value.

9. Relaxed matching. After the strict assignment σ, additional predictions are marked positive. The paper describes a tIoU threshold for this; the released code uses a top-1 rule instead (see Paper vs code) §3.4app. A.2.

10. Schedule §3.4app. A.2:

    • Step 1: train with the strict assignment and completeness frozen (eqs. 2–3).
    • Step 2: relaxed assignment, fine-tuning. The paper says the boundary and classification parts are tuned here.
    • Step 3: train only the completeness head (eq. 5), with the rest fixed. The paper's exact epoch counts are not stated.

11. Inference §3.5. Every query's boundaries are decoded and scored without NMS. The final score averages the foreground probability and the completeness score with equal weight §3.5.

# Boundary-attentive features  [§3.5] [code: models/rtd.py#L97-L106]
# snippets: (T, 2048) two-stream features
# start_p, end_p: (T,) boundary probabilities, min-max normalised
def boundary_attentive(snippets, start_p, end_p):
    # Linear projection to 256 dims  [code: models/rtd.py#L98-L100]
    projected = linear_projection(snippets)          # (T, 256)
    # Scale boundary weights by alpha_r = 2  [§4.3] [code: models/rtd.py#L101]
    start_w = 2.0 * start_p                          # (T,)
    end_w = 2.0 * end_p                              # (T,)
    # Weight separately, then concatenate on channels  [§3.5]
    by_start = projected * start_w[:, None]         # (T, 256)
    by_end = projected * end_w[:, None]             # (T, 256)
    return concatenate(by_start, by_end)             # (T, 512)

# Encoder and parallel decoder  [§3.3] [§3.5] [code: models/transformer.py#L54-L110]
# features: (T, 512); queries: (Q, 512) learned; pos: (T, 512) sine code
def decode(features, queries, pos):
    # Three-layer MLP over features joined with position code  [§3.5]
    memory = mlp3(concatenate(features, pos))        # (T, 512)
    # Decoder starts from zeros; queries supply position  [code: transformer.py#L101]
    layers = decoder6(zeros_like(queries), memory, queries, pos)
    # Keep the output of every layer for auxiliary losses  [code: main.py#L93-L97]
    return layers                                    # (6, Q, 512)

# Three heads, applied to the last decoder layer  [§3.3] [code: models/rtd.py#L112-L118]
# hidden: (Q, 512)
def heads(hidden):
    # Boundary head: centre and length, sigmoid to [0, 1]  [§3.3]
    boxes = sigmoid(mlp3_boxes(hidden))              # (Q, 2)
    # Foreground probability from a 2-way softmax  [§3.3]
    p_bc = softmax(linear_cls(hidden))[:, 0]         # (Q,)
    # Completeness: conv over queries, then MLP  [§3.4] [code: models/rtd.py#L115-L118]
    p_c = sigmoid(mlp3_iou(conv1d_stack(hidden)))    # (Q,)
    return boxes, p_bc, p_c

# Relaxed assignment, stage 2  [§3.4] [code: models/matcher.py#L159-L168]
# rows, cols: strict matches; tiou: (G, Q) overlap of each truth with each query
def relax_top1(rows, cols, tiou):
    # For every ground-truth instance, take its best-overlapping query  [app. A.2]
    for gt in range(number_of_truths):
        best_query = argmax(tiou[gt])                # one query per truth
        # Add it as a positive if no query already matches it  [§3.4]
        if best_query not in rows:
            rows.append(best_query)
            cols.append(gt)
    return rows, cols

Hyperparameters

Results

Author-reported. Setup: THUMOS14 with I3D features; training on the 200 validation videos with annotations and evaluation on the 213 test videos with annotations §4.1. Timing on one RTX 2080 Ti §4.2. Metric: average recall at a given average number of proposals (AR@AN), using tIoU 0.5 to 1.0 in steps of 0.05 on THUMOS14 §4.2.

The table footnote says RTD-Net results use P-GCN I3D features, while §4.2 states I3D. ActivityNet-1.3 validation tab. 2: RTD-Net gets AR@1 33.05, AR@100 73.21, AUC 65.78. BMN reports AR@100 75.01 and AUC 67.10; the authors note that RTD-Net is the only method there without NMS.

Temporal action detection, mAP at tIoU 0.7 / 0.6 / 0.5 / 0.4 / 0.3 on THUMOS14 test, with P-GCN tab. 7:

ActivityNet-1.3 detection mAP at tIoU 0.95 / 0.75 / 0.5 / average tab. 8: RTD-Net 8.61 / 30.68 / 47.21 / 30.83; BMN 8.29 / 34.78 / 50.07 / 33.85.

Runtime §4.2 app. C: RTD-Net takes 0.114 s per sample, against 5.804 s for BSN, of which 5.794 s is post-processing. A runtime breakdown on a 3-minute video tab. H lists RTD-Net at 49.29 ms for boundary prediction plus re-weighting, 0.32 ms for the MLP encoder, 8.97 ms for the decoder and 0.89 ms for the heads. The paper does not reconcile the two timing figures. GPU memory is 1,519 MB against 7,152 MB for BMN app. C.

Ablations (THUMOS14 unless noted):

• Scale factor tab. 3, AR@50: none 36.22; α_r = 1 40.39; α_r = 2 41.52; α_r = 5 39.76.
• Feature encoder tab. 4, AR@50: MLP 41.52; Transformer 33.69 (receptive field 64) and 36.01 (field 16).
• Relaxed matcher tab. 5: AR@50 on THUMOS14 goes from 41.07 without relaxation to 41.52 with it; AUC on ActivityNet-1.3 goes from 65.50 to 65.78.
• Completeness head tab. 6, AR@50: classification only 41.08; with completeness 41.52.
• Decoder layers tab. F, AR@50: 1: 32.76; 2: 38.92; 3: 40.71; 6: 41.52; 9: 38.36.
• Modality tab. K, AR@50: RGB 37.28; flow 38.75; early fusion 41.52.
• NMS tab. G: adding soft-NMS raises AR@50 to 42.02 but lowers AR@200 to 54.98 (from 56.41).
• Generalisation to unseen ActivityNet classes tab. I: trained on seen plus unseen classes, unseen-validation AR@100 73.09 and AUC 65.52.

Paper vs code

Official code: MCG-NJU/RTD-Action, linked from the abstract, at commit a1b357e (Apache-2.0). No search was needed. I read the training scripts, model, matcher, loss, dataset and evaluation code. The release contains only THUMOS14. The README mentions an ActivityNet release, but no ActivityNet loader or config is in this snapshot README.md.

• Matcher weights. Paper: ℓ1 weight 1, tIoU weight 5, p_bc weight 2 §3.4 eq. 1. Code: class cost 1, box L1 cost 5, tIoU cost 2 main.py matcher.py. The weights of the box and overlap terms are swapped.
• Box representation. Paper: L1 on start and end eq. 4. Code: L1 on centre and length rtd.py box_ops.py.
• "GIoU" is tIoU. The code's generalized_prop_iou computes plain tIoU, with no enclosing-interval term box_ops.py. The overlap loss, loss_giou, is 1 − tIoU rtd.py. This agrees with the paper's eq. 4; the name is misleading.
• Classification loss. Paper: binary cross-entropy scaled by γ eq. 2. Code: two-way cross-entropy with the no-object class weighted 0.1 rtd.py rtd.py, averaged over all queries.
• Boundary loss normalisation. Paper: divided by the number of positives N_pos eq. 3. Code: divided by the number of ground-truth boxes in the batch rtd.py.
• Completeness loss. Paper: squared error over all training proposals eq. 5. Code: smooth-L1 on a sampled subset, weighted 100. The subset holds every proposal with max tIoU above 0.7, the same number drawn from the 0.3–0.7 range, and twice that number drawn from below 0.3, with replacement rtd.py rtd.py. Sampling with replacement is not stated in the paper.
• Completeness architecture. Paper: temporal convolutions followed by one fully connected layer §3.4. Code: two 1D convolutions over the query axis, then a three-layer MLP rtd.py rtd.py.
• Encoder. Paper: three-layer MLP §3.5. Code: three layers, with the count hard-coded transformer.py. The code also applies a linear projection before the boundary weighting, which the text does not describe rtd.py.
• Relaxation rule. Paper: predictions above a tIoU threshold become positive §3.4 app. A.2. Code default: top-1 per ground truth, with the threshold rule (default 0.7) available but unused by the scripts main.py matcher.py.
• Latent bug in top-k. With relax_topk above 1, the recorded ground-truth index is the flattened position, not the truth's index matcher.py. The scripts never set it above 1, so this does not affect the released results.
• Stage 2 trains only the classifier. Paper: the fine-tuning stage tunes the classification and boundary heads app. A.2. Code: only the classification head has trainable parameters in stage 2; the boundary head is frozen main.py. The localisation loss in stage 2 therefore has no trainable parameters behind it.
• Auxiliary decoder losses. Not in the paper. Code: on by default, one per intermediate decoder layer main.py rtd.py.
• Stage 3 loss. Code: the completeness loss is smooth-L1, which equals half the squared error for errors below 1 rtd.py.
• Boundary predictor. Paper: a three-layer convolutional network trained at frame level §3.4. Code: reads precomputed scores from CSV and does not contain the training code thumos14.py.
• Positional code. The sinusoidal table uses base 100 rather than the common 10000 position_embedding.py. Only the position index is used; location values are ignored position_embedding.py. Not stated in the paper.
• Short videos. Videos shorter than 100 snippets are zero-padded in location, which maps to snippet 0, and in scores thumos14.py.
• Checkpoint selection. Paper: not stated. Code: evaluates on the THUMOS14 test split every epoch and keeps the checkpoint with the best AR sum main.py. No validation split is used, so reported test numbers are selected on test.
• Evaluation. Proposals from all windows of a video are pooled, sorted by score and never suppressed. AN is set by the fraction of the pooled list kept, then interpolated thumos14_eval.py thumos14_eval.py. The paper states AR@AN but not this procedure.
• Stage schedule. Paper: a two-step scheme §3.4. Released scripts: three stages with the epochs and learning rates in the hyperparameter table train.sh train.sh train.sh.
• README figure. The README lists AR@100 as 49.33 README.md; the paper has 49.32 tab. 1.
• Window normalisation. Coordinates are scaled by window length × interval (100 × 5 = 500) and features are indexed by frame ÷ 8, both hard-coded in the loader thumos14.py thumos14.py main.py.
• Classifier label. Table 7 rows for RTD-Net say "UNet" while the caption names UntrimmedNet tab. 7. The released code does not cover this step.

Pitfalls

• Matcher and loss weights. Copying the paper's weights (1, 5, 2) into eq. 1 gives the wrong assignment compared with the code. Decide which you are reproducing and say so.
• Parameterisation. The paper uses start and end; the code predicts centre and length with a sigmoid. Converting between them changes the L1 scale and the loss values.
• Name of the overlap loss. Calling the overlap term GIoU would add an enclosing-interval term the code does not use. Use plain tIoU.
• Completeness head axis. Its convolutions run over queries, not over time. A reimplementation that convolves over time will differ.
• Stage 2. Freezing the boundary head means relaxed positives change only the classification head. Freezing or training the boundary head changes results; the paper's text suggests training it.
• Loss normalisation. Dividing by ground-truth count (code) and by positive count (paper) differ once relaxation adds positives.
• Missing details. Relaxation threshold, γ in eq. 2, α and β in eq. 3, the completeness weight and the epoch counts are not stated in the paper. The values here come from the code only.
• Test-set selection. Checkpoints are chosen on the test split, so the reported numbers are optimistic for a held-out test.
• Evaluation protocol. AR@AN depends on pooling all windows and interpolating over the fraction of proposals kept; a per-video top-N count gives different numbers.
• Missing inputs. The TEM score files and I3D features must be downloaded separately; the boundary predictor is not in the repository.
• Runtime figures. The 0.114 s timing and the 59 ms breakdown are not reconciled; take both as author-reported and check on your hardware.
• Padding. Short videos reuse snippet 0's features and zero scores, which can bias proposals near the start.
• No ActivityNet code. ActivityNet and HACS numbers cannot be reproduced from this snapshot.

Lineage

• Builds on: DETR for the decoder and set-prediction loss §2 §3.4; BSN for the boundary predictor §3.4; DRN for the two-step completeness training §3.4.
• Compared against: BSN, BMN, MGG, DBG, G-TAD, BC-GNN, CTAP, TURN and others tab. 1 tab. 7 tab. 8.
• Contrasted with: a top-k to top-1 scheme in a PSS-head detector, which the appendix discusses app. A.2.
• Superseded by: not stated in the source.