---
kind: concept
slug: temporal-action-proposal-generation
title: Temporal action proposal generation
summary: >-
  Produces candidate start and end times for action instances in untrimmed video, for a later classifier to label. Anchor-based windows and boundary-pairing methods need matching and NMS; a direct transformer-decoder variant emits a fixed set of segments with no NMS. Only one card so far, and its numbers are author-reported on THUMOS14.
tags: [video, temporal-localization, detection]
papers: ["2102.01894"]
related: [anchor-free-detection, bipartite-matching, set-prediction, object-queries, bounding-box-regression-loss]
status: draft
updated_at: 2026-11-06
---

## Overview

Given an untrimmed video, a temporal action proposal generator returns
candidate time segments, each with a start and an end, that should cover the
action instances in it [arxiv:2102.01894 §1][arxiv:2102.01894 §3.1]. The
proposals are then scored and labelled by a separate stage. The paper
measures proposal quality as average recall at a given number of proposals
(AR@AN), using tIoU thresholds from 0.5 to 1.0 in steps of 0.05 on THUMOS14
[arxiv:2102.01894 §4.2].

The shared requirement is a set of segments that covers every instance at a
bounded output size. The variants differ in how candidates are generated and
whether a clean-up step is needed afterwards.

## Variants

### Anchor-based dense windows

Many windows of several lengths are placed over the video and each is scored.
The paper's view is that covering all instance durations this way is hard to
do at a reasonable cost [arxiv:2102.01894 §2]. No card on this page covers
this family yet.

### Boundary scoring and pairing

Each frame gets a start and an end probability; starts are then matched with
ends using local context. The paper says these methods are sensitive to noise,
often produce incomplete proposals, and rely on hand-designed matching and
non-maximum suppression (NMS) [arxiv:2102.01894 §1][arxiv:2102.01894 §2]. No
card on this page covers this family yet.

### Direct set prediction with a transformer decoder

Card: [Relaxed Transformer Decoders for Direct Action Proposal Generation](/papers/2102.01894)
(RTD-Net). Its mechanism, as the paper describes it:

- **Boundary-attentive features.** Features are multiplied by the predicted
  start and end scores, scaled by a factor of 2, and the two products are
  concatenated [arxiv:2102.01894 §3.5][arxiv:2102.01894 tab. 3].
- **Parallel decoding of learned queries.** A DETR-style decoder with six
  layers turns a fixed set of learned queries into segments in one pass
  [arxiv:2102.01894 §3.3][arxiv:2102.01894 app. A.2].
- **Three heads per query:** a boundary head, a foreground probability and a
  completeness estimate of how well the query overlaps a ground-truth
  instance [arxiv:2102.01894 §3.3][arxiv:2102.01894 §3.4].
- **Training:** Hungarian matching, then extra "relaxed" positives after the
  strict assignment, then a three-stage schedule [arxiv:2102.01894 §3.4][arxiv:2102.01894 app. A.2].
- **Inference:** every query yields a segment with no NMS, scored by the mean
  of the foreground and completeness outputs [arxiv:2102.01894 §3.5].

Author-reported THUMOS14 test results: AR@50 41.52, AR@200 56.41 and AR@500
62.91, against BMN with soft-NMS at 39.36, 54.70 and 62.07
[arxiv:2102.01894 tab. 1]. Relaxed matching adds 0.45 AR@50 over strict
matching alone in the authors' ablation [arxiv:2102.01894 tab. 5].

## When to use which

The choice depends on the constraints below. Each row states only what the
cited results support; all numbers are author-reported.

| Situation | Use | Why |
|---|---|---|
| Want no hand-tuned NMS and no anchor design | Direct decoder | The method emits segments with no NMS [arxiv:2102.01894 §3.5]. Adding soft-NMS to it raised AR@50 to 42.02 but lowered AR@200 to 54.98 [arxiv:2102.01894 tab. G]. |
| Need the best recall at 50, 200 or 500 proposals on THUMOS14 | Direct decoder | Highest of the three methods compared at those counts [arxiv:2102.01894 tab. 1]. At AR@100 BC-GNN with soft-NMS is higher, 49.60 against 49.32 [arxiv:2102.01894 tab. 1]. |
| Only RGB or only optical flow is available | Expect lower recall than two-stream | The paper's ablation gives AR@50 of 37.28 for RGB, 38.75 for flow and 41.52 for fused features [arxiv:2102.01894 tab. K]. |
| Recall on ActivityNet-1.3 at 100 proposals matters most | Compare with boundary methods before choosing | RTD-Net reports AR@100 73.21, against 75.01 for BMN [arxiv:2102.01894 tab. 2]. |
| Inference speed or GPU memory is the limit | Direct decoder, after checking on your hardware | Reported 0.114 s per sample against 5.804 s for BSN, of which 5.794 s is post-processing [arxiv:2102.01894 §4.2][arxiv:2102.01894 app. C]. GPU memory 1,519 MB against 7,152 MB for BMN [arxiv:2102.01894 app. C]. Timed on one RTX 2080 Ti. |
| Proposals feed a P-GCN detector | Direct decoder, per the paper | With P-GCN, the reported detection mAP is higher than G-TAD at every tIoU from 0.3 to 0.7 [arxiv:2102.01894 tab. 7]. |

The paper's ablations also show that one decoder layer is far below six: AR@50
32.76 against 41.52 [arxiv:2102.01894 tab. F]. Start from the paper's depth
rather than a smaller one.

## Implementation sketch

The core of the direct variant, in pseudocode. Names are ours; the paper
gives the steps but not a reference implementation.

```pseudocode
# Inputs: snippet features X (T, D); start and end scores s, e (T,)
# Q learned queries; no NMS at any stage
def propose(X, s, e, queries):
    # Scale each boundary score to [0, 1] in the window [arxiv:2102.01894 §3.5]
    s, e = normalise(s), normalise(e)
    # Re-weight features by twice each score, then join them
    # [arxiv:2102.01894 tab. 3]
    H = concat(X * (2 * s)[:, None], X * (2 * e)[:, None])
    # Encode with a three-layer MLP plus position [arxiv:2102.01894 §3.5]
    memory = mlp3(add_position(H))
    # Decode all queries in parallel, six layers [arxiv:2102.01894 app. A.2]
    hidden = decoder6(queries, memory)
    # Heads: segment, foreground, completeness [arxiv:2102.01894 §3.3]
    segments = boundary_head(hidden)
    p_fg, p_c = fg_head(hidden), completeness_head(hidden)
    # Score is the mean of the two confidences; keep every query
    # [arxiv:2102.01894 §3.5]
    return segments, (p_fg + p_c) / 2
```

```pseudocode
# Training assignment, after the strict stage [arxiv:2102.01894 §3.4]
def assign(predictions, truths, strict):
    # Hungarian matching on the cost of eq. 1 [arxiv:2102.01894 eq. 1]
    pairs = hungarian(cost(predictions, truths))
    # Relaxed stage: add extra positives [arxiv:2102.01894 §3.4]
    return pairs + relaxed_positives(predictions, truths, pairs)
```

## Contradictions

- **Feature source of Table 1.** Section 4.2 says the main experiments use
  I3D features [arxiv:2102.01894 §4.2]. The footnote to the RTD-Net row in
  Table 1 says its results use P-GCN I3D features [arxiv:2102.01894 tab. 1].
- **Timing.** The text gives 0.114 s per sample [arxiv:2102.01894 §4.2]. The
  runtime breakdown for a three-minute video in Table H lists 49.29 ms for
  boundary prediction and re-weighting, 0.32 ms for the encoder, 8.97 ms for
  the decoder and 0.89 ms for the heads [arxiv:2102.01894 tab. H]. The paper
  does not reconcile the two.
- **Paper and released code.** The card for the paper lists where the released
  code differs from the text: matcher weights, loss forms, the stage that
  trains the boundary head, and checkpoint selection on the THUMOS14 test
  split. See the card's Paper vs code section before reimplementing from the
  text. This is not a claim the paper makes about itself.

## Papers

- [Relaxed Transformer Decoders for Direct Action Proposal Generation](/papers/2102.01894):
  a DETR-style decoder that outputs temporal segments directly, with boundary
  re-weighting, relaxed matching and a completeness head; no NMS.