End-to-end Temporal Action Detection with Transformer

In plain words

Given a long video, temporal action detection asks which actions happen and when each one starts and ends. Older systems first propose many candidate time spans, score them, and then clean up the overlaps with hand-written rules. That makes them hard to train as one piece.

TadTR treats the problem like a small group of guessers. A fixed number of learned slots each look over the whole video, pick out the context they need, and each propose one action with its start, end and confidence. During training, each slot is paired with at most one true action, so the model learns not to report the same action twice, and no rule is needed to remove duplicates afterwards.

Three changes make a standard Transformer suited to video. Attention looks only at a few nearby points around each guess instead of everywhere. The guessed boundaries are refined in several passes. A separate score head judges how well each guessed span lines up with an action, so the ranking is not left to the class score alone.

TL;DR

TadTR is an end-to-end Transformer for temporal action detection that follows the DETR set-prediction pattern. It needs no anchors, no proposal stage and no non-maximum suppression. Its three additions are temporal deformable attention, segment refinement and an actionness regression head. The authors report state-of-the-art self-contained results on THUMOS14 and HACS Segments with a low computation cost. The code is Apache-2.0 and uses pre-extracted features.

Problem

Temporal action detection (TAD) must output, for every action instance, a label, a start and an end in an untrimmed video §I. Earlier pipelines are multi-stage, two-stage, one-stage or bottom-up, and most rely on NMS, grouping or hand-set anchors §I, fig. 1, §II. These steps are not learnable, which limits flexibility and blocks end-to-end gradient flow §I.

A direct Transformer is not a good fit. Neighbouring frames are very similar and action boundaries are vague, so the detector needs locality awareness. Dense attention is insensitive to small local changes and is costly on long sequences §I, §III-A. The paper trains and tests on pre-extracted features from a frozen video encoder, not end to end through the video network §III, §IV-A. Its own limitations are: it may miss short actions, it misses actions when a video has more true actions than queries, and the scheme is still bound to offline features §IV-D, conclusion.

Method

Section numbers follow the printed IEEE layout (Roman numerals, lettered subsections). Tables and figures are numbered in order of appearance. The paper has no algorithm box, so the pseudocode below is ours. It is written from the paper and the code, and the code differences are listed under Paper vs code.

Input and encoder §III-A. A video is a sequence of features, one per snippet of about 8 frames. A 2048-dimensional I3D feature sequence is projected to C = 256 §III-A, §IV-A. Sinusoidal position embeddings are added [§III-A, eq. 3]. The encoder has L_E layers. Each layer has a temporal deformable self-attention (TDSA) module and a feed-forward network (FFN). A residual connection and layer normalisation follow each sub-layer §III-A.

Temporal deformable attention (TDA) §III-A, eq. 1, eq. 2. For each head m and query q, the module predicts K sampling offsets and K attention logits from the query feature. The offsets are added to a reference point t_q. The memory is sampled by linear interpolation at those positions. Each head's output is a softmax-weighted sum of the samples, and the heads are concatenated and projected. In the encoder the query is the frame itself, so the reference point is the frame's own position §III-A. In the decoder the reference point is the predicted centre of the action being refined §III-A.

Decoder §III-A. The decoder has L_D layers. It takes N_q learned action queries, shared by all videos. Each layer has a self-attention step across the queries, a temporal deformable cross-attention (TDCA) step into the encoder memory, and an FFN. The first reference point is predicted from the query embedding by a linear layer and a sigmoid. The queries are shared across videos, so the network learns the distribution of action locations in the training set §III-A.

Prediction heads and segment refinement §III-A, eq. 4, eq. 5. After each decoder layer, an FFN predicts a class probability for each query and a segment. The segment is a normalised centre t and length d. The refinement is incremental: layer l predicts offsets relative to the segment from layer l−1, passed through a sigmoid and its inverse. The reference point of TDCA at layer l is the refined centre from layer l−1 §III-A.

Actionness regression §III-A. The predicted segment is expanded by a factor ε around its centre. Temporal RoIAlign with T_R bins pools the encoder features inside that span. An FFN maps the pooled feature to a score ĝ. The regression target is the maximum IoU between the predicted segment and all ground-truth actions of the video §III-A. The paper reports this head adds about 6.59 % of the full model's FLOPs §III-A.

Training: matching §III-B, eq. 6, eq. 7. Ground truth is padded with a no-action symbol to N_q targets. A one-to-one assignment minimises the total cost. The cost of a matched pair is a classification term plus a segment term, and the cost is zero for unmatched targets. The classification term is cross-entropy by default and focal loss in the improved setting, and the Hungarian algorithm solves the assignment §III-B, §IV-A. Because each target gets one prediction, duplicate detections are avoided without NMS §III-B. The paper's default is cross-entropy. The code uses only a focal-style term [code: models/matcher.py#L72-L77]. See Paper vs code.

Training: loss §III-B, eq. 8, eq. 9. The segment term is a weighted sum of an IoU loss (1 − IoU) and an L1 loss. The total loss sums, over all queries, the classification loss, the segment loss for matched queries, and the actionness L1 loss. Focal loss is used for the classification term §III-B. Auxiliary decoder-layer losses are used in the code, but the paper does not mention them tadtr.py.

Inference §III-B. Only the last decoder layer is used. A detection's score is the square root of the class probability times the actionness score §III-B. The paper says no NMS is needed. For THUMOS14 with overlapping windows, detections in the overlap regions are merged by NMS §IV-B, §IV-A.

Pseudocode of the forward pass and the decoder's segment update, for the architecture as described and as the code implements it:

# TadTR forward pass for one feature window  [§III-A] [fig. 3]
# Input:  feats (T, 2048) I3D snippet features
# Output: class probs (N_q, C), segments (N_q, 2), actionness (N_q,)

def tadtr_forward(feats):
    # Project features to C = 256  [§III-A]
    # (the code adds a GroupNorm after the projection)
    memory = project(feats)                       # (T, 256)
    # Sinusoidal position added to the encoder input  [§III-A] [eq. 3]
    pos = sine_position(T)                        # (T, 256)
    for layer in encoder_layers:                  # L_E layers  [§III-A]
        memory = tdsa_layer(memory + pos, memory)
    # Learned queries, shared across videos  [§III-A]
    queries = query_embeddings                    # (N_q, 256)
    # First reference centre from the query embedding  [§III-A]
    centre = sigmoid(linear_ir(queries))          # (N_q, 1)
    return decode(queries, centre, memory)

# Decoder with iterative segment refinement  [§III-A] [eq. 4] [eq. 5]
# Input:  queries (N_q, C), reference centre (N_q, 1), memory (T, C)
# Output: per layer: class logits and segments (centre, length)

def decode(queries, centre, memory):
    # Each layer refines the previous layer's segment  [§III-A]
    length = None
    outputs = []
    for layer in decoder_layers:                  # L_D layers  [§III-A]
        # Self-attention across action queries  [§III-A]
        queries = self_attention(queries, queries)
        # TDCA: sample memory around the reference  [§III-A] [eq. 1]
        queries = tdca(queries, reference(centre, length), memory)
        queries = ffn(queries)
        # Offsets from the head  [§III-A]
        delta_c, delta_d = segment_head(queries)
        # Centre update in logit space  [eq. 4]
        centre = sigmoid(delta_c + inverse_sigmoid(centre))
        # Length update: layer 1 has no previous length  [eq. 5]
        if length is None:
            length = sigmoid(delta_d)
        else:
            length = sigmoid(delta_d + inverse_sigmoid(length))
        # The next layer's reference is detached  [code: transformer.py#L299]
        outputs.append((class_head(queries), centre, length))
    return outputs

# Training loss for one video  [§III-B] [eq. 6] [eq. 7] [eq. 8] [eq. 9]
# Input:  predictions from the last decoder layer; ground-truth segments
# Output: scalar loss

def training_loss(preds, gt_segments, gt_labels):
    # Hungarian matching of N_q predictions to ground truth  [eq. 6]
    # Cost = classification + L1 + (1 - IoU) terms  [eq. 7]
    matched = hungarian(cost(preds, gt_segments, gt_labels))
    # Classification over all queries; unmatched queries are no-action  [eq. 9]
    loss_cls = focal_loss(preds.logits, matched.labels)
    # Segment loss on matched queries only  [eq. 8] [eq. 9]
    loss_seg = l1(preds.seg[matched], gt[matched]) + iou_loss(...)
    # Actionness target: maximum IoU with ground truth  [§III-A]
    target_g = max_iou(preds.seg, gt_segments).detach()
    loss_act = l1(preds.actionness, target_g)     # [eq. 9]
    return loss_cls + loss_seg + loss_act

# Inference for one window  [§III-B] [code: models/tadtr.py#L413-L416]
# Input:  last-layer probs (N_q, C), segments (N_q, 2), actionness (N_q,)
# Output: detections (segment, label, score)

def detect(probs, segments, actionness):
    # Paper: score is sqrt(prob * actionness)  [§III-B]
    # Code: score is prob * actionness, with no square root
    scores = probs * actionness[:, None]          # (N_q, C)
    # Keep the top-scoring (query, class) pairs for the window  [code]
    return top_k(scores, segments, k=100)

Hyperparameters

Values are from the paper unless the Where column says code. "not stated" means neither the paper nor the code gives a value.

Results

All numbers are author-reported. Setup: single P100 GPU, batch size 1 for timing §IV-B. "Self-contained" means one network with no external classifier. Star rows use IBIF and focal loss §IV-B, §IV-A.

THUMOS14 test set, mAP at IoU 0.3–0.7 tab. 1. Average mAP and mAP@0.5 are shown. Time is per video; GFLOPs per video.

HACS Segments, validation set tab. 2:

ActivityNet-1.3, validation set tab. 3. The first block is self-contained with two-stream (TS) features. The second block uses an ensemble of action classifiers, with TS, I3D or TSP features.

Ablations on the default setting tab. 4 (average mAP). THUMOS14 uses TadTR-lite, per the text; the table does not give a Table 1 row for this setting.

Encoder choice on HACS tab. 5:

Segment refinement on ActivityNet, focal setting tab. 6:

Depth and query count (HACS unless noted; avg mAP) tab. 7, tab. 8:

Sweeps run on half the schedule (15 epochs, HACS) tab. 9, tab. 10:

Actionness hyperparameters on HACS tab. 11:

Paper vs code

Code: xlliu7/TadTR at commit 983ae14bcec8d764c5fc916700f1a5dade403b92, Apache-2.0. The paper links this repository in its abstract, and the owner is the first author. Shipped configs: only configs/thumos14_i3d2s_tadtr.yml. No ActivityNet or HACS configs, and no pre-extracted features or pretrained weights are in the repository (they are external downloads listed in the README). The code's default test path, with CWF, is what the THUMOS14 numbers depend on. Discrepancies:

• Loss weights. Paper λ_act = 5 §IV-A; code default 4 opts.py. Paper's eq. 9 has no classification weight; code applies 2 opts.py.
• Matching classification term. Paper's default is cross-entropy, with focal loss in the improved setting §III-B, §IV-A. The matcher always uses a focal-style cost and has no cross-entropy path matcher.py. The loss is also focal only tadtr.py. Cross-entropy cannot be reproduced from the code.
• Training schedule. Code defaults are 16 epochs with decay at 14, which is the focal setting for THUMOS14 opts.py opts.py. The paper's default is 30 epochs with decay at 25 §IV-A. The code does not expose a switch.
• LR multipliers. The paper multiplies the learning rate of the attention-weight and sampling-offset projections by 0.1 §IV-A. The code names only sampling_offsets and reference_points opts.py main.py. The attention-weight projection gets the full learning rate.
• Encoder depth and FFN width. The paper uses L_E = 2 §IV-A and C_F = 2048 §III-A. The THUMOS14 config uses 4 encoder layers and 1024 FFN width thumos14_i3d2s_tadtr.yml thumos14_i3d2s_tadtr.yml. The opts default is 2048 opts.py.
• Window stride for training. Paper: windows of 128 with stride 64, so overlap 0.5 §IV-A. Config: slice overlap 0.75, so stride 32 thumos14_i3d2s_tadtr.yml data_utils.py. Test overlap 0.25 matches the paper's stride 96 thumos14_i3d2s_tadtr.yml.
• Instance filtering. The paper's default keeps instances overlapping a window by at least 1 s, and uses IBIF (integrity ≥ 0.75) only in the improved setting §IV-A. The code applies IBIF with threshold 0.75 to every windowed video. I found no length-based path data_utils.py data_utils.py.
• Actionness target. The paper takes the maximum IoU with ground truth "of the video" §III-A. The code concatenates ground-truth segments from all videos in the batch, then takes the maximum over that pool tadtr.py. A query can therefore be matched against another clip's action. Whether this matters depends on the clip-normalised coordinates.
• Actionness details not in the paper. The RoI boxes are detached [code: models/tadtr.py#L142-L143], and so is the target IoU tadtr.py. The RoI head is a three-layer MLP with two ReLUs and a sigmoid [code: models/tadtr.py#L115-L122].
• Score. Paper: score = sqrt(p · g) §III-B. Code: p · g, no square root tadtr.py.
• Position embedding. Paper eq. 3 uses the raw index τ §III-A, eq. 3. Code normalises positions by the valid length to 2π before the sine and cosine position_encoding.py.
• Offset units. The paper says Δt ∈ [0, 1] §III-A, yet its initialisation (k, 0, −k, 0, …) is in feature steps §IV-A. The code uses steps, divided by the sequence length temporal_deform_attn.py temporal_deform_attn.py.
• Cross-attention reference for layers ≥ 2. The paper uses the refined centre as a single reference §III-A. The code keeps the width too and scales the offsets by half the width, divided by K temporal_deform_attn.py.
• Detached references. The reference for the next decoder layer is detached transformer.py. The paper does not state this.
• Frame coordinates. Frame i sits at normalised position (i + 0.5)/T, and sampling uses bilinear grid_sample with align_corners=False transformer.py temporal_deform_attn.py. The paper's X((t + Δt)·T_S) is ambiguous by half a step.
• Input projection. Paper: linear projection to C §III-A. Code: 1×1 convolution to 256, then GroupNorm with 32 groups tadtr.py.
• Level embedding. The code adds a learned level embedding to the position embedding. There is one level transformer.py. The paper does not mention it.
• Query embedding. The learned query is split into a position half and a content half. The position half is added to the queries in self-attention and in cross-attention, and it feeds the initial reference transformer.py transformer.py transformer.py.
• Segment head. A three-layer MLP with last-layer bias −2 on the length term tadtr.py tadtr.py. Not in the paper.
• Auxiliary losses. The code applies the same loss to every decoder layer except the last, with separate weights per layer tadtr.py tadtr.py. The paper's loss has no auxiliary term.
• Evaluation. The code keeps the top 200 detections per video, with no NMS in the default path tad_eval.py. For THUMOS14 it merges detections in the overlap regions with NMS when test overlap is positive tad_eval.py tad_eval.py. The code drops "Ambiguous" THUMOS14 labels and ActivityNet segments of 0.02 or less data_utils.py data_utils.py. None of this is in the paper.
• Unused settings. eos_coef is defined but not read opts.py.
• Actionness and CUDA. The actionness head needs the CUDA RoIAlign extension. With disable_cuda the actionness head is turned off main.py. The CUDA deformable-attention backend is not provided temporal_deform_attn.py.

Pitfalls

• Headline ActivityNet number uses TSP features. The 36.75 mAP figure in the abstract uses TSP features tab. 3. Self-contained results on ActivityNet use two-stream features and reach 28.21 (TadTR) and 29.90 (TadTR*) tab. 3. Use the matching feature set when comparing.
• Star rows are a different setting. The 56.7 THUMOS14 figure is TadTR*, which uses IBIF and focal loss §IV-B. The plain TadTR row, at 49.3, uses the default setting tab. 1. The ablation's THUMOS14 value of 47.92 matches no Table 1 row tab. 4.
• Ablation schedules. Tables 9 and 10 run on a 15-epoch schedule tab. 9 tab. 10. Their numbers are not comparable with the full-schedule headline results.
• Ambiguous "TadTR" in the text. "TadTR achieves 15.1 % higher average mAP than A2Net" uses the starred 56.7, not the plain 49.3 §IV-B, tab. 1. The 155 ms "on THUMOS14" in the introduction is TadTR-lite, not TadTR §I, tab. 1.
• Small text–table mismatches. The text says 9.7 % over AGT at mAP@0.5, but 60.1 − 50.2 = 9.9 §IV-B. The text says +2.03 % for L_D from 2 to 4, but the table gives 30.83 − 28.84 = 1.99 §IV-C, tab. 7. The text says dense attention costs 5.7× as much, but the table gives 564.5 / 100.5 = 5.6× §IV-C, tab. 4. These do not change the conclusions, but use the table values.
• Dense attention is slower and less accurate, not just weaker. The paper reports 5.6× the FLOPs and −23.77 mAP on THUMOS14 §IV-C, tab. 4.
• Query count caps recall. The model misses actions when a video has more true actions than N_q. The authors name this as a limitation §IV-D.
• Short actions are weak. The authors report lower performance on short actions and attribute it to the Transformer's weaker short-range modelling §IV-C, fig. 5 §IV-D.
• Feature dependence. Results depend on pre-extracted I3D or TSN features from frozen encoders §III, §IV-A. The paper names end-to-end training of the video encoder as future work §V.
• Training cost. The paper reports about 10 minutes on THUMOS14, 36 minutes on ActivityNet and 150 minutes on HACS on one P100 §IV-A. Timings are for batch size 1 on that GPU §IV-B.
• Fixed-length features on ActivityNet. The paper resizes ActivityNet and HACS features to 100 snippets by linear interpolation §IV-A. This step is not in the repository, so the pipeline that produced the reported numbers cannot be checked from the code.
• Reproducibility from the repo. The repo has only the THUMOS14 config, and its training schedule and encoder depth differ from the paper's default (see Paper vs code). Use the paper's values for other datasets, and expect to need the features and pretrained weights from the README downloads.

Lineage

• Builds on: DETR, for set prediction with a one-to-one bipartite matching and learned queries. TadTR replaces 2D attention with temporal deformable attention and adds refinement and actionness.
• Builds on: Deformable DETR (arXiv:2010.04159), for deformable attention, reference points and iterative refinement. Not compiled here; the card does not use its content.
• Compared against: G-TAD, which uses dense snippet-level graphs and is the main TAD baseline on HACS and ActivityNet in the tables tab. 2 tab. 3.
• Compared against: concurrent Transformer detectors AGT, RTD-Net and A2Net (not compiled here). §II, tab. 1