ActionFormer: Localizing Moments of Actions with Transformers

In plain words

Finding an action in a video means answering two questions: what is happening, and when does it start and stop. ActionFormer does not first guess a list of candidate clips. It treats every moment of the video as a possible action. For each moment it estimates which action, if any, is going on and how far away the start and the end are. Overlapping guesses for the same action are then merged into one result.

The model is a Transformer. Each moment can look at its neighbours in time, but only at nearby ones, which keeps the cost manageable on long videos. The model works on features that were extracted from the video beforehand, so the video network itself is not trained here.

The authors report about 71% mAP at a 0.5 overlap threshold on THUMOS14, a standard benchmark. These numbers are theirs, not independently checked.

TL;DR

ActionFormer localizes and labels actions in untrimmed video in a single pass, with no action proposals and no anchor windows. A Transformer with local self-attention builds a multi-scale feature pyramid, and shared 1D convolutional heads predict a class score and start/end distances at every step. The authors report 71.0% mAP at tIoU 0.5 on THUMOS14 (I3D), 35.6% average mAP on ActivityNet-1.3 (I3D) and 23.5% / 21.9% average mAP on EPIC-Kitchens 100 (verb / noun). Use it as a simple, strong baseline on pre-extracted features, and check the hyperparameter differences between paper and code before reproducing.

Problem

Temporal action localization (TAL) asks for the onset, offset and category of every action instance in an untrimmed video §3. Earlier methods usually depend on action proposals or anchor windows, and add complexity in proposal generation, anchor design, losses, architectures and output decoding §1. The authors want a single-stage, anchor-free model that keeps this pipeline minimal and still uses long temporal context §1, §2.

Method

Formulation §3. A video is a sequence of T feature vectors, one per discretized time step. The paper turns TAL into sequence labelling: every step t predicts a vector (p(a_t), d^s_t, d^e_t). p(a_t) gives C binary class probabilities and d^s_t, d^e_t are positive distances from t to the onset and offset, defined only when t lies inside an action. An instance is decoded from the step with the highest class score eq. 2.

a_t = \arg\max\, p(a_t), \qquad s_t = t - d^s_t, \qquad e_t = t + d^e_t eq. 2

Note that the paper's inference paragraph in §3.4 swaps the labels of s_t and e_t; see Pitfalls.

The network is an encoder-decoder. The encoder g maps the input to a feature pyramid Z = {Z¹, …, Z^L}, and the decoder h maps the pyramid to per-step outputs §3. Working on the feature grid rather than absolute time lets the model take videos at different frame rates §3.

Encoder §3.1. The encoder has two parts.

1. A projection of each input feature into D dimensions with a shallow 1D convolutional network with ReLU §3.1 tab. D. Adding position embeddings is optional; the paper removes them by default because they lowered accuracy in its experiments §3.1.
2. A stack of Transformer layers. Each layer applies layer normalization before a multi-head local self-attention block (MSA) and before an MLP block, with a residual connection after each §3.1 eq. 6. The GELU activation is used in the MLP §3.1. The per-channel scaling factors α and ᾱ are learnable §3.1. Attention is restricted to a window of size W around each step, which reduces the cost from O(T²D + D²T) to O(W²TD + D²T) §3.1. The same window size is used on every pyramid level, so W = 19 on a 16x-downsampled level covers 304 steps §3.1.

\bar{Z}^\ell = \alpha^\ell\,\mathrm{MSA}(\mathrm{LN}(Z^{\ell-1})) + Z^{\ell-1}, \qquad \hat{Z}^\ell = \bar{\alpha}^\ell\,\mathrm{MLP}(\mathrm{LN}(\bar{Z}^\ell)) + \bar{Z}^\ell, \qquad Z^\ell = \downarrow(\hat{Z}^\ell) eq. 6

The downsampling operator ↓ is a strided depthwise 1D convolution, with ratio 2 §3.1. The architecture in the appendix has 7 Transformer units: two at stride 1 (the stem) and five with 2x downsampling (the branch) tab. D §3.4. The stem's second unit and each branch unit contribute one pyramid level, giving six levels with strides 1, 2, 4, 8, 16 and 32 tab. D.

Decoder §3.2. The decoder is shared across all pyramid levels.

• Classification head: three 1D convolutions with kernel size 3 and 512 channels, layer normalization on the first two, ReLU, and a sigmoid per class §3.2 tab. D. The paper does not state the bias initialisation; the code sets the prior probability to 0.01 meta_archs.py.
• Regression head: the same design with two outputs (d^s, d^e), and a ReLU at the end so that the distances are positive §3.2 tab. D. The code also multiplies the output by a learnable scale per level before the ReLU [code: libs/modeling/meta_archs.py#L134-L136, #L156].

Each pyramid level has a regression range [min, max] that bounds the distances the level is responsible for §3.2 tab. D.

Design choices §3.3.

• Feature pyramid: 2x downsampling, with the regression range roughly doubling per level §3.3.
• Loss eq. 7. The total loss for a video sums a focal loss over classes (L_cls) and, on positive steps, a DIoU loss on the distances (L_reg). It is divided by the number of positive steps T₊. λ_reg is 1 by default eq. 7. The loss is applied to all pyramid levels and averaged over videos §3.3.

\mathcal{L} = \frac{1}{T_+} \sum_t \left( \mathcal{L}_{cls} + \lambda_{reg}\,\mathbb{1}_{c_t}\,\mathcal{L}_{reg} \right) eq. 7

• Center sampling §3.3. A step is positive only if it lies within an interval around the action's centre whose half-width is α times the stride of its pyramid level, with α = 1.5 §3.3. Center sampling does not change inference.

Training and inference §3.4.

• Training uses Adam with a linear warm-up, which the paper calls important for convergence §3.4. Variable-length inputs are padded or cropped to a fixed maximum length and masked §3.4.
• The paper says the regression range is normalised by the level's stride §3.4. The code instead divides only the regression targets by the stride (see Paper vs code).
• At inference the full sequence is fed in; there are no position embeddings §3.4. Every step decodes one candidate, which is then filtered with Soft-NMS §3.4.

The following is the training objective and per-step target assignment, as stated in the paper and the official code. Each comment line cites its source.

# ActionFormer training step  [§3] [§3.3] [eq. 7]
# Input:  features (B, T, C_in); per video, segments (N, 2) and labels (N)
# Output: scalar loss

def training_step(features, segments, labels):
    # 1. Encode the features into a pyramid of L levels  [§3.1] [tab. D]
    pyramid = encode(features)

    # 2. Predict a class score and two distances at every level and step  [§3.2]
    scores, distances = decode_heads(pyramid)

    # 3. Assign each step a class target and distance target  [§3.3]
    #    (positive if it lies in the centre region of an action)
    #    [code: meta_archs.py#L473-L495]
    targets = assign_targets(segments, labels, pyramid)

    # 4. Focal loss over all levels and classes  [§3.3] [eq. 7]
    cls_loss = focal_loss(scores, targets.classes)

    # 5. DIoU loss on the distances at positive steps only  [§3.3] [eq. 7]
    reg_loss = diou_loss(distances, targets.distances)

    # 6. Normalise by the number of positive steps (T+ in eq. 7)  [eq. 7]
    return (cls_loss + reg_loss) / num_positive_steps

# Decoding one step into an action  [eq. 2] [code: meta_archs.py#L686-L689]
# Input:  step index t (feature grid), distances d_start and d_end, scores
# Output: (start, end, label, confidence) on the feature grid

def decode_step(t, d_start, d_end, scores):
    # 1. Most likely class at this step  [eq. 2]
    label = argmax(scores)
    # 2. Onset is t minus the start distance, offset is t plus the end distance
    #    [eq. 2]
    start = t - d_start
    end = t + d_end
    return start, end, label, scores[label]

Paper vs code (summary). The official code follows the paper's structure (anchor-free, per-step output, pyramid, focal + DIoU loss, Soft-NMS), but differs in several details that matter to an implementer. The full list is in Paper vs code.

Hyperparameters

Values in the middle column are the paper's unless the notes say otherwise. "Not stated" means the paper does not give it; where the code gives a value, it is in the notes.

Results

Author-reported. Setup: I3D or R(2+1)D features, trained on the validation split and evaluated on the test split for THUMOS14 §4.1; ActivityNet-1.3 with the validation split for evaluation §4.2; EPIC-Kitchens 100 with the validation split §4.3. Baselines are those listed in the paper's tables, not the full literature.

Per tIoU on THUMOS14 (I3D), the paper reports 82.1, 77.8, 71.0, 59.4 and 43.9 at 0.3, 0.4, 0.5, 0.6 and 0.7 tab. 1.

The paper also reports a challenge entry on EPIC-Kitchens 100 that combines SlowFast and ViViT features, with 21.36% (validation) and 20.95% (test) average mAP, ranked 2nd §4.3. This is not the main pipeline.

Ablations (THUMOS14, I3D features, average mAP over 0.3–0.7; no score fusion unless noted) tab. 3a, tab. 3b, tab. 3c:

Appendix results: loss weight λ_reg ∈ {0.2, 0.5, 1, 2, 5} gives 65.6, 66.7, 66.8, 66.2 and 65.4 tab. A. Feature stride 8 gives 66.3 and stride 16 gives 61.9, against 66.8 at stride 4 tab. C.

Paper vs code

Official code: github.com/happyharrycn/actionformer_release, commit 61ea7eb9308a568b0cf45e3804830836e30061de, MIT license. The repository is linked from the paper's abstract. Compared against the code at that commit.

• Where downsampling happens. The paper applies ↓ after the MLP, as a strided depthwise convolution on the block output eq. 6 §3.1. The code downsamples inside the attention block, with stride-2 depthwise convolutions on the query, key and value, and downsamples the residual path with a max-pool; the MLP then runs at the reduced length blocks.py blocks.py blocks.py. This changes what each level computes.
• Optimizer and weight decay. The paper uses Adam with weight decay 1e-4 §3.4 app. C. The code default is AdamW config.py, and all three dataset configs set weight decay to 0.05 thumos_i3d.yaml anet_i3d.yaml epic_slowfast_verb.yaml. Biases, normalization weights and the scale parameters are excluded from decay train_utils.py.
• Epochs. Paper: 50 (THUMOS14), 15 (ActivityNet), 30 (EPIC) app. C. Code: 30, 10 and 16 thumos_i3d.yaml anet_i3d.yaml epic_slowfast_verb.yaml. Warm-up is added on top; the total is epochs + 5 train.py.
• ActivityNet settings. The paper uses 160-step inputs, window 11, and 512 channels app. C tab. D §4.2. The code's "updated" config uses 192 steps, window 7 (full attention on the last level), 256 channels and λ_reg = 2.0. The config header says the paper's setting is reproduced by setting max_seq_len to 160 and the window to [11, 11, 11, 11, -1, -1] anet_i3d.yaml anet_i3d.yaml anet_i3d.yaml anet_i3d.yaml anet_i3d.yaml.
• Regression range units and EPIC overrides. The paper's Table D says the ranges are in seconds for THUMOS14 and EPIC-Kitchens 100 tab. D, and §3.4 says the range is normalised by the stride §3.4. The code compares raw distances in feature-grid steps, before any division by stride [code: libs/modeling/meta_archs.py#L501-L506]; only the targets are divided by stride meta_archs.py. The EPIC-Kitchens config uses different ranges, [0,4), [2,8), [4,16), [8,32), [16,64), [32,∞) [code: configs/epic_slowfast_verb.yaml#L19], not the paper's table.
• Regression loss. Eq. 7 and the text name DIoU, but a later sentence says the paper "adopts a differentiable IoU loss" with a GIoU reference §3.3. The code uses 1D DIoU losses.py.
• Loss normaliser. Eq. 7 divides by the positive count T₊ of each video. The code divides by a moving average of positive counts (momentum 0.9) initialised to a fixed value [code: libs/modeling/meta_archs.py#L553-L557, #L571-L572, #L584].
• Center sampling. The paper's interval is [c − αT/T^ℓ, c + αT/T^ℓ] §3.3. The code clips that interval to the action's own boundaries, so a step must be inside the action as well meta_archs.py.
• Target assignment among several actions. Not stated in the paper. The code picks the shortest qualifying action and gives multi-hot labels for actions whose lengths are within 1e-3 meta_archs.py.
• Soft-NMS voting. Not stated in the paper. The code applies box voting only in class-agnostic mode, that is, for ActivityNet. For THUMOS14 and EPIC-Kitchens (multi-class NMS), no voting is applied, even though the THUMOS14 config sets a voting threshold [code: libs/utils/nms.py#L161-L180] thumos_i3d.yaml.
• Time conversion. Not stated in the paper. The code converts a feature-grid position to seconds as (position × stride + half the window length) / fps, clipped to [0, duration] meta_archs.py.

The per-step outputs, the focal and DIoU losses, the 2x pyramid and the center-sampling factor of 1.5 match between paper and code [code: libs/modeling/meta_archs.py#L197-L199] meta_archs.py. The differences above are in how the pyramid is computed and in training and decoding details.

Pitfalls

• Swapped onset and offset in the paper's inference text. Eq. 2 defines s_t = t − d^s and e_t = t + d^e. The inference paragraph in §3.4 writes e_t = t − d^s and s_t = t + d^e §3.4 eq. 2. The code uses the eq. 2 convention: left boundary = t − d^s, right = t + d^e [code: libs/modeling/meta_archs.py#L686-L689]. Following §3.4 literally would give segments whose start is after their end.
• Feature-grid units everywhere. Distances, regression ranges and the duration threshold are in feature-grid steps, not seconds [code: libs/modeling/meta_archs.py#L501-L506] meta_archs.py. Converting to seconds requires the stride, frame count and fps [code: libs/modeling/meta_archs.py#L739-L743].
• Early loss scale depends on an initial constant. Because the normaliser is a moving average started at a fixed value, the first few iterations are scaled by that constant meta_archs.py. Keep the initial value when reproducing; the code's values differ by dataset (100, 250, 200).
• Cost of local attention is not free on GPUs. The local Transformer runs about 2× slower than the conv baseline at similar MACs, because self-attention parallelises poorly and the authors used PyTorch primitives rather than custom kernels §4.4 tab. 3b. Full attention costs 57.8 GMACs against 45.3 for window 19 tab. 3b. Training uses sequences of up to 2304 steps, padded and masked app. C.
• Pre-extracted features and a fixed vocabulary. The authors name these as main limitations; they also need many labelled videos app. E.
• ActivityNet numbers depend on external classifier scores. The reported ActivityNet results combine the model's scores with external classification scores §4.2 app. C. The code runs this via an external score file [code: configs/anet_i3d.yaml#L65]. Numbers without that step are not comparable.
• Paper's internal numbers do not all agree. The text says removing the pyramid costs 19.3% average mAP, but the table gives 66.8 − 47.6 = 19.2 §4.4 tab. 3c. The text on batch-length truncation mentions a 512-step input and a 1.1% drop, but the table has a 576-step row with 65.7 app. A tab. B.
• Run-to-run variance. The authors report up to 0.5% average mAP between machines on THUMOS14, and up to 0.8% on EPIC-Kitchens, with the same seed app. C. Treat differences smaller than that as noise.
• Boundaries are the weak point. The authors see a strong score peak near action centres, which they suspect comes from center sampling, while the boundary regression looks less accurate §4.4. Their failure analysis points to boundary errors and background confusion app. D.

Lineage

• Builds on: the per-step, anchor-free formulation of the single-stage methods A2Net and AFSD §3 §2; feature pyramids and center sampling from single-stage object detectors (FPN-style pyramid, RetinaNet focal loss, FCOS center sampling) §2; local self-attention adapted from an efficient-attention work the paper cites §3.1; Transformers and vision Transformers §2. Named as the paper names them; no arXiv ids are given here.
• Compared against: TAL-MR, G-TAD, BC-GNN, MUSES, TCANet, AFSD, PBRNet, VSGN, P-GCN, and TadTR, the concurrent Transformer for TAL that uses set prediction §4.1 §4.2 §2.
• Superseded by: not stated.