Learning Salient Boundary Feature for Anchor-free Temporal Action Localization

In plain words

Finding an action in a long video means saying when it starts, when it ends and what it is. The usual methods first guess many candidate time windows, then score them. Many of those guesses are wasted, and the method needs settings for the window sizes that have to be tuned by hand.

This paper removes those guesses. At every moment of the video the model asks two simple questions: how far away is the start of the action, and how far away is its end? Nothing is pre-set. The first answers are then sharpened. For each candidate, the model looks at the moments near its start and its end, and keeps the one that looks most like a real boundary. Extra training rules push that look-up to pick real boundaries rather than background.

The point is a lighter model with fewer settings to tune, which is also quicker to run. Results are author-reported.

TL;DR

• AFSD is a purely anchor-free temporal action localizer: no pre-defined windows, and one prediction per moment §1 §3.
• The core new piece is boundary pooling, which takes the maximum over a start or end region of each proposal, not the mean §3.2 eq. 4.
• Two consistency losses (activation-guided learning and boundary contrastive learning) push the pooled boundary features toward real boundaries §3.3.
• Author-reported: 52.0 average mAP on THUMOS14 (I3D) and 34.4 on ActivityNet 1.3 tab. 1.
• Several details (Soft-NMS threshold, region ratios, loss weights, the consistency pass) differ between paper and code. See Paper vs code before implementing.

Problem

Anchor-based methods place many pre-set windows per location; with C anchors and T frames they produce C·T proposals §1. Actionness-guided methods score each frame and enumerate boundary pairs; BSN is cited as producing O(T²) proposals §1. Both need their own classifier or a second stage, and anchor methods are sensitive to the number and size of anchors §1.

The authors' premise is that a well-designed anchor-free model should be competitive with anchor-based ones, as the object-detection literature suggests §1. The open question is how to get boundary features that are accurate enough for each proposal without anchors §1 fig. 2.

Method

Overview §3 fig. 3. A video is encoded by an I3D backbone into a 3D feature. The spatial dimensions are flattened into a 1D sequence, and a temporal feature pyramid is built §3. A basic predictor gives one coarse proposal per location. A saliency-based refinement module then takes each coarse proposal, pools boundary features for it, and outputs refined offsets, refined class scores and a quality score §3 fig. 3 fig. 4.

Basic prediction module §3.1. At each pyramid level, two temporal convolution branches project the features into a localization space and a classification space. A shared convolution outputs the coarse start and end distances and a class score per location §3.1. For location i at level l, the coarse start and end times are eq. 1:

\hat\psi_i = i\,2^{l} - \hat d^{s}_i, \qquad \hat\xi_i = i\,2^{l} + \hat d^{e}_i eq. 1

Every pyramid location gives one proposal, so level l contributes T_l proposals §3.1.

Saliency-based refinement module §3.2 fig. 4.

1. The localization feature f_loc is projected into a start-sensitive feature f^s and an end-sensitive feature f^e, each with a temporal convolution, group normalisation and ReLU §3.2 eq. 2.
2. For proposal k, build a start region and an end region from its width ŵ_k. The paper defines ŵ_k = ψ̂_k − ξ̂_k §3.2, which is negative when ψ̂ is the start; eq. 15 uses ξ̂ − ψ̂ instead. Read it as the positive length (see Pitfalls). The two ratios δ_a and δ_b set how much of the proposal lies outside and inside the boundary §3.2 eq. 3:

T^{k}_{s} = \left[\hat\psi_k - \frac{\hat w_k}{\delta_a},\ \hat\psi_k + \frac{\hat w_k}{\delta_b}\right], \qquad T^{k}_{e} = \left[\hat\xi_k - \frac{\hat w_k}{\delta_b},\ \hat\xi_k + \frac{\hat w_k}{\delta_a}\right] eq. 3

3. Boundary pooling takes, for each channel i, the maximum of f^s (or f^e) over the region §3.2 eq. 4:

\hat f^{s}(k,i) = \max_{j \in T^{k}_{s}} f^{s}(j,i), \qquad \hat f^{e}(k,i) = \max_{j \in T^{k}_{e}} f^{e}(j,i) eq. 4

4. Because the deeper pyramid levels are too short to pool from, a frame-level feature f_frame is added. It is built by upsampling and convolving the bottom pyramid feature, and it is pooled with the same projection and maximum §3.2.
5. The original feature, the pooled boundary features and the frame-level boundary features are concatenated and passed through one temporal convolution §3.2 eq. 5.
6. From this refined feature the module predicts offsets for start and end, and a refined class score §3.2.

Boundary consistency learning §3.3. Boundary pooling does not guarantee that the pooled feature sits on a real boundary, so two losses regularise it.

• Activation-guided learning. The start and end features are squashed with tanh and averaged over channels §3.3 eq. 6. They are compared with binary targets g^s and g^e that mark a neighbourhood of each ground-truth start and end. The neighbourhood size is not stated §3.3 eq. 7. The loss is the sum of two binary cross-entropies §3.3 eq. 8.
• Boundary contrastive learning. An action is cut into two fragments A1 and A2, with a random background clip between them. The end feature of A1 should be close to the start feature of A2, and far from a background boundary feature §3.3. The triplet loss is eq. 9:

\ell_{trip} = \max\left(\lVert f^{e}_{A_1} - f^{s}_{A_2}\rVert^{2} - \lVert f^{e}_{A_1} - f_{Bg}\rVert^{2} + 1,\ 0\right) eq. 9

The background boundary feature f_Bg is either the start or the end one §3.3. A video enters this pool only if it has an action longer than 2·w_min and a background clip of length w_min, where w_min is the shortest action in the video §3.3.

• The consistency loss is the sum ℓ_con = ℓ_act + ℓ_trip §3.3 eq. 10.

Training §3.4.

• Coarse labels: location i is positive for ground-truth j when ψ_j ≤ i ≤ ξ_j §3.4.
• Refined labels: a location is positive if the temporal IoU of its coarse proposal with the ground truth exceeds 0.5 §3.4.
• Losses: the total is eq. 11:

\mathcal{L} = \ell^{C}_{cls} + \lambda\,\ell^{C}_{loc} + \ell^{R}_{cls} + \lambda\,\ell^{R}_{loc} + \gamma\,\ell_{q} eq. 11

Both classification terms are softmax focal losses §3.4 eq. 12. The coarse localization term is a temporal-IoU loss §3.4 eq. 13. The refined localization term is an L1 loss on offsets §3.4 eq. 13.

• Quality loss: the quality η is trained with a binary cross-entropy whose target is the temporal IoU of the refined boundary with the ground truth §3.4 eq. 14. The paper explains why it replaces FCOS centerness: deciding the exact start or end frame is vague for actions §3.4.
• Schedule: each batch first optimises L, then, for the videos that qualify, trains with ℓ_con §3.4.

Inference §3.4 eq. 15. The refined boundaries add a fraction of the coarse width to the coarse boundaries. The class score is the mean of the coarse and refined scores, multiplied by the quality §3.4 eq. 15:

\hat w = \hat\xi - \hat\psi, \quad \tilde\psi = \hat\psi + \tfrac12\hat w\,\Delta\hat\psi, \quad \tilde\xi = \hat\xi + \tfrac12\hat w\,\Delta\hat\xi, \quad \hat y = \tfrac12(\hat y^{C} + \hat y^{R})\,\eta eq. 15

All predictions are then passed to Soft-NMS to suppress duplicates §3.4.

Pseudocode for the training step, rebuilt from the text above (no algorithm box in the paper):

# Training step for one clip, rebuilt from the paper's text  [§3] [§3.4]
# Input:  clip (B, 3, T, H, W); gt segments per clip (start, end, class)
# Output: scalar loss, used for one optimizer step

def training_step(clip, gt_segments):
    # I3D features, then a temporal feature pyramid  [§3]
    pyramid = build_pyramid(i3d_features(clip))
    # Coarse start/end distances and class scores per location  [§3.1]
    coarse = basic_predictor(pyramid)
    # Coarse segment from each location's time and distances  [§3.1] [eq. 1]
    coarse_segments = decode(coarse.dist_start, coarse.dist_end)
    # Start and end regions scaled by each proposal's width  [§3.2] [eq. 3]
    windows = boundary_windows(coarse_segments, delta_a, delta_b)
    # Maximum of the start and end features inside each window  [§3.2] [eq. 4]
    boundary_feats = boundary_max_pool(start_feat, end_feat, windows)
    # Concatenate features, then convolve  [§3.2] [eq. 5]
    refined_feat = conv_1x1(
        concat(pyramid, boundary_feats, frame_boundary_feats)
    )
    # Refined offsets, refined class scores and quality  [§3.2] [§3.4]
    refined = refine_heads(refined_feat)
    # Coarse positives: location inside a gt; refined: tIoU > 0.5  [§3.4]
    labels = assign_labels(coarse_segments, refined, gt_segments)
    # Weighted sum of focal, tIoU, L1 and quality terms  [§3.4] [eq. 11]
    loss = (
        cls_coarse + lam * loc_coarse
        + cls_refined + lam * loc_refined
        + gamma * quality_loss
    )
    return loss

Boundary consistency pseudocode (ℓ_con, applied to a split clip):

# Boundary consistency loss for one split clip  [§3.3] [eq. 6–10]
# Input:  start and end features (T, C) for the split clip; three pooled
#         boundary features: end of A1, start of A2, start of background
# Output: l_con = l_act + l_trip

def consistency_loss(start_feat, end_feat, targets, anchor, pos, neg):
    # Tanh, then channel mean: start/end confidence  [§3.3] [eq. 6]
    start_conf = mean(tanh(start_feat), axis=channels)
    end_conf = mean(tanh(end_feat), axis=channels)
    # Binary cross-entropy against neighbourhood targets  [§3.3] [eq. 7–8]
    l_act = bce(start_conf, targets.start) + bce(end_conf, targets.end)
    # Triplet on pooled features; margin 1, squared distance  [§3.3] [eq. 9]
    l_trip = triplet(anchor, pos, neg, margin=1.0)
    # Consistency loss is the sum of the two terms  [§3.3] [eq. 10]
    return l_act + l_trip

Pseudocode uses the paper's symbols: lam is λ and gamma is γ. The neighbourhood used for targets is not stated in the paper.

Inference pseudocode:

# Inference for one clip: decode, score, suppress  [§3.4] [eq. 15]
# Input:  coarse and refined outputs for every location
# Output: per-class segments with scores

def infer(outputs):
    # Refined boundaries: coarse boundaries shifted by half the width  [§3.4]
    refined_segments = refine(outputs.coarse, outputs.offsets)
    # Mean of coarse and refined class scores, times quality  [§3.4]
    scores = mean(softmax(outputs.coarse_cls), softmax(outputs.refined_cls))
    scores = scores * sigmoid(outputs.quality)
    # Soft-NMS over the segments of each class  [§3.4]
    return soft_nms(refined_segments, scores)

Hyperparameters

Values from the paper are in the first rows. "Code" entries are read from the official repository and are not in the paper.

Results

Author-reported. Setup: THUMOS14 with 200 validation and 212 test videos over 20 classes, and ActivityNet 1.3 with 19,994 videos and 200 classes, split 2:1:1 §4.1. Metric is mean average precision (mAP) at several temporal-IoU thresholds. The baselines are the methods listed in tab. 1, with the backbone each one uses. The paper does not say which of RGB, flow or their average the main-table row uses; §4.1 says test results are averaged over RGB and flow.

Main comparison tab. 1:

Source: tab. 1. Bold is the paper's. On ActivityNet 1.3, GTAN has the best mAP at 0.5 (52.6) and BC-GNN the best at 0.95 (9.4) tab. 1.

Ablations on THUMOS14, RGB model only, mAP at 0.5, 0.6, 0.7 and average over 0.3–0.7 tab. 2:

Source: tab. 2. Panel (c)'s row labels are as printed in the paper; see Pitfalls for how they relate to the equations.

Inference speed on THUMOS14 tab. 3 (frames per second, as reported):

Source: tab. 3. The timing protocol is not stated in the paper.

Paper vs code

Repository: TencentYoutuResearch/ActionDetection-AFSD, linked in the abstract (the paper's own link), commit ed86a0df91e58baa7d78c796ed29cff82b1f3fa6. Licence: Apache-2.0 in LICENSE, with BSD-3, MIT and PSF third-party components listed in the same file. The paper's own code link is the basis for using it. The paper's supplement is not in the arXiv LaTeX source, and the repo ships supplement.pdf, which was not read here. The ActivityNet path (AFSD/anet/) has its own model and loss files, which were not read, so the notes below cover the THUMOS14 path, plus the ActivityNet config values.

1. Clip overlap. Paper: adjacent clips overlap by m frames, with m = 30 in training and 128 in testing §4.1. Code: clip_stride is 30 for training and 128 for test, and offsets advance by the stride thumos14.yaml thumos14.yaml thumos_dataset.py. Overlap is clip length minus stride, so training overlap is 226 frames, not 30. Test overlap of 128 matches.

2. Start and end region ratios. Paper: δ_a = 4 and δ_b = 10 §4.1 eq. 3. Under eq. 3, that gives the inside share (right of the start point) as 1/10 of the width and the outside share as 1/4. The code does the reverse: outside is plen/10 and inside is plen/4 BDNet.py BDNet.py.

3. Region ratio for ℓ_con. Paper: δ_b = 100 for the consistency pass §4.1. Code: the self-supervised branch builds its frame segments with the same plen/4 and plen/10 as the main path BDNet.py. No separate ratio is used.

4. Coarse localization loss. Paper: temporal-IoU loss §3.4 eq. 13. Code: GIoU loss (loss_type='giou') multisegment_loss.py. The temporal-IoU is used for labels, not for this loss.

5. Quality target gradient. Paper: BCE against the temporal IoU of the refined boundary §3.4 eq. 14. Code: the target is computed from the same differentiable predictions and is not detached; the BCE then passes gradient into the target too multisegment_loss.py. The paper does not say whether the target is detached.

6. Refined offset targets. Paper: the offset label Δ is not defined in the text §3.4. Code: the target is the difference between the coarse target and the coarse prediction, divided by half the predicted width multisegment_loss.py.

7. Refined positive threshold. Paper: tIoU above 0.5 §3.4. Code: the threshold is piou, which defaults to 0 config.py. The README passes 0.5 for THUMOS14 README.md and 0.6 for ActivityNet README.md. Running without the flag does not follow the paper.

8. Loss weights. Paper: λ = 10 on THUMOS14, 1 on ActivityNet; γ = 1 §4.1. Code: lw defaults to 10 and cw to 1 config.py; the README passes these values README.md. Consistent for THUMOS14.

9. Consistency pass. Paper: for each batch, the model is first optimised on L, then trained with ℓ_con §3.4. Code: the consistency terms are summed into one cost, one backward pass and one optimizer step train.py train.py. The paper's two-step schedule is not in the code.

10. Activation loss scope. Paper: ℓ_act is part of ℓ_con, which is used for pool videos §3.3 §3.4. Code: the frame-level activation loss is added to the main cost for every clip, and the proposal-level activation losses carry an extra weight of 0.1 train.py train.py. The 0.1 weight is not in the paper.

11. Triplet distance and weight. Paper: squared Euclidean distance, margin 1 §3.3 eq. 9. Code: nn.TripletMarginLoss with default distance (not squared), margin 1 train.py. Weight: the paper gives none; the code multiplies by ssl, default 0.1 config.py train.py, and by per-scale weights of 1, 0.1 and 0.1 train.py.

12. Triplet inputs. Paper: anchor is the end feature of A1, positive the start feature of A2, negative a background boundary feature §3.3 eq. 9. Code: the same roles, from three pyramid scales with scale factors 1, 4 and 4 BDNet.py BDNet.py.

13. Activation targets. Paper: a neighbourhood B(·) of each boundary, size not stated §3.3 eq. 7. Code: each boundary gets a box of width max(length/10, 2) frames centred on it thumos_dataset.py.

14. Eligibility for the pool. Paper: an action longer than 2·w_min and a background of length w_min §3.3. Code: the action must be longer than 2·th and the background window must be longer than th, with one attempt per sample thumos_dataset.py thumos_dataset.py thumos_dataset.py. The th value is the ceiling of the shortest action in the video thumos_dataset.py.

15. Soft-NMS threshold. Paper: tIoU threshold 0.5 (THUMOS14) and 0.85 (ActivityNet) §4.1. Code: the values are passed as σ of the Gaussian weight, and nms_thresh is defined but never used thumos14.yaml test.py test.py segment_utils.py. Soft-NMS in the code uses weights exp(-IoU²/σ) with a score floor of 0.001 segment_utils.py segment_utils.py.

16. Score fusion. Paper: RGB and flow results averaged for locations and class scores §4.1. Code: loc, conf, prop_conf and center are averaged across the two models test.py. Consistent.

17. Final score. Paper: ŷ = ½(ŷ^C + ŷ^R)·η eq. 15. Code: softmax of both class heads averaged, times the sigmoid of the quality head test.py. Consistent.

18. Inference-only steps not in the paper. The code clamps decoded segments to [0, clip length] test.py, keeps classes above 0.01 thumos14.yaml, and keeps at most 5000 segments thumos14.yaml.

19. Start and end projections. Paper: two separate convolutions with group norm and ReLU §3.2 eq. 2. Code: one convolution with twice the channels, split in half BDNet.py BDNet.py. The group norm then runs over 2C channels, not C per branch.

20. Frame-level start and end. Paper: frame-level feature by upsampling and convolutions, then start and end projections §3.2. Code: a 512-channel deconvolution with 256 frames, split into a start half and an end half BDNet.py.

21. Training settings. Adam, learning rate 1e-5, weight decay 1e-3, batch 1, 16 epochs §4.1. Code matches for THUMOS14 thumos14.yaml train.py. The ActivityNet config uses learning rate 1e-4 and weight decay 1e-4 anet.yaml. The paper's implementation paragraph does not separate the two datasets.

22. Pretrained weights. Paper: I3D fine-tuned from Kinetics §4.1. Code: loads models/i3d_models/rgb_imagenet.pt thumos14.yaml BDNet.py. The file name says ImageNet. Which weights it holds was not checked.

23. Batch norm. Code freezes batch-norm layers and their affine parameters in the backbone thumos14.yaml BDNet.py. The paper does not mention this.

Pitfalls

• Three sources disagree on the region ratios. Paper defaults give inside 1/10 and outside 1/4 of the width §4.1 eq. 3. The ablation table's best row is the one labelled δ_a < δ_b tab. 2c; under eq. 3 that means a larger background share. The text says a larger share inside the action is better §4.3. The code uses inside 1/4 and outside 1/10 BDNet.py. The code matches the text, not the stated defaults. Which setting reproduces the reported numbers was not tested.
• Sign of the proposal width. The paper defines ŵ = ψ̂ − ξ̂ §3.2, which is negative for a start ψ̂ before an end ξ̂, yet calls it a length. Eq. 15 uses ξ̂ − ψ̂ eq. 15. The code's width is the sum of the two distances, which is positive BDNet.py. Using the paper's literal sign would flip the regions in eq. 3.
• Soft-NMS threshold is a σ in the code. The paper's "threshold" values are used as the Gaussian σ, and the threshold parameter is never read test.py. An implementer following the paper will suppress differently.
• piou defaults to 0. Run without the README flag and the refined labels change config.py.
• Extra forward passes for the consistency loss. For every clip that qualifies, training runs one more forward and backward on a split clip train.py. This roughly adds a second network pass for those samples. The cost is not in the paper.
• Quality target not detached. Gradients reach the quality target through the regression head multisegment_loss.py. Whether this matters for results is not tested here.
• Triplet distance. Squared distance in the paper, non-squared in the code. The margin of 1 means different things under the two.
• Text and table disagree on the gains. The introduction claims a 3.7-point gain at mAP@0.5, and §4.2 claims 7.7 points at mAP@0.6. The table gives 55.5 − 51.6 = 3.9 at 0.5 (best competitor G-TAD+PGCN) and 43.7 − 38.0 = 5.7 at 0.6 (best competitor BU-TAL) tab. 1. Neither figure is reproduced from the table. Use the table values.
• Fused or RGB-only? The main table's row is not labelled RGB or fused; the ablations are RGB only §4.3. Expect the headline number to need the fusion step, since §4.1 says test results average RGB and flow §4.1.
• Timing. No protocol is given for the FPS table tab. 3. The repository times a forward pass on random 256-frame input with 20 repeats BDNet.py. Whether the paper's numbers used the same method is not stated.
• Zero padding for short clips. Clips shorter than the clip length are zero-padded thumos_dataset.py. The paper does not say how short clips are handled.
• Numbers the paper leaves out. Pyramid depth and width, group-norm groups, focal α and γ, the neighbourhood size for activation targets, the consistency loss weight, and the offset normalisation are all absent from the paper text. Values in Hyperparameters come from the code and are marked as such.

Lineage

• Builds on: anchor-free object detection, mainly FCOS for the centerness-style quality idea and the boundary feature from BorderDet, both cited in §2 and §3.4 §2 §3.4. Feature pyramid and I3D are used as in prior work §3 §4.1.
• Compared against: anchor-based (SSAD, TURN, R-C3D, CBR, TAL, GTAN) and actionness-guided (CDC, SSN, BSN, BMN, DBG, G-TAD, BU-TAL, BC-GNN) methods, plus A2Net and G-TAD+PGCN tab. 1.
• Superseded by: not stated in this wiki.