Query-Dependent Video Representation for Moment Retrieval and Highlight Detection

In plain words

Video search by sentence has two jobs. Given a text query such as "the dog catches a frisbee", the system must find the time span in a long video where that happens (moment retrieval), and it must rate each short clip for how well it matches the query (highlight detection). Many earlier models looked at the video and the sentence together only in a shallow way, so the sentence had little say in how each clip was described.

QD-DETR makes the sentence matter from the start. Every clip first "reads" the query through a cross-attention step, so its description depends on what was asked. The model is also trained with deliberately mismatched pairs: a real video paired with another video's sentence must get low clip scores. This forces the scores to depend on the actual match rather than on what the video looks like in general.

A learned "saliency token" joins the video clips inside the network and works out its own scoring rule for each video-and-query pair, instead of using one fixed scoring head for everything. The model then predicts the moment span with a detection-style decoder and the clip scores with a ranking loss.

TL;DR

• QD-DETR is a DETR-style transformer for moment retrieval (MR) and highlight detection (HD) from text queries §1 §3.
• Its distinct parts are: cross-attention from video clips to query words at the first encoder layers §3.1 eq. 1; negative video-query pairs trained with a suppression loss §3.2 eq. 2; and a learned saliency token that gives an input-dependent saliency predictor §3.3 eq. 3.
• Author-reported gains on QVHighlights, TVSum and Charades-STA are large, but the paper's split and setup details are partly unclear (see Pitfalls).
• Code is MIT-licensed and public; the code differs from the paper in several loss details (see Paper vs code).

Problem

Moment retrieval asks for a start and end time of the moment that matches a text query. Highlight detection asks for a per-clip relevance score, the saliency score §3. Before this work, transformer models for these tasks either concatenated video and text features into one input, or used the text only to build the decoder query, so the video clip features were not tied to the query §3.1. The paper argues that because the task needs every clip judged against the query, the query must enter the clip representation directly §1 §3.1.

The paper also argues that training with only positive video-query pairs gives too little variety: clips within one video look alike, so the model can rank clips without really using the query §3.2.

Assumptions: the method uses fixed, pre-extracted video and text features (frozen backbones) §3. It expects the query to carry meaningful content. The authors note that noisy or mismatched ground-truth text can make training less effective §5.

Method

Inputs. Video clips v_1..v_L and query words t_1..t_N are produced by frozen encoders. Goal: a span (m_c, m_\sigma) and one saliency score per clip §3.

Step 1: cross-attentive encoder (CATE) §3.1 eq. 1. The first encoder layers are cross-attention layers. Queries come from the video clips, keys and values from the query words. Each clip is a weighted sum of word values, with weights from a softmax over the words only. The result is passed through an MLP and merged into the clip representation as in a standard transformer layer. The output is X = \{x_v^1..x_v^L\}, the query-dependent video tokens §3.1.

Step 2: negative pairs §3.2 eq. 2. For each training pair, the video is also paired with a query taken from another pair in the batch. The negative pair's saliency scores are pushed toward zero by L_{neg} = -\log(1 - S(x_v^{neg})), where S is the saliency predictor §3.2 eq. 2. The paper says the same video must get different scores for a positive and a negative query, which discourages the model from scoring clips without the query §3.2.

Step 3: input-adaptive saliency predictor §3.3 eq. 3. A learnable saliency token is concatenated with the query-dependent video tokens and passed through the transformer encoder. The saliency token and each video token are projected by separate linear maps w_s and w_v; the saliency score is their scaled dot product, S(x_v^i) = (w_s^T x_s \cdot w_v^T x_v^i)/\sqrt{d} §3.3 eq. 3.

Step 4: decoder §3.4. The decoder queries are 1-D dynamic anchors, each given by a center m_c and a duration m_\sigma. They are refined layer by layer, pool features around the center, and modulate the cross-attention map by duration §3.4. The text query is not fed to the decoder, because the query is already in the video tokens §3.4.

Losses §3.4 eq. 4–8.

• Moment retrieval, L_{mr}: L1 on (m_c, m_\sigma), generalized IoU, and foreground/background cross-entropy §3.4 eq. 4.
• Highlight detection, L_{hl} = \lambda_{margin} L_{margin} + \lambda_{cont} L_{cont} eq. 7.
  • L_{margin}: a hinge that keeps high-rank clips above low-rank ones by a margin \Delta eq. 5. The margin value \Delta is not stated in the paper.
  • L_{cont}: a rank-aware contrastive loss over saliency levels, with temperature \tau eq. 6. Clips above a rank threshold are positives; clips below it and the negative-pair clips are negatives §3.4.
• Total: L_{total} = L_{hl} + L_{mr} + \lambda_{neg} L_{neg} eq. 8.

Pseudocode (forward pass). Paper-level description; the code's exact layers are in Paper vs code.

# QD-DETR forward pass  [§3] [eq. 1] [eq. 3]
# Input:  video (B, L, Dv), query (B, N, Dt)
# Output: saliency (B, L), spans (B, Q, 2), neg_saliency (B, L)

def qd_detr(video, query):
    # Project both inputs to the hidden width d  [§6]
    video_tokens = project_video(video)            # (B, L, d)
    query_tokens = project_query(query)            # (B, N, d)
    # Clips attend to query words (cross-attention) [§3.1] [eq. 1]
    video_tokens = cross_attention(video_tokens, query_tokens)
    # Prepend a learned saliency token; self-attend over clips [§3.3]
    tokens = concat(saliency_token, video_tokens)  # (B, L+1, d)
    tokens = self_attention(tokens)
    saliency_feat = tokens[:, 0]                   # (B, d)
    video_tokens = tokens[:, 1:]                   # (B, L, d)
    # Scaled dot product gives one score per clip [§3.3] [eq. 3]
    saliency = dot(w_v(video_tokens), w_s(saliency_feat)) / sqrt(d)
    # Negative pair: same video, query of the next item in batch [§3.2]
    neg_query = roll(query_tokens, shift=1, dims=batch)
    neg_saliency = score_pair(video, neg_query)    # same path as above
    # Decoder: anchors (center, width) attend to video tokens [§3.4]
    spans = decoder(anchor_queries, video_tokens)  # (B, Q, 2)
    return saliency, spans, neg_saliency

# Training loss for one batch  [§3.2] [§3.4] [eq. 2] [eq. 4–8]
# Weights from [§6]; lambda_neg = 1 as in [eq. 8]

def training_loss(outputs, targets):
    # Moment localization: L1 + gIoU + foreground/background CE [eq. 4]
    loss_mr = moment_loss(outputs.spans, targets.moment)
    # Hinge on score gaps between high- and low-rank clips [eq. 5]
    loss_margin = margin_loss(outputs.saliency, targets.pairs, delta)
    # Rank-aware contrastive loss over saliency levels, tau = 0.5 [eq. 6]
    loss_cont = rank_contrastive(outputs.saliency, targets.levels, tau)
    # Suppress saliency of mismatched pairs, -log(1 - S) [eq. 2]
    loss_neg = mean(-log(1 - outputs.neg_saliency))
    loss_hl = lambda_margin * loss_margin + lambda_cont * loss_cont
    return loss_hl + loss_mr + lambda_neg * loss_neg

Key equations:

\text{Attention}(Q_v, K_t, V_t) = \text{softmax}\left(\frac{Q_v K_t^{T}}{\sqrt{d}}\right) V_t \quad [\text{eq. 1}]

L_{neg} = -\log\left(1 - S(x_v^{neg})\right) \quad [\text{eq. 2}]

S(x_v^i) = \frac{w_s^T x_s \cdot w_v^T x_v^i}{\sqrt{d}} \quad [\text{eq. 3}]

L_{cont} = -\sum_{r=1}^{R} \log \frac{\sum_{x \in X_r^{pos}} \exp(S(x)/\tau)}{\sum_{x \in X_r^{pos} \cup X_r^{neg}} \exp(S(x)/\tau)} \quad [\text{eq. 6}]

L_{total} = L_{hl} + L_{mr} + \lambda_{neg} L_{neg} \quad [\text{eq. 8}]

Hyperparameters

Values stated in the paper unless marked "code". The paper's supplementary training section is cited as §6 and the query-length section as §7 (source order; not checked against a compiled PDF).

Results

Author-reported. Setup: QVHighlights test split via the official evaluation server §4.1; TVSum with per-category training and HIT@1/mAP §4.1–4.2; Charades-STA R1@0.5/0.7 by feature type §4.2.

QVHighlights test tab. 1. QD-DETR is reported as mean ± std over five runs. V = video only; V+A = video + audio. R1 = recall@1 at IoU thresholds; mAP at IoU 0.5, 0.75 and their average; HD mAP and HIT@1 on the "≥ Very Good" saliency level.

The paper's text says QD-DETR beats the baselines on all metrics and that the video-only gain at strict IoU reaches "up to 36%" in R1@0.7 and mAP@0.75 (relative, against Moment-DETR) §4.2. The gain with audio is stated as 11.84% on average for MR against UMT §4.2 tab. 1.

TVSum highlight detection (mAP) tab. 2. Trained per category; Avg is the mean over 10 categories. Categories: VT, VU, GA, MS, PK, PR, FM, BK, BT, DS. Video only unless marked.

The paper reports +4.2% average over the previous SOTA with V+A §4.2; this is relative (86.6 vs UMT's 83.1, a gain of 3.5 points) tab. 2. Video-only: 85.0 vs 73.3 tab. 2.

Charades-STA moment retrieval (R1@0.5 / R1@0.7) tab. 3. Compare only within a feature type; the paper enumerates each method with its backbone for a fair comparison §4.2.

† = video and audio as the source tab. 3.

Ablation on QVHighlights, val split as captioned tab. 4. CATE = cross-attentive encoder; Neg. = negative pairs; ST = saliency token; DAM = dynamic anchor moments in the decoder. Metrics as in tab. 1.

Encoder ablation against extra self-attention layers tab. 5. Compares CATE with a 4-layer self-attention encoder (SATE), on Moment-DETR and QD-DETR, to check whether gains come from more layers.

† = only query features are used as the encoder condition tab. 5. The paper concludes the CATE gain comes from emphasising the query, not from the extra layers §4.3.

Query length §7, table in supplementary. On the validation set split into short (≤ 8 words), medium (9–13) and long (> 13) queries, QD-DETR beats Moment-DETR in every group. Reported mAP gains: MR +36.7 / +28.0 / +26.3% and HD +7.3 / +11.8 / +11.1% for short / medium / long (relative) §7. The paper says robustness is to meaningless words, not to wrong descriptions §7.

Saliency statistics and qualitative results. The authors plot positive vs negative saliency distributions and observe that overlap shrinks as components are added fig. 4 §4.3. Qualitatively, saliency is highest for positive queries, middle for semi-positive, and lowest for negative queries fig. 5 §4.4.

Paper vs code

Official repository: wjun0830/QD-DETR (linked in the abstract and in the supplementary), owner is the first author's account README.md. Commit f8628f79f7c651b586300b142dbe9b85e43857cc, MIT license LICENSE. Searched: links in the paper and abstract; the repo is the one the paper names.

• Cross-attention is multi-head, with residual and FFN. Paper Eq. 1 is a single softmax with \sqrt{d} §3.1 eq. 1. Code uses nn.MultiheadAttention with 8 heads, adds a residual, then LayerNorm and FFN transformer.py transformer.py. Keys and values come from the raw text tokens inside the MHA projections, not from the separate p_k, p_v of the paper.
• Text is dropped after the cross-attention stack. Only the global token and video tokens go to the self-attention encoder and decoder transformer.py. Consistent with the paper's claim that the decoder needs no text §3.4.
• Negative pair construction. Paper says "mix the video and query from different pairs" §3.2. Code rolls the query tensor by one position in the batch, so the negative for item b is the query of item b+1, and the last wraps to the first model.py. A batch of size 1 yields a positive pair labelled negative (see Pitfalls).
• Negative loss form. Paper Eq. 2 is -\log(1 - S(x)) with S the saliency predictor §3.2 eq. 2. Code applies a sigmoid to the saliency logit and sums over clips, then averages over the batch model.py. The paper does not state whether S includes a sigmoid.
• Margin loss scaling. Code multiplies the margin loss by 2 and divides by batch size times number of pairs, with the comment "to keep the loss the same scale" model.py. Paper Eq. 5 has no such factor, and \Delta is not stated; code default is 0.2 config.py.
• Rank-contrastive loss. Paper says the loop runs over R rank values and Eq. 6 sums over r §3.4 eq. 6. Code loops thresholds 1 to 11 (range(1, 12)), divides the total by 12, and uses a batch-mean model.py model.py. The paper does not state R.
• Saliency level source. Code builds per-clip saliency levels by summing the three annotator scores for each relevant clip, with the remaining clips at zero start_end_dataset.py. The compiled paper does not say how the levels are derived.
• Optimizer. Paper says Adam with weight decay 1e-4 §6. Code uses torch.optim.AdamW inference.py.
• Gradient clipping. Code clips the gradient norm to 0.1 config.py train.py. Not in the paper.
• Saliency token. Code adds a learnable, random-init global token and a learnable position embedding model.py; its output is the saliency feature, scored by saliency_proj1 on clips and saliency_proj2 on the token, divided by \sqrt{d} model.py. Matches the paper's Eq. 3 in form.
• Device. The decoder's initial target is created with .cuda() transformer.py, so CPU-only runs fail without edits.
• Hyperparameters that match. QVHighlights: 200 epochs, batch 32, lr 1e-4 §6 config.py. TVSum: 2000 epochs, batch 4, lr 1e-3 §6 train_tvsum.sh train_tvsum.sh. Loss weights 10 / 1 / 4 §6 config.py.
• License line. LICENSE is MIT and reads "Copyright (c) 2022 Anonymous". The README says the code is MIT. Use code_license: MIT.

No discrepancies found in the decoder dimensions or the layer counts.

Pitfalls

• Batch size 1 breaks negatives. The cyclic shift of the query batch maps a single item to itself, so the "negative" pair is a positive. Keep batch size greater than 1 model.py.
• Split of the ablation tables is unclear. Tab. 4's caption says "val split", but its row (a) equals Moment-DETR's test numbers in tab. 1 tab. 1 tab. 4. Row (j) equals the QD-DETR† row in tab. 5 tab. 5. Treat ablation numbers as unverified for split.
• Relative vs absolute gains. The "up to 36%" and "+4.2%" claims are relative §4.2. Absolute gains are smaller: 11.96 R1@0.7 points over Moment-DETR (video), 3.5 average points over UMT (video+audio) tab. 1 tab. 2.
• Charades features differ by row. Compare only within the same backbone (VGG, C3D, SF+CLIP) tab. 3 §4.2. Charades features come from other repositories (UMT, VSLNet, Moment-DETR), so exact preprocessing is not in this repo §6.
• Margin Δ and λ_neg semantics. Δ is not stated in the paper. Code uses 0.2 and folds λ_neg into one weight of 1 config.py config.py.
• Noisy or wrong queries hurt. The paper's own limitation says that mismatched ground-truth text degrades training §5. Check the query quality before training.
• Frozen features. The method is set up on frozen video and text encoders §3; the paper does not evaluate end-to-end training.
• Negative BCE scale. Code sums the negative loss over clips rather than averaging, so its magnitude grows with clip count (up to 75 clips in code defaults) model.py config.py. The paper's Eq. 2 does not specify a normalization.
• Pretraining is not described in the compiled paper. The repo ships a pretraining script pretrain.sh; the paper's results do not mention it, so do not assume it was used.

Lineage

• Builds on: Moment-DETR (named in the paper as the baseline; no arXiv id in the bibliography) §1 §3. Uses the dynamic-anchor decoder idea from DAB-DETR (arXiv:2201.12329) §3.4 [egbib].
• Compared against: UMT (CVPR, per the bibliography; no arXiv id given there), Moment-DETR, XML, CAL, MCN, SL-Module, TCG, VSLNet, and others, per table tab. 1–3.
• Ranking loss adapted from Ranking Info Noise Contrastive Estimation (AAAI 2022) §3.4 [egbib].
• Superseded by: not known from content already in this wiki.