Deconfounded Video Moment Retrieval with Causal Intervention

In plain words

Searching a video with a sentence, such as "person opens a door", means finding the stretch of the video that matches the sentence. Trained models can learn a shortcut. In the training data, some kinds of sentences usually point at the start of a video and others at the end. A model can learn "open means early" and score well without looking at what happens on screen.

This paper calls the position of the moment in the video a hidden confounder: it links the input to the answer without being the reason for the answer. The proposed method, DCM, tries to take position out of the decision. It splits each candidate stretch into what it shows and where it sits, makes the matching consider every possible position with equal weight, and trains on mismatched videos so that guessing by position is penalised. The authors argue that the usual test split hides this problem, so they also test on videos where the position statistics have been changed on purpose.

TL;DR

• Video moment retrieval (VMR) models can score well by exploiting where target moments usually sit in training videos §1.
• DCM disentangles each candidate's content from its location, then applies backdoor adjustment by attending over the locations of all candidates in the same video §3.1, §3.2, eq. 9.
• It trains with an extra counterfactual loss on query and unrelated-video pairs, and it is added to two matching heads, CMI and TCN §3.3.3, §3.3.4.
• Author-reported gains are largest on the out-of-distribution (OOD) tests, which change the position statistics §4.2.1, tab. 2.
• Code is public; the repository has no LICENSE file, so it is cited, not copied README.md.

Problem

Most VMR models rank candidate moments by a matching score with the query and are trained and tested on IID splits §1, §4.1.3. The authors show that the temporal location of a moment is a confounder: long-tailed annotations, and query words such as "open" or "leave" that correlate with start or end positions, let the model predict location instead of content §1, fig. 1. Two known biased baselines reach strong IID scores from location alone: a frequency prior over locations (Freq-Prior) and a model that sees the query and the position but not the video (Blind-TAN) §4.1.4, tab. 2. The IID test cannot reveal this bias, so the paper adds OOD tests that insert random video at the start of each test video §4.1.3.

Dataset statistics: ActivityNet-Captions (ANet-Cap) 19,207 videos, average video 117.6 s, moment 36.2 ± 40.2 s; Charades-STA 6,672 videos, 30.6 s, moment 8.2 ± 3.6 s; DiDeMo 10,464 videos, 30 s, moment 6.85 ± 3.5 s tab. 1. Charades-STA has no validation split tab. 1.

Method

The paper's method is written for proposal-based VMR. A video is cut into T clips and every ordered span of clips is a candidate moment, so there are T(T+1)/2 candidates §3.3.1. T is 16 for ANet-Cap and Charades-STA and 6 for DiDeMo §4.1.5.

1. Query encoding. Word vectors are 300-d GloVe. A three-layer unidirectional LSTM reads the query, and its last hidden state is q with d = 512 §3.3.2, §4.1.5.
2. Candidate tensor. Clip features are pooled into a d × T × T tensor, one vector per candidate, with a stacked pooling network following the 2D-TAN work the paper cites §3.3.1.
3. Disentangling. Each candidate v is split into content c^v = g_c(v) and location ℓ^v = g_ℓ(v), both described as two fully connected layers §3.1, eq. 1. Two constraints train the split. The reconstruction constraint pulls ℓ^v towards a fixed positional embedding p of the candidate's start and end §3.1. The independence constraint, implemented with distance correlation, pushes ℓ^v away from c^v §3.1, §3.3.4.
4. Intervention target. The paper replaces P(Y | Q, V) with P(Y | do(Q, V)), which by backdoor adjustment equals a sum over locations of P(Y | Q, V, ℓ) P(ℓ), with P(ℓ) fixed to 1/N §3.2, eq. 2. The sum over N candidate locations is approximated by moving the expectation inside the sigmoid (NWGM) §3.2, eq. 4–5.
5. Expected location. The expectation of h_qv(ℓ) is a scaled dot-product attention over all candidate locations of the same video, with queries from q and c^v, keys from ℓ, and values from W2 ℓ §3.3.3, eq. 9. The moment representation becomes v̄ + E[h], where v̄ = c^v + W2 ℓ^v §3.3.3.
6. Matching heads. Two heads are used: CMI, modified from a published fusion module, and TCN, a 2D convolutional head over the candidate tensor §3.3.3, eq. 6–8. For TCN, the expectation is passed through the ReLU using an approximation the paper attributes to a dropout analysis §3.3.3, eq. 8.
7. Scores. The candidate score is s = σ(E_ℓ[f(q, v, ℓ)]) §3.3.3.
8. Loss. Positive pairs use binary cross-entropy against a scaled IoU. Counterfactual pairs, which pair the same query with a different video, use target 0 §3.3.4, eq. 10. The total loss is the sum of both BCE terms plus λ1 times reconstruction and λ2 times independence §3.3.4, eq. 11.
9. Inference. The score uses only the candidates of the test video. The original positional embedding (t_s, t_e) is not used at inference §3.3.3.

The paper has no algorithm box. The pseudocode below follows §3.1 to §3.3.4 and eq. 1 to eq. 11, using the TCN head. The CMI head has no published code (see Paper vs code).

# DCM training loss, TCN head  [§3.1, §3.2, §3.3.3, §3.3.4]
# Input:  query_tokens (B, L, 300); clip_feats (B, T, D); neg_feats (B, T, D)
#         iou_target (B, N); N = T(T+1)/2 candidates
# Output: scalar loss
def dcm_loss(query_tokens, clip_feats, neg_feats, iou_target):
    # Score the positive pair and the counterfactual pair  [§3.3.3]
    query = lstm_last_state(query_tokens)           # (B, d), d = 512
    scores, content, location = score_candidates(query, clip_feats)
    neg_scores, _, _ = score_candidates(query, neg_feats)
    # Positive term: BCE against the scaled IoU  [§3.3.4, eq. 10]
    loss_pos = bce_with_logits(scores, scaled_iou(iou_target))
    # Counterfactual term: same query, another video, target 0  [§3.3.4]
    loss_neg = bce_with_logits(neg_scores, zeros_like(neg_scores))
    # Reconstruct location from the positional embedding of (start, end)
    # [§3.1]
    loss_recon = mean(l2_norm(location - positional_embedding()))
    # Make location independent of content, distance correlation  [§3.1]
    loss_indep = distance_correlation(content, location)
    # Weighted sum of the four terms  [§3.3.4, eq. 11, §4.1.5]
    return loss_pos + loss_neg + 1.0 * loss_recon + 0.001 * loss_indep


# Score every candidate of one video  [§3.3.3, eq. 7, eq. 9]
def score_candidates(query, clip_feats):
    # Candidate tensor: one d-vector per (start, end) pair  [§3.3.1]
    moments = candidate_tensor(clip_feats)          # (B, N, d)
    # Split each candidate into content and location  [§3.1, eq. 1]
    content = g_content(moments)                    # (B, N, d)
    location = g_location(moments)                  # (B, N, d)
    # Attention over the location bank of all N candidates in the video
    # Queries come from the query and the content  [§3.3.3, eq. 9]
    queries = w5(query) + w6(content)               # (B, N, d)
    keys = w7(location)                             # (B, N, d)
    weights = softmax(queries @ keys.T / sqrt(d))   # (B, N, N)
    # Expected location: weighted values scaled by P(l) = 1/N  [§3.2]
    values = w2(location)                           # (B, N, d)
    expected_location = weights @ values / num_candidates
    # Intervened representation: content, location and its expectation
    # [§3.3.3]
    fused = content + values + expected_location    # (B, N, d)
    # TCN matching head over the candidate map  [§3.3.3, eq. 7-8]
    scores = tcn_head(query * fused)                # (B, N) logits
    return scores, content, location


# Inference: rank the candidates of the test video  [§3.3.3]
def predict(query_tokens, clip_feats):
    # The positional embedding of (t_s, t_e) is not used here  [§3.3.3]
    query = lstm_last_state(query_tokens)
    scores, _, _ = score_candidates(query, clip_feats)
    return argmax(sigmoid(scores))                  # top-1 candidate

Key equations, as written in the paper:

Eq. 5, the NWGM approximation that moves the expectation inside the sigmoid:

P(Y|do(Q,V)) := \mathbb{E}_{\ell}[\sigma(f(\mathbf{q},\mathbf{v},\ell))] \approx \sigma\big(\mathbb{E}_{\ell}[f(\mathbf{q},\mathbf{v},\ell)]\big) eq. 5

Eq. 7, the TCN matching function, with the intervened moment representation:

f(\mathbf{q},\mathbf{v},\ell) = \mathbf{w}^T\Big(\phi^*_{\mathbf{V}}\Big(\bar{\mathbf{q}} \odot \big(\bar{\mathbf{v}}+h_{qv}(\ell)\big)\Big)\Big) eq. 7

Eq. 9, the expectation of h_qv over the location bank:

\mathbb{E}_{\ell}[h_{qv}(\ell)] = \sum_{\ell}\big[\mathrm{softmax}(\mathbf{K}^T\mathbf{m}/\sqrt{d}) \odot \mathbf{L}'\big]P(\ell), \quad \mathbf{m}=\mathbf{W}_5\mathbf{q}+\mathbf{W}_6\mathbf{c}^v,\ \mathbf{K}=\mathbf{W}_7\mathbf{L}^T,\ \mathbf{L}'=\mathbf{L}\mathbf{W}_2^T eq. 9

Eq. 11, the full training objective:

\mathcal{L} = \mathcal{L}^{+}_{bce} + \mathcal{L}^{-}_{bce} + \lambda_1\mathcal{L}_{recon} + \lambda_2\mathcal{L}_{indep} eq. 11

Hyperparameters

The "Where" column cites the paper. Where the paper is silent or the code differs, the row says so and cites the code. The Paper vs code section explains each difference.

Results

Author-reported. Setup: three datasets, IID and two OOD test rounds, with the best model on the original test set §4.1.3. R@1 at IoU thresholds, and mIoU, as defined in §4.1.2. Means over runs; the source gives ± std only on mIoU tab. 2.

ANet-Cap and Charades-STA use IoU > 0.5 and 0.7. DiDeMo uses IoU > 0.7 and 1.0 in the same two columns.

ANet-Cap (C3D) and Charades-STA (C3D, I3D, VGG) tab. 2

DiDeMo (flow): columns below are R@1 IoU > 0.7, R@1 IoU > 1.0, mIoU tab. 2.

The text reports new SOTA results for TCN+DCM and CMI+DCM at IoU > 0.7 on IID tests §4.2.2: 27.7 on ANet-Cap, 34.4 / 37.8 / 28.0 on Charades-STA (C3D / I3D / VGG). The SOTA rows are the reported results of other papers §4.1.4.

Ablation, TCN+DCM tab. 3: columns are ANet-Cap IoU 0.5 / 0.7, Charades-STA (I3D) 0.5 / 0.7, DiDeMo 0.7 / 1.0.

The OOD-1 figures for ANet-Cap and Charades-STA show the largest drops when the intervention or the counterfactual loss is removed. The text's relative drops in R@1 IoU > 0.7 for "w/o causal intervention" (15.2%, 19.3%, 4.5%) match these cells §4.3.2.

Figures, described and cited, not reproduced: fig. 1 shows the causal graph and the skewed moment-location histograms on Charades-STA. fig. 2 is the pipeline overview. fig. 3 illustrates the intervention. fig. 4 plots distance correlation per dataset and R@1 across four query subsets with temporal words. fig. 5 plots R@1 by moment length. fig. 6 shows one Charades-STA query before and after the OOD insertion §4.3, §4.3.4.

Paper vs code

Official repository: Xun-Yang/Causal_Video_Moment_Retrieval, linked from the paper's abstract and owned by the first author, at commit f2cf09f6b93fb7b428ad779323c8afe83a7a28c5. The repository has no LICENSE file, so no code is quoted here. The README gives the scripts and cites a 2D-TAN repository for part of the code README.md.

1. Disentangling layers. Paper: g_c and g_ℓ are two fully connected layers §3.1. Code: both are 1×1 convolutions on the candidate map, and the location branch is squashed by tanh tcn_DCM.py tcn_DCM.py.
2. Attention. Paper: one softmax over candidate locations, with the expectation scaled by P(ℓ) = 1/N §3.2, eq. 9. Code: four heads tcn_DCM.py, softmax with dropout 0.1 tcn_DCM.py tcn_DCM.py, and a plain weighted sum with no 1/N factor tcn_DCM.py. With N = 136 candidates for T = 16, the code's attention term is about N times the paper's expression, unless the following layers absorb the scale.
3. Attention query. Paper: m = W5 q + W6 c, with no ReLU §3.3.3, eq. 9. Code: Q(ReLU(q)) + M(ReLU(c)) tcn_DCM.py.
4. Residual and ReLU. Paper: the fused representation is c + W2 ℓ + E[h] §3.3.3. Code: the location value is ReLU(fc(ℓ)) and is added as a residual to the attention output, then ReLU is applied again tcn_DCM.py tcn_DCM.py tcn_DCM.py tcn_DCM.py.
5. Normalisation. Code L2-normalises the query-moment product before the convolutions tcn_DCM.py. Eq. 8 has no such step.
6. Reconstruction target. Paper: p is the positional embedding of the candidate's (t_s, t_e) §3.1. Code: a sine embedding of the clip-index pair, normalised by the number of clips, so the target depends on T position_encoding.py position_encoding.py tcn_DCM.py.
7. Extra regulariser. Code adds 1e-5 times the Frobenius norms of content and location tcn_DCM.py. Eq. 11 has no such term.
8. Distance correlation pooling. Code computes it on all valid candidates of the batch pooled together, not per video tcn_DCM.py tcn_DCM.py. Paper: the loss is averaged over the batch §3.3.4.
9. Reconstruction weight on DiDeMo. Paper: λ1 = 1 §4.1.5. Code: DiDeMo YAML sets 0.5 cfg_6x6_pool_k5l3_didemo.yaml; the DiDeMo script does not override it train_didemo_DCM.sh.
10. Negative sampling. Paper: not stated. Code: a uniformly random other training video, paired with the same query charades.py activitynet.py didemo.py tcn_DCM.py.
11. Learning rate. The YAML values (5e-4, 1e-4, 5e-4) differ from the paper's 1e-4. The scripts override all three to 1e-4, so the paper's value is what runs train_charades_DCM.sh cfg_16x16_pool_k5l8_charades_I3D.yaml.
12. LR schedule. Code builds a MultiStep schedule with milestones, but scheduler.step() is commented out, so the learning rate stays constant train_net.py trainer.py.
13. Epochs. Paper: not stated. Code: 25 (Charades-STA, script; YAML 30), 20 (ANet-Cap), 20 (DiDeMo; YAML 30) train_charades_DCM.sh cfg_6x6_pool_k5l3_didemo.yaml.
14. ANet-Cap clip count. The script uses the 16-clip config, which matches the paper's T = 16. A 32-clip config also exists and is not used by the DCM script train_activity_DCM.sh cfg_32x32_pool_k5l3_activitynet.yaml.
15. Model selection. Paper: best model on the original test set §4.1.3. Code: the checkpoint is chosen on the VAL split by R@1 at IoU 0.5 plus 0.7 (ANet-Cap, Charades-STA) or 0.7 plus 1.0 (DiDeMo) trainer.py. For Charades-STA and DiDeMo, VAL is the test split cfg_16x16_pool_k5l8_charades_I3D.yaml cfg_6x6_pool_k5l3_didemo.yaml.
16. Seeds. Paper: 10 runs (Charades-STA, DiDeMo) and 5 runs (ANet-Cap) tab. 2, note 1. Code: the scripts loop over seed 0 only, and the multi-seed loops are commented out train_charades_DCM.sh train_activity_DCM.sh train_didemo_DCM.sh.
17. CMI head. Paper: CMI+DCM is reported in tab. 2 §4.2.1 and its kernel is 3×3 §4.1.4. Code: no CMI model exists. The registry lists TCN_BASE, TCN_DCM and TCN_BLIND only [code: framework/modeling/init.py#L1-L8]. CMI+DCM results cannot be checked against code.
18. Unused predictor depth. The configs set NUM_STACK_LAYERS to 4 or 8, but the TCN head is fixed at three convolutions and the key is never read defaults.py tcn_DCM.py.
19. OOD units for DiDeMo. Paper: ρ in seconds, {15, 30} §4.1.3. Code: sets named _3 and _6, with the padding in clips paths_catalog.py didemo.py. The clip length that maps clips to seconds is not stated in the code.
20. Random OOD features. Paper: normal distribution §4.1.3. Code: fixed arrays in data/, with no generation script in the repository charades.py activitynet.py didemo.py.
21. Non-maximum suppression. Paper: not stated. Code: NMS runs before ranking, keeping at most five predictions eval_test.py. Ranking, and so R@1, is computed after NMS.
22. mIoU. The code takes the mean of the three largest IoUs in overlap[0] eval_test.py. overlap[0] holds one value for the top prediction, so this equals top-1 IoU, which matches the paper's definition §4.1.2. The slicing is a no-op and is not a discrepancy.

Pitfalls

• Text and table disagree in places. The text reports a 27.8% DiDeMo result at IoU > 0.7 §4.2.2. In tab. 2, 27.8 sits in the IoU > 1.0 column for CMI+DCM, and TCN+DCM has 27.6 there tab. 2. The text's ANet-Cap relative gains for IoU > 0.5, 2.82% IID and 23.68% OOD §4.2.1, do not reproduce from the table: TCN 43.3 to 44.9 is about 3.7%, and OOD-1 16.4 to 18.2 is about 11.0%.
• Test set used for selection. For Charades-STA and DiDeMo the checkpoint is chosen on the test split, which the paper does not state. Selecting on the test split can inflate the reported numbers for these two datasets.
• OOD sets are not identical across papers. For ANet-Cap, long moments (length ≥ 0.5 of the video, about 4k test queries) are excluded from OOD testing §4.1.3 activitynet.py. Any comparison must apply the same exclusion.
• Pooled distance correlation is expensive. Distance correlation runs over every valid candidate in the batch. For ANet-Cap with B = 64 and T = 16, that is 64 × 136 = 8,704 rows. Each pairwise distance matrix is then 8,704 × 8,704, about 300 MB in float32, and several such matrices are created per call. Memory, not compute, is the likely limit.
• Numerical floor in distance. The pairwise distance adds eps = 1e-9 inside the square root util.py util.py. The diagonal is therefore about 3e-5, not zero, which slightly shifts the centred distance matrices.
• Two forward passes per step. Each training step runs the matching head on the positive pair and again on a negative video, so the head's training cost roughly doubles tcn_DCM.py.
• Missing learning-rate schedule. Paper and YAML files suggest milestones, but the code never steps the scheduler trainer.py. A reimplementation that adds a schedule will not match the reported runs.
• Scale of the expectation. Eq. 9 multiplies the attention sum by P(ℓ) = 1/N, but the code does not. The two differ by a factor of N (136 for T = 16). An implementation that follows eq. 9 literally will have a much smaller attention term than the code, and the residual term is not scaled the same way. Pick one convention and check it against the reported numbers.
• Scaled IoU bounds are undefined in the paper. The paper says "scaled IoU" without bounds. The code maps IoU into [0, 1] with lower bounds of 0.5 or 0.7 and clamps below them loss.py cfg_6x6_pool_k5l3_didemo.yaml.
• Confounder set and inference. Training and inference both attend over the candidates of one video, so the "location bank" is the video's own candidate set, not a dataset-wide one §3.2 tcn_DCM.py tcn_DCM.py.
• Ablation effect is mixed. Removing the location feature leaves IID results close to the full model on Charades-STA (59.8 vs 59.7 at IoU 0.5), and removing the independence loss hardly changes them tab. 3. Gains are concentrated on OOD tests.
• Upstream code. The README says parts of the code come from a 2D-TAN repository README.md. That repository's licence was not checked. This card quotes nothing from the code, so the question does not affect the card, but anyone who copies code from the repository should check it first.

Lineage

• Builds on: the proposal-based matching framework with a 2D candidate map, referred to as 2D-TAN in the code and in the paper §3.3.1, §3.3.3. The paper's arXiv id is not given in the text, so the card does not link it.
• Builds on: backdoor adjustment and do-calculus from causal inference, and the NWGM approximation and distance correlation §3.2, §3.1. Cited by name only.
• Compared against: Freq-Prior and Blind-TAN as bias baselines §4.1.4; SCDM, 2D-TAN, DRN, LGI and VSLNet as SOTA on IID tests §4.1.4 tab. 2.
• Related card: QD-DETR works on the same moment-retrieval task with a different architecture. This paper does not compare with it.
• Superseded by: not known from content already in this wiki.