wikillm

Paper · 2021-06-21 · cs.CV, cs.CL

Interventional Video Grounding with Dual Contrastive Learning

IVG-DCL localizes the start and end of a moment matching a text query. It applies backdoor adjustment, approximating the hidden dataset confounder from caption roles, actions and objects, and adds two mutual-information contrastive losses. Author-reported mIoU: 48.02 on Charades-STA, 28.26 on TACoS, 44.21 on ActivityNet Captions. No model code released.

Visual groundingCausal inferenceContrastive learningMutual information maximizationvideo-groundingtemporal-localizationcausal-inferencecontrastive-learning

arXivPDFCodeMarkdown13 min read

AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Video grounding means finding when, in a video, the thing described by a sentence happens. The answer is a start time and an end time. Models trained on these datasets can take shortcuts. If a certain kind of action usually shows up with certain objects in the training videos, the model learns to match those words rather than checking the video itself.

This paper frames the problem with causal reasoning. It tries to remove the effect of a hidden factor that creates those shortcuts. The factor is not stored in the data, so the method estimates it from the words in the captions: who acts, what they do, and what they act on. On top of this, the model adds a contrastive training signal. It pulls the clips inside the target moment closer to the query than the clips outside it, and it gives the start and end clips a clearer picture of the rest of the video. These extra losses are used only during training. The authors report better scores than earlier methods on three public benchmarks, and those numbers are their own.

TL;DR

IVG-DCL is a text-to-video moment localization model evaluated on Charades-STA, TACoS and ActivityNet Captions. It builds on the VSLBase backbone and adds backdoor adjustment with a confounder approximated from caption roles, actions and objects, plus two mutual-information contrastive losses §2.5 §2.6. Author-reported mIoU is 48.02, 28.26 and 44.21, against VSLNet's 45.15, 24.11 and 43.19 tab. 1 tab. 2 tab. 3. The official repository contains no model code yet, and several sentences in the text disagree with the tables.

Problem

Video grounding returns the start and end time of the moment in an untrimmed video that matches a text query § Introduction. The authors argue that models which learn the likelihood of the answer given the input can absorb spurious correlations between words and video features, because the training data is biased § Introduction. Their example: queries containing "hold" with "people" and "vacuum" are frequent in training, while "fix" with the same objects is rare, so a query about fixing a vacuum can be grounded to a holding moment § Introduction.

The paper says that re-sampling and re-weighting methods were designed for classification and are hard to apply to temporal localization, and that pre-trained encoders can inherit the bias § Introduction. Causal debiasing methods for images approximate the confounder from object classes or pre-trained classifier outputs; the authors say these consider only objects §3.5. The method assumes that training videos have moment annotations and caption texts. The caption texts are needed to build the confounder vocabulary §2.5.3.

Method

Overview §2.1. Two encoders produce contextualized video and query features. Two contrastive modules, VQ-CL and VV-CL, act on these features. A context-query attention fusion module follows. A deconfounded prediction then gives start and end distributions. Four losses are combined: two contrastive losses, and cross-entropy losses for the start and end fig. 2.

Features §2.2. The video is a sequence of T clip features of dimension d_v, extracted with C3D. The query is a sequence of N word features of dimension d_w, from pre-trained GloVe embeddings. The ground-truth times are converted to feature indices as I_s = T × 𝒯^s / 𝒯 and I_e = T × 𝒯^e / 𝒯, where 𝒯 is the video duration §2.2.

Encoders §2.2. Two linear layers project video and query to a common dimension d. Each stream then passes through four convolution layers, one multi-head attention layer and one feed-forward layer. The video and text encoders share parameters §2.2.

Contrastive modules §2.3. Clips inside the target moment are positives and clips outside it are negatives. The mutual-information (MI) estimate is

\mathcal{I}^{vq}_{\theta}(\mathbf{q}, \mathbf{V}') := E_{\mathbf{V}'_a}[sp(\mathcal{C}(\mathbf{q},\mathbf{V}'))] - E_{\mathbf{V}'_b}[sp(\mathcal{C}(\mathbf{q},\mathbf{V}'))] eq. 1

where C is an MI discriminator from R^{d_v} × R^{d_v} to R, and sp(z) = log(1 + e^z) is the softplus eq. 1. Query vector q is the max-pool of the query features fig. 3. The losses are L_vq = −I^vq eq. 2 and L_vv = −I^vv(s) − I^vv(e) eq. 3. The VV terms use the same estimator, computed between the start clip (or end clip) and the video clips §2.3.2. Training maximizes the sum of the three MI terms eq. 4.

Fusion §2.4. Context-query attention (CQA) produces the bi-modal interaction, and one feed-forward layer follows:

\mathbf{X} = FFN(CQA(\mathbf{V}', \mathbf{Q}')) eq. 5

Causal model §2.5. The structured causal model has an unobserved confounder Z (dataset selection bias) that affects video V and query Q, and a mediator D that feeds the start and end predictions S and E fig. 4 §2.5.1. Without intervention, the model learns P(S,E|Q,V) = Σ_z P(S,E|Q,V,z) P(z|Q,V) eq. 6. Backdoor adjustment gives the interventional form below eq. 7.

P(S,E|do(\{Q,V\})) = \sum_z P(\{S,E\} | do(\{X\}), z)\, p(z) eq. 7

Latent confounder approximation §2.5.3. The confounder vocabulary comes from captions. The authors run an Open Information Extraction model (RnnOIE, Stanovsky et al. 2018) on each caption to extract (subject, verb, object) tuples. This yields three vocabulary sets: roles, actions and objects. The prior of each phrase z in a set C is

p(z) = \frac{\#z}{\sum_{i \in C} \#i}, \forall z \in C eq. 8

where #i counts occurrences of phrase i in the dataset eq. 8. A phrase is embedded as z = g(embed(z)), where g is a linear projection that makes dim(z) = dim(x) eq. 9. The deconfounded prediction uses Normalized Weighted Geometric Mean (NWGM), which replaces the sum over z of the per-phrase predictions by one prediction on the weighted average:

P(S,E|do(X=\mathbf{X})) \approx P\Big(S,E \,\Big|\, \sum_z (\mathbf{X} \oplus \mathbf{z})\, p(z)\Big) eq. 10

where ⊕ adds a vector to every column of the matrix eq. 10. Start and end indices I_s and I_e come from this prediction. Each is trained with cross-entropy, L_s and L_e §2.5.3.

Objective §2.6. The total loss is

\mathcal{L} = \alpha \mathcal{L}_{vq} + \beta \mathcal{L}_{vv} + \mathcal{L}_{s} + \mathcal{L}_{e} eq. 11

with α and β weighting the two contrastive losses eq. 11. The contrastive modules (DCL) are ignored at inference, because they need the ground-truth moment to choose positives §2.6.

Training step, in the paper's own terms. The paper has no algorithm box; this sketch follows §2 and eq. 1–eq. 11 and is not a verbatim listing.

Pseudocode
# Inputs: video (T, d_v) clip features, query (N, d_w) word features
# Targets: start_idx and end_idx, clip indices of the moment  [§2.2]
# Output: scalar training loss

def mutual_info(anchor, clips, positive_mask):
    # Estimate MI as the softplus difference over positives and
    # negatives  [§2.3.1] [eq. 1]. C is an unspecified discriminator.
    scores = softplus(discriminator(anchor, clips))
    return mean(scores[positive_mask]) - mean(scores[~positive_mask])

def training_step(video, query, start_idx, end_idx):
    # 1. Project both inputs to dimension d, then encode them with one
    #    shared encoder: four conv layers, attention, feed-forward  [§2.2]
    video_ctx = encoder(linear_video(video))         # (T, d)
    query_ctx = encoder(linear_query(query))         # (N, d)

    # 2. VQ term: query vector against clips; positives are clips
    #    inside the target moment  [§2.3.1] [eq. 2]
    query_vec = max_pool(query_ctx)                  # (d,)
    inside = clips_inside(start_idx, end_idx)        # positive mask
    loss_vq = -mutual_info(query_vec, video_ctx, inside)

    # 3. VV term: start clip and end clip against all clips  [§2.3.2] [eq. 3]
    loss_vv = -(mutual_info(video_ctx[start_idx], video_ctx, inside)
                + mutual_info(video_ctx[end_idx], video_ctx, inside))

    # 4. Context-query attention, then one feed-forward layer  [§2.4] [eq. 5]
    fused = feed_forward(cqa(video_ctx, query_ctx))  # shape not stated

    # 5. Per-phrase confounder prior and embedding  [§2.5.3] [eq. 8] [eq. 9]
    for phrase in confounder_vocabulary:
        prior[phrase] = count(phrase) / total_count(phrase_set)
        confounder[phrase] = linear_g(embed(phrase))

    # 6. Deconfounded input, NWGM approximation: broadcast-add each
    #    confounder to fused, weight by prior  [§2.5.3] [eq. 10]
    deconfounded = sum(
        (fused + confounder[phrase]) * prior[phrase]
        for phrase in confounder_vocabulary
    )

    # 7. Start and end cross-entropy losses over the T clips  [§2.5.3]
    start_logits, end_logits = span_head(deconfounded)  # head not stated
    loss_s = cross_entropy(start_logits, start_idx)
    loss_e = cross_entropy(end_logits, end_idx)

    # 8. Weighted multi-task loss; alpha and beta are set in [§3.2]  [eq. 11]
    return alpha * loss_vq + beta * loss_vv + loss_s + loss_e

Hyperparameters

The table of values below comes from hyper_param.tex in the paper's source. That file is not \input by the main document, so the values are not in the compiled paper. Entries marked "compiled" are in the text itself.

NameValueWhereNotes
Batch size16hyper_param.tex (not compiled)One value is listed for all three datasets; per-dataset values not stated
OptimizerAdamhyper_param.tex (not compiled)
Learning rate0.0005hyper_param.tex (not compiled)Schedule not stated
"Decay rate"0.01hyper_param.tex (not compiled)Meaning not stated (weight decay or LR decay)
Gradient clipping1hyper_param.tex (not compiled)Norm or value clipping not stated
Word embedding dimension300hyper_param.tex; compiled in §3.2GloVe
Char embedding dimension50hyper_param.tex (not compiled)Character embedding not described in the compiled text
Dropout0.2hyper_param.tex (not compiled)Where applied not stated
Convolution kernel size7hyper_param.tex (not compiled)
Attention heads8hyper_param.tex (not compiled)
Hidden dimension128hyper_param.tex (not compiled)Likely the shared dimension d of §2.2; link not stated
Max video clips128hyper_param.tex (not compiled)Whether longer videos are subsampled is not stated
α (contrastive, VQ)0.1compiled §3.2; tab. 7
β (contrastive, VV)0.01compiled §3.2; tab. 7
Video featuresC3D; not fine-tunedcompiled §2.2 §3.2; tab. 1 caption
Feature dimension d_vnot statedhyper_param.tex (line commented out)
Epochs, iterations, LR schedule, seedsnot stated

Results

Author-reported. Setup: the model is VSLBase re-implemented in PyTorch, with C3D features and GloVe embeddings, and the feature extractor is not fine-tuned §3.2 tab. 1. Metric: "R@n, IoU = μ", the share of queries with at least one correct moment among the top-n §3.2. The paper does not define mIoU.

Dataset sizes §3.1: Charades-STA has 12,408 training and 3,720 test moment annotations, and no validation split is stated. TACoS has 10,146 training, 4,589 validation and 4,083 test instances. ActivityNet Captions has 37,421 training and 17,505 test moments.

Charades-STA tab. 1

ModelIoU=0.3IoU=0.5IoU=0.7mIoU
MAN (ranking)-46.5322.72-
2D-TAN (ranking)-39.7023.31-
DEBUG (regression)54.9537.3917.6936.34
VSLBase (regression, backbone)61.7240.9724.1442.11
VSLNet (regression)64.3047.3130.1945.15
This paper (IVG-DCL)67.6350.2432.8848.02

TACoS tab. 2

ModelIoU=0.1IoU=0.3IoU=0.5IoU=0.7mIoU
2D-TAN (ranking)47.5937.2925.32--
VSLBase (regression, backbone)-23.5920.4016.6520.10
VSLNet (regression)-29.6124.2720.0324.11
This paper (IVG-DCL), tab. 249.3638.8429.0719.0528.26
This paper (IVG-DCL), tab. 550.9938.7928.8918.2528.35

ActivityNet Captions tab. 3

ModelIoU=0.3IoU=0.5IoU=0.7mIoU
2D-TAN (ranking)59.4544.5126.54-
DEBUG (regression)55.9139.72-39.51
VSLBase (regression, backbone)58.1839.5223.2140.56
VSLNet (regression)63.1643.2226.1643.19
This paper (IVG-DCL)63.2243.8427.1044.21

Comparison with other causal models, Charades-STA tab. 4

Causal modelIoU=0.3IoU=0.5IoU=0.7mIoU
FWA60.6740.9922.8541.70
CWA64.3548.6030.3045.63
VCR-CNN65.6247.5828.4745.94
IVG (this paper)67.6350.2432.8848.02

Comparison with other causal models, TACoS tab. 5

Causal modelIoU=0.1IoU=0.3IoU=0.5IoU=0.7mIoU
FWA48.3436.8726.1916.1726.47
CWA49.5936.3727.0717.6227.18
VCR-CNN47.9136.8727.2716.9726.76
IVG (this paper)50.9938.7928.8918.2528.35

Ablation, Charades-STA tab. 6

ModelIoU=0.3IoU=0.5IoU=0.7mIoU
Full model67.6350.2432.8848.02
w/o IVG64.7047.6030.8045.34
w/o QV-CL66.3049.3131.1947.15
w/o VV-CL66.7549.1632.1747.24
w/o DCL (both CL)65.2148.7631.9346.94
w/o IVG + DCL61.7240.9724.1242.11

Sensitivity to {α, β}, Charades-STA tab. 7

{α, β}IoU=0.3IoU=0.5IoU=0.7mIoU
{0.1, 0.01}67.6350.2432.8848.02
{0.5, 0.1}64.4449.9731.4546.09
{0.1, 0.5}63.2344.6827.3744.08
{1.5, 1}64.3344.7827.0444.89
{1, 1}62.9343.4926.0843.65

Headline claims as stated: the method "achieves the new state-of-the-art" on Charades-STA, +3.33 IoU=0.3 and +2.87 mIoU over VSLNet §3.4. It "performs best" on TACoS across IoU=0.1, 0.3, 0.5 and mIoU §3.4. On ActivityNet, the authors claim state-of-the-art "in most settings" §3.4; the table supports this, but 2D-TAN is higher at IoU=0.5 (44.51 vs 43.84) tab. 3. Only the Charades-STA gains are shown in the causal and ablation tables. The paper reports single numbers with no variance or seed information.

Paper vs code

The abstract links https://github.com/nanguoshun/IVG, and the owner is the first author's GitHub handle. The paper links no other repository or project page, so no further search was needed. The repository at commit 857dea6 contains only README.md, which says the PyTorch implementation is "Coming Soon!" There is no model, training code, configuration or LICENSE file. Because no model code is released, the hyperparameters, the discriminator C, the contrastive sets and the span head cannot be checked against code. The abstract's statement "Our code is available on GitHub" is not borne out by the repository at the recorded commit.

Pitfalls

  • The hyperparameter table is not in the compiled paper. hyper_param.tex exists in the source but is not \input, so the values in the Hyperparameters section come from an orphaned file. An implementer should take them as the authors' intended values, not as printed in the paper.
  • Text and tables disagree in several places. Charades mIoU over VSLBase is 6.09 in the text but 48.02 − 42.11 = 5.91 from tab. 1 §3.4. The text says removing IVG costs "4 points on average"; tab. 6 gives about 2.6 averaged over the four metrics. The text says removing both IVG and DCL costs "10 points on average"; the table gives about 7.5. The text says removing DCL costs "about 2 points"; the table gives about 1.5.
  • TACoS "Ours" has two sets of numbers. tab. 2 reports 49.36 / 38.84 / 29.07 / 19.05 / 28.26, and the causal comparison tab. 5 reports 50.99 / 38.79 / 28.89 / 18.25 / 28.35 for the same model. The paper does not say which is final.
  • VSLBase IoU=0.7 differs by source table. It is 24.14 in tab. 1 and 24.12 in the "w/o IVG + DCL" row of tab. 6.
  • The MI estimator is labelled Jensen-Shannon but written differently. §2.3.1 cites a "Jensen-Shannon MI estimator" and writes I = E_pos[sp(C)] − E_neg[sp(C)] eq. 1. The paper does not derive this form, and the sign convention of C is not explained. Because the code is missing, the exact estimator cannot be confirmed.
  • Discriminator C is not specified. §2.3.1 defines C only as a map from R^{d_v} × R^{d_v} to R. Its architecture, any normalization, and whether it is shared between the VQ and VV terms are not stated.
  • Expectations are not estimated. The paper writes E over the positive and negative clips but does not say whether they are averages, sums, or sampled subsets, nor the size of each set.
  • VV positives and negatives are not defined in the text. §2.3.2 says the VV MI is computed "between the start/end boundaries and the other clips". The figure caption calls the features inside the moment, other than start and end, a separate purple set, but does not say whether they are positives. Treat this as ambiguous.
  • The confounder sum is ambiguous. §2.5.3 builds three vocabulary sets (roles, actions, objects), but eq. 10 sums over z ∈ C for one set C. The paper does not say whether the sum runs over one set, all three, or a weighted combination. The embedding size of embed and the output dimension of g beyond dim(z) = dim(x) are not stated.
  • The start/end head is not specified. §2.5.3 applies cross-entropy to start and end, but the layer that turns the deconfounded representation into per-clip logits is not described. Decoding (for example, requiring start ≤ end) is not stated.
  • Index conversion is not specified. I_s = T × 𝒯^s / 𝒯 §2.2 gives a real number. The rounding rule is not stated.
  • Notation is inconsistent. Q' is written as R^{N×d_v} and V' as R^{T×d_v} §2.2, but the text says both are projected to dimension d. The loss names switch between L_vq (equations eq. 2 eq. 11) and L_qv (figure caption fig. 3), and between VQ-CL and QV-CL tab. 6. The qualitative section names the model "ICV-DCL" §3.8.
  • Section numbering follows the LaTeX source. The source comments out \section{Introduction}, so the introduction is unnumbered in the compiled paper, and the numbers in this card count Related Work as §1. I did not check against a compiled PDF, since none is in the source directory.
  • Causal baselines are the authors' adaptations. FWA, CWA and VCR-CNN were adapted to video by the authors. CWA's "class" is the averaged word embedding of object phrases §3.5. These baselines are not the original image methods, so their numbers should not be read as those methods' published results.
  • Cost of the contrastive terms. They need the ground-truth moment at training time to label positives and negatives, and they are dropped at inference §2.6. The paper does not report their training-time cost.
  • Features are fixed. The paper states that the C3D feature extractor is not fine-tuned tab. 1, so results depend on the pre-extracted features.

Lineage

  • Builds on: VSLBase and VSLNet (cited as zhang2020span) for the backbone and span-prediction framing §2.2 §3.2. Deep InfoMax (hjelm2018learning) for the MI estimator §2.3.1. Context-query attention (seo2016bidirectional) for the fusion module §2.4. Backdoor adjustment and the SCM (pearl2016causal) for the causal model §2.5. NWGM (Xu et al. 2015) for the deconfounded approximation §2.5.3. Image causal debiasing (yue2020interventional) for the general idea § Introduction.
  • Compared against: ranking methods (2D-TAN, MAN, CTRL and others), regression methods (DEBUG, DRN, VSLBase, VSLNet), and reinforcement-learning methods (SM-RL, RWM, TripNet) §3.3 tab. 1 tab. 2 tab. 3. Causal baselines FWA, CWA and VCR-CNN §3.5.
  • Superseded by: not stated, and no later wiki content is known.

Source and license

Adapted from “Interventional Video Grounding with Dual Contrastive Learning” by Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, Wei Lu (arXiv:2106.11013), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/nanguoshun/IVG at commit 857dea683bc5 (none (no LICENSE file in repository)). Any quoted code is shown under that license.