wikillm

Paper · 2020-11-01 · cs.CV, cs.AI, cs.CL, cs.LG

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

COOT encodes video and text at clip/sentence and video/paragraph levels with transformers, pools frames and words with attention-aware aggregation, and aligns the levels with contrastive losses plus a cycle-consistency loss. Author-reported: 60.8 R@1 for paragraph-to-video retrieval on ActivityNet-captions val1, against 44.4 for HSE.

Cross-modal retrievalContrastive learningCross-attentionCycle consistencyvideo-text-retrievalhierarchical-transformercycle-consistency

arXivPDFCodeMarkdown18 min read

AI-compiled by claude-haiku-5-5 on 2026-11-12, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

A video is made of frames, and it can be cut into clips. A paragraph is made of sentences, and each sentence is made of words. Matching a video to its text means matching all of these levels, and what a clip means depends on the scene around it.

COOT builds one model for video and one for text, and both work at each level. Within a clip, the model learns which frames matter and pools them; within a sentence, it does the same for the words. A second transformer lets each clip or sentence look at the whole video or paragraph, so the local detail is read in context. Training pulls matched pieces together at every level, and adds a check that a clip and a sentence lead back to each other.

TL;DR

COOT is a video-text representation model that treats both videos and paragraphs as hierarchies: frames to clips to video, and words to sentences to paragraph §1 §2. It combines attention-aware pooling inside each clip and sentence, a contextual transformer that links local and global context, and contrastive alignment at three levels with a cycle-consistency loss between clips and sentences §2 §3. The authors report gains on ActivityNet-captions and YouCook2 retrieval. Some of the paper's own numbers disagree with each other, and the official code differs from the paper in ways that change scale (see Pitfalls and Paper vs code).

Problem

Video-text tasks involve several granularities, such as frames and words, clips and sentences, or videos and paragraphs. The introduction argues that long-range temporal context, such as a multi-step action like making tea, is needed to capture a whole video. The authors build on hierarchical alignment losses from earlier work (HSE) and add modelling of interactions within a level and between local and global context §1. The authors also argue that [CLS] tokens, average pooling and max pooling treat each feature independently and miss the relationships between features §2.2.

Method

The model has two symmetric branches, one for video and one for text. Each branch has three parts §2 fig. 1:

  1. A temporal transformer (self-attention) over the features of each clip or sentence §2.
  2. Attention-aware feature aggregation (Attention-FA) that turns the features of a clip into one clip embedding, and of a sentence into one sentence embedding §2.2 eq. 4.
  3. A contextual transformer (CoT) that, for each video and paragraph, combines local clip or sentence embeddings with a global context that comes from all frames or all words §2.3 eq. 5 fig. 2.

Training combines alignment losses on three levels with a clustering term and a cross-modal cycle-consistency loss §2 §3 eq. 9.

The video branch, as one pass §2 fig. 1:

Pseudocode
# Video branch, clip level and global context  [§2] [fig. 1]
# Input: frames (F, d_in); clip_index: list of frame ids per clip
# Output: clip_emb (C, 384); global_ctx (384,)

def encode_video(frames, clip_index):
    # Temporal transformer over the frames of each clip  [§2]
    clip_hidden = [temporal_transformer(frames[idx]) for idx in clip_index]
    # Attention-aware pooling of each clip's features  [§2.2] [eq. 4]
    clip_emb = stack([attention_pool(h) for h in clip_hidden])  # (C, 384)
    # Global context: same encoder and pooling over all frames  [§2.3]
    global_ctx = attention_pool(temporal_transformer(frames))  # (384,)
    return clip_emb, global_ctx

Attention-aware pooling §2.2 eq. 4. The paper writes one head with a per-element attention vector. With the input features X of one clip:

A = \mathrm{softmax}(W_2\,Q + b_2)^\top, \qquad Q = \mathrm{GELU}(W_1 K^\top + b_1), \qquad K = X eq. 4

The pooled output is \hat{x} = \sum_i a_i \odot x_i, where a_i is the attention vector of the i-th feature and \odot is element-wise multiplication §2.2. The paper says this differs from standard attention because the key is reused as the query and only two learned maps are used §2.2.

Contextual transformer §2.3 eq. 5 fig. 2:

Pseudocode
# Contextual transformer over clip embeddings  [§2.3] [fig. 2]
# Input: clip_emb (C, 384); global_ctx (384,)
# Output: video embedding (768,)

def contextual_transformer(clip_emb, global_ctx):
    # Local self-attention, with positional encoding added  [§2.3]
    local = self_attention(clip_emb + sincos_position(clip_emb))  # (C, 384)
    # Cross-attention: query from global context, keys and values local  [§2.3]
    context = cross_attention(global_ctx, local, local)  # (384,)
    # Video embedding: mean of local, concatenated with context  [§2.3]
    return concat(mean(local, axis=0), context)  # (768,)

H_{attn} = \mathrm{softmax}\!\left(\frac{Q K^\top}{\sqrt{d}}\right) V, \qquad Q = \mathcal{W}_q\, g_v,\; K = \mathcal{W}_k \{h_i\},\; V = \mathcal{W}_v \{h_i\} eq. 5

H_{attn} is passed through a feed-forward layer to give H_{context}, and the video embedding is \mathrm{concat}(\mathrm{mean}(\{h_i\}), H_{context}) §2.3. The paper uses the same design for paragraphs §2.

Contrastive losses. Eq. 1 is the margin loss for one positive pair against two negatives, with cosine distance D §2. Eq. 2 applies it to all in-batch negatives on three levels: video and paragraph, clip and sentence, and the two global contexts §2 §4.

L(\mathcal{P},\mathcal{N},\alpha) = \max(0,\alpha + D(x,y) - D(x',y)) + \max(0,\alpha + D(x,y) - D(x,y')) eq. 1

with D(x,y) = 1 - x^\top y / (\lVert x\rVert \lVert y\rVert) eq. 1.

\ell^{L}_{align} = \sum_{k,\,i,\,k'\neq k,\,i'\neq i} L\big((\vartheta^k_i,\delta^k_i),\ \{(\vartheta^k_i,\delta^{k'}_{i'}),(\vartheta^{k'}_{i'},\delta^k_i)\},\beta\big) eq. 2

\ell^{H}_{align} = \sum_{k,\,k'\neq k} L\big((\vartheta^k,\delta^k),\ \{(\vartheta^k,\delta^{k'}),(\vartheta^{k'},\delta^k)\},\alpha\big) eq. 2

\ell^{g}_{align} = \sum_{k,\,k'\neq k} L\big((g_v^k,g_p^k),\ \{(g_v^k,g_p^{k'}),(g_v^{k'},g_p^k)\},\alpha_g\big) eq. 2

Clustering term §2 eq. 3. The paper writes it with (1,1) pairs and margins \gamma and \eta; it pushes apart negative pairs within each modality and does not change positive pairs.

\ell_{cluster} = \sum_{k,\,i,\,k'\neq k,\,i'\neq i} L\big((1,1),\ \{(\vartheta^k_i,\vartheta^{k'}_{i'}),(\delta^{k'}_{i'},\delta^k_i)\},\gamma\big) + \sum_{k,\,k'\neq k} L\big((1,1),\ \{(\vartheta^k,\vartheta^{k'}),(\delta^{k'},\delta^k)\},\eta\big) eq. 3

The sketch below implements the eq. 1–2 margin loss. Pairs are L2-normalised first §4 app. A.1:

Pseudocode
# Hinge loss over all in-batch pairs, both directions  [§2] [eq. 1] [eq. 2]
# left, right: (B, d) L2-normalised; row i of each is a matched pair
# Output: scalar

def contrastive(left, right, margin):
    # Cosine similarity of every left item with every right item  [§2]
    scores = left @ right.T  # (B, B)
    positive = diagonal(scores)  # (B,) matched pair scores
    # Hinge on every non-matching pair, both directions  [eq. 1]
    cost_right = relu(margin + scores - positive[:, None])  # (B, B)
    cost_left = relu(margin + scores - positive[None, :])  # (B, B)
    # Matched pairs are not negatives  [§4]
    off_diagonal = 1 - eye(B)  # (B, B)
    return sum((cost_right + cost_left) * off_diagonal)

The full training loss §2 §3 eq. 9:

\ell_{final} = \ell^{L}_{align} + \ell^{H}_{align} + \ell^{g}_{align} + \ell_{cluster} + \lambda\,\ell_{CMC} eq. 9

Cross-modal cycle consistency §3 fig. 3. For a sentence embedding \delta_i, take its soft nearest neighbour among the clip embeddings, cycle back to the sentence sequence to get a soft position \mu, and penalise \lVert i - \mu\rVert^2 §3.

\bar{\vartheta}_{\delta_i} = \sum_{j=1}^{n} \alpha_j \vartheta_j, \qquad \alpha_j = \frac{\exp(-\lVert \delta_i - \vartheta_j\rVert^2)}{\sum_{k=1}^{n} \exp(-\lVert \delta_i - \vartheta_k\rVert^2)} eq. 6

\mu = \sum_{j=1}^{m} \beta_j\, j, \qquad \beta_j = \frac{\exp(-\lVert \bar{\vartheta} - \delta_j\rVert^2)}{\sum_{k=1}^{m} \exp(-\lVert \bar{\vartheta} - \delta_k\rVert^2)} eq. 7

\ell_{CMC} = \lVert i - \mu\rVert^2 eq. 8

The paper applies the cycle in both directions, from text to video and from video to text, and sums them §3. Pseudocode:

Pseudocode
# Cycle consistency, one direction  [§3] [eq. 6] [eq. 7] [eq. 8]
# seq: (N, d) items to cycle; other: (M, d) items of the other modality

def cycle_loss(seq, other):
    # Soft nearest neighbour of each item in the other sequence  [eq. 6]
    weights = softmax(-sq_dist(seq, other), axis=1)  # (N, M)
    nearest = weights @ other  # (N, d)
    # Cycle back: soft position of each nearest neighbour in seq  [eq. 7]
    back = softmax(-sq_dist(nearest, seq), axis=1)  # (N, N)
    soft_index = back @ arange(N)  # (N,)
    # Penalise the squared distance from the original index  [eq. 8]
    return mean((arange(N) - soft_index) ** 2)

Full loss with the cycle term, in both directions §3 §4:

Pseudocode
# Total loss for one batch  [§2] [§3] [eq. 9]
# lam: 0.01 on ActivityNet, 0.001 on YouCook2  [app. A.1]

def total_loss(video, text, lam):
    # Alignment on three levels, margin 0.2  [§4] [eq. 2]
    loss = contrastive(video.emb, text.emb, margin=0.2)
    loss += contrastive(video.clip, text.sent, margin=0.2)
    loss += contrastive(video.ctx, text.ctx, margin=0.2)
    # Clustering within each modality, high and low level  [§2] [eq. 3]
    loss += contrastive(video.emb, video.emb, margin=0.2)
    loss += contrastive(video.clip, video.clip, margin=0.2)
    # Cycle consistency, starting from clips and from sentences  [§3]
    loss += lam * cycle_loss(video.clip, text.sent)
    loss += lam * cycle_loss(text.sent, video.clip)
    return loss

Note on the clustering lines in the last block: the paper's eq. 3 is written with (1,1) pairs, and the code uses a different form (see Paper vs code). The block shows the structure the paper describes. It also keeps the text-side clustering terms out for brevity; they mirror the video side §2.

Hyperparameters

Values from the paper's hyperparameter table, which lists the final choices for the three best models tab. 7 app. A.1. "Range" is the range the authors searched by manual search and BOHB app. A.1.

HyperparameterConsidered rangeActivityNetYouCook2 ResNet/ResNeXtYouCook2 HowTo100MWhere
OptimizerAdam, RAdam, SGDAdamRAdamRAdamtab. 7
Learning rate1e-5 to 1e-21e-33.6e-49e-4tab. 7
Weight decay0 to 1e-22e-52e-50tab. 7
Momentum0.5 to 0.990.90.560.56tab. 7
Adam beta20.9 to 0.99990.9990.980.98tab. 7
Adam epsilon1e-10 to 1e-71e-81.5e-91.5e-9tab. 7
Warmup epochs0 to 8300tab. 7
ROP patience2 to 10255tab. 7
ROP cooldown0 to 3333tab. 7
Attention layers1 to 3111tab. 7
Attention dimension256 to 1024384384384tab. 7
Attention heads1 to 8888tab. 7
Attention FF dimension1x to 2x1x1x1xtab. 7
AF dimension1x to 2x2x2x2xtab. 7
AF heads1 to 8222tab. 7
Number of AF modules1 to 2111tab. 7
Dropout0% to 10%2.5%1%5%tab. 7
Gaussian noise on frame features0 to 100.010tab. 7

The "1x" and "2x" entries are multiples of the attention dimension tab. 7. ROP is the reduce-on-plateau scheduler app. A.1. The code uses the same values for the three models anet_coot.yaml yc2_2d3d_coot.yaml yc2_100m_coot.yaml.

Other settings:

  • Batch size: 64 video/paragraph pairs; all clips and sentences of each pair are used; all non-matching in-batch combinations are negatives §4 anet_coot.yaml.
  • Margins: the text lists \alpha = \alpha_g = \beta = \gamma = \mu = 0.2 §4. See Pitfalls for the \eta and \mu mismatch in eq. 3. The code uses 0.2 for all terms anet_coot.yaml.
  • Cycle weight \lambda: 0.01 on ActivityNet, 0.001 on YouCook2 app. A.1 anet_coot.yaml yc2_100m_coot.yaml.
  • Cycle sampling: one clip per video and one sentence per paragraph gave the best results §4. The code draws one random index per sequence loss_fn.py loss_fn.py.
  • Activation: GELU, chosen over ReLU, SELU and ELU app. A.1 anet_coot.yaml.
  • Hidden size: 384, with a pointwise linear layer reducing the input feature dimension §4 anet_coot.yaml.
  • Attention-FA pooler: 2 heads, hidden size 768 tab. 7 anet_coot.yaml.
  • Weight initialisation: truncated normal, standard deviation 0.01, samples beyond 2 standard deviations redrawn app. A.1 anet_coot.yaml utils_torch.py.
  • Learning-rate schedule: reduce on plateau, factor 10, patience and cooldown as in tab. 7 app. A.1 anet_coot.yaml.
  • Stopping: stop after no improvement for 15 epochs app. A.1. The code uses 16 anet_coot.yaml. Max epochs: not stated in the paper; 100 in the code anet_coot.yaml.
  • Early-stop metric: sum of R@1 over both retrieval directions on ActivityNet; on YouCook2, sum of R@1 for clip-sentence and sentence-clip retrieval app. A.1 trainer_retrieval.py.
  • Text features: BERT-base uncased; per-token outputs of the last two layers, 1536-d §4 anet_coot.yaml.
  • Video features: ActivityNet 2048-d at 3.8 FPS, from the HSE work §4 anet_coot.yaml. YouCook2 option A is ResNet-152 plus ResNeXt-101, 4096-d at 3 FPS §4 yc2_2d3d_coot.yaml. YouCook2 option B is a video-text embedding network pretrained on HowTo100M, 512-d at 0.6 FPS §4 yc2_100m_coot.yaml.
  • Frame sampling: up to 80 frames per clip and per video. If there are more, split into 80 equal intervals and sample one frame per interval in training; take the centre frame in validation §4 anet_coot.yaml.
  • ActivityNet preprocessing: expand each clip to at least 10 frames by adding frames at both ends app. A.1 anet_coot.yaml.
  • Retrieval: L2-normalised output embeddings, cosine similarity, best match by highest score app. A.1.
  • Hardware and time: training takes under 3 hours on two GTX 1080 Ti GPUs, not counting data loading §5 (author-reported).

Results

Author-reported. Setup: ActivityNet-captions (about 20k videos, 72k clip-sentence pairs; train, val1 and val2 splits) and YouCook2 (2000 videos, 14k clips, 89 recipe types), with R@K and median rank §4. The ± values are as printed in the paper; the number of runs is not stated tab. 2 tab. 3. Entries marked * are computed by the authors tab. 3 tab. 4 tab. 5.

Table 1: ablation on ActivityNet-captions val1 tab. 1 §5. Pooling is the low-level (frame to clip, word to sentence) pooling. CMC is the cycle loss, and CoT is the contextual transformer. HSE is reproduced by the authors.

ModelPoolingCMCCoTP→V R@1P→V R@5P→V R@50V→P R@1V→P R@5V→P R@50Param (M)
HSEMaxnono45.6 ± 0.376.1 ± 0.796.0 ± 0.344.9 ± 0.575.8 ± 1.295.8 ± 0.426.1
HSEMaxyesno46.6 ± 0.478.1 ± 0.397.3 ± 0.146.4 ± 0.377.6 ± 0.397.1 ± 0.326.1
COOTCLSnono49.4 ± 1.477.7 ± 1.395.7 ± 0.249.7 ± 1.977.8 ± 0.995.8 ± 0.34.9
COOTAVGnono52.6 ± 0.680.6 ± 0.497.0 ± 0.252.1 ± 0.480.8 ± 0.297.0 ± 0.24.9
COOTMaxnono58.2 ± 0.584.9 ± 0.298.1 ± 0.158.7 ± 0.586.0 ± 0.298.2 ± 0.14.9
COOTAFAnono59.0 ± 0.585.4 ± 0.298.2 ± 0.059.8 ± 0.685.8 ± 0.898.2 ± 0.15.8
COOTMaxyesyes59.4 ± 0.986.1 ± 0.698.3 ± 0.060.5 ± 0.187.1 ± 0.298.5 ± 0.16.7
COOTAFAnoyes59.8 ± 1.186.3 ± 0.398.5 ± 0.160.1 ± 0.187.1 ± 0.498.5 ± 0.17.6
COOTAFAyesno59.5 ± 0.585.5 ± 0.498.1 ± 0.060.5 ± 0.786.2 ± 0.598.2 ± 0.15.8
COOTAFAyesyes60.8 ± 0.686.6 ± 0.498.6 ± 0.160.9 ± 0.387.4 ± 0.598.6 ± 0.07.6

Table 2: video-paragraph retrieval on ActivityNet-captions val1 tab. 2 §5. Same video features as HSE, for a fair comparison §5. P→V is paragraph to video; V→P is video to paragraph. MR is median rank.

MethodP→V R@1P→V R@5P→V R@50P→V MRV→P R@1V→P R@5V→P R@50V→P MR
LSTM-YT0.04.024.01020.07.038.098
No Context5.014.032.0787.018.045.056
DENSE14.032.065.03418.036.074.032
VSE11.734.785.710----
FSE18.244.889.1716.743.188.47
HSE44.4 ± 0.576.7 ± 0.397.1 ± 0.1244.2 ± 0.676.7 ± 0.397.0 ± 0.32
COOT60.8 ± 0.686.6 ± 0.498.6 ± 0.1160.9 ± 0.387.4 ± 0.598.6 ± 0.01

The paper's text summarises this as about 16.6% better R@1 than HSE on average §5. See Pitfalls for how that figure is computed.

Table 3: retrieval on YouCook2 tab. 3 §5. Paragraph to video (P→V) and sentence to clip (S→C). Random is the chance baseline. TrainSet names the data the model was trained on. "HowTo100M" rows use features pretrained on that dataset. △ marks the video-text embedding features (option B).

MethodTrainSetP→V R@1P→V R@5P→V R@10P→V MRS→C R@1S→C R@5S→C R@10S→C MR
Random-0.211.092.192290.030.150.31675
Miech et al. 2019HowTo100M43.1*68.6*79.1*2*6.117.324.846
ActBERTHowTo100M----9.626.738.019
MIL-NCEHowTo100M61.9*89.4*98.9*1*15.138.051.210
HGLMMYouCook2----4.614.321.675
Miech et al. 2019YouCook232.3*59.2*70.9*4*4.213.721.565
COOTYouCook250.4 ± 2.679.4 ± 0.687.4 ± 0.81.3 ± 0.65.9 ± 0.716.7 ± 0.624.8 ± 0.849.7 ± 2.9
Miech et al. 2019HowTo100M+YouCook259.6*86.0*93.6*1*8.224.535.324
COOTHowTo100M△+YouCook277.2 ± 1.095.8 ± 0.897.5 ± 0.31.0 ± 0.016.7 ± 0.440.2 ± 0.352.3 ± 0.59.0 ± 0.0

Without HowTo100M pretraining, COOT beats the YouCook2-trained baselines on both tasks §5. With the HowTo100M-pretrained features, the paper reports that it beats MIL-NCE on P→V R@1 §5. See Pitfalls for the size of that gap.

Captioning §5.1 tab. 4 tab. 5. The authors feed COOT clip features (and optionally video features) into the MART captioner, in place of MART's RGB and optical-flow features §5.1. They state that this uses about 100 times fewer features per video §5.1 (author-reported).

Table 4: captioning on YouCook2, val split tab. 4 §5.1. Features: RGB+Flow (MART's original features) or COOT. R@4 is repetition, where lower is better.

FeaturesMethodTrainSetB@3B@4RougeLMETEORCIDEr-DR@4 ↓
RGB+FlowVTransformerYouCook213.08*7.6232.18*15.6532.267.83
RGB+FlowTransformerXLYouCook211.46*6.5630.78*14.7626.356.30
RGB+FlowMARTYouCook212.83*8.0031.97*15.9035.744.39
COOT clipMARTYouCook214.178.6933.0116.1138.288.07
COOT video+clipMARTYouCook215.759.4434.3218.1746.066.30
COOT clipMARTHowTo100M△+YC217.1210.9137.5918.8554.075.11
COOT clipMART w/o recurrenceHowTo100M△+YC217.1610.6937.4319.1854.855.45
COOT clipVTransformerHowTo100M△+YC217.6211.0937.6319.3454.674.57
COOT video+clipVTransformerHowTo100M△+YC217.7911.0537.5119.7955.575.69
COOT video+clipMARTHowTo100M△+YC217.9711.3037.9419.8557.246.69

Table 5: captioning on ActivityNet-captions, ae-test split of MART tab. 5 §5.1.

FeaturesMethodTrainSetB@3B@4RougeLMETEORCIDEr-DR@4 ↓
RGB+FlowVTransformerActivityNet16.27*9.3129.18*15.5421.337.45
RGB+FlowTransformerXLActivityNet16.71*10.2530.53*14.9121.718.79
RGB+FlowMARTActivityNet16.43*9.7830.63*15.5722.165.44
COOT video+clipTransformerXLActivityNet16.9410.5730.9314.7622.0415.85
COOT video+clipVTransformerActivityNet16.8010.4730.3715.7625.9019.14
COOT clipMART w/o recurrenceActivityNet15.419.3728.6615.6122.0512.03
COOT video+clipMART w/o recurrenceActivityNet16.5910.3329.9315.6425.4117.03
COOT clipMARTActivityNet16.5310.2230.6815.9123.985.35
COOT video+clipMARTActivityNet17.4310.8531.4515.9928.196.64

Not reproduced here: the appendix ablations on text encoder, loss combinations, pooling variants and noisy labels app. A.2, and the ActivityNet val2 retrieval table app. A.3. Only the average-pooling choice from app. A.2 appears under Paper vs code. The qualitative examples app. A.4 and captioning details app. A.5 are not reproduced.

Paper vs code

Official code: https://github.com/gingsi/coot-videotext, commit 5b15397e03d61b748bfc9ff16903ed63a6854267, Apache-2.0. The paper's abstract links this repository. The following differences change numbers, so implement the code's version if you want to reproduce the reported results, and the paper's version if you want the stated method.

  • Soft nearest-neighbour distance. The paper's eq. 6 uses the squared L2 distance over features eq. 6. The code computes the mean over the feature dimension, which divides the distance by the dimension (384 here) loss_fn.py loss_fn.py anet_coot.yaml. The softmax temperature is 1 loss_fn.py. The effective temperature is therefore about 384 times the paper's.
  • Normalisation of contrastive losses. The paper's eq. 2 is a sum over all pairs eq. 2. The code divides each margin loss by the batch size squared loss_fn.py. With batch size 64, that is by 4096. The λ values were chosen with the code's scaling, so the paper's λ and the code's λ are not interchangeable.
  • Cycle loss input. The alignment and clustering losses use L2-normalised embeddings trainer_retrieval.py. The cycle loss is fed the un-normalised clip and sentence embeddings trainer_retrieval.py. The paper does not say which is used.
  • Cycle loss averaging. The paper's eq. 8 is a per-item squared distance eq. 8. The code averages over the batch and over positions, and samples one random position per sequence loss_fn.py loss_fn.py.
  • Clustering form. Eq. 3 is written with (1,1) pairs eq. 3. The code computes a within-modality margin loss on the similarity matrix of each modality and averages the video and text terms trainer_retrieval.py. The paper's form and the code's form are not the same formula.
  • Global-context clustering. Off in both configs (weight 0), which matches the appendix note that clustering the global context was not helpful app. A.2 anet_coot.yaml. Latent bug: the branch that handles this term multiplies by weight_low_internal trainer_retrieval.py. It is not reached while the weight is 0.
  • Early stopping. The paper says 15 epochs app. A.1. The code stops when the count of non-improving epochs reaches 16 anet_coot.yaml trainer_base.py.
  • Attention-FA. The paper's eq. 4 has one head, no masking and no dropout §2.2 eq. 4. The code uses multiple heads (2 on ActivityNet), a softmax over time per head and per channel with padded positions set to −∞, and dropout before the activation, after the second linear layer, and on the attention weights poolers.py poolers.py anet_coot.yaml.
  • Global-context concatenation. Matches the paper: the global network applies cross-attention to the clip features, then concatenates the result with the mean of the clip features transformer_legacy.py anet_coot.yaml.
  • Average pooling. The appendix describes a non-standard average, with padding positions in the sum and division by the non-padding length app. A.2. The code's avg_special pooler sums over the sequence and divides by the length, without zeroing padding poolers.py. Whether padding is zero upstream is not checked.
  • Mixed precision. The code trains and validates in fp16 anet_coot.yaml. The paper does not state this.
  • Matching elsewhere. The paper's text, the hyperparameter table and the code agree on the optimiser, learning rate, weight decay, momentum, betas, epsilon, warmup, scheduler, dropout, attention sizes and feature dimensions tab. 7 app. A.1 anet_coot.yaml.

Pitfalls

  • Parameter count disagrees. The text says the model has 10.6M parameters, 60% fewer than HSE §5. Table 1 lists 7.6M for the full model and 26.1M for HSE tab. 1. 10.6M is 59% below 26.1M, and 7.6M is 71% below. Use the table.
  • "16.6% better R@1" is absolute, not relative. HSE's averaged R@1 is 44.3 and COOT's is 60.85, a gap of 16.55 points tab. 2. The relative gain is about 37%.
  • "7.2% average improvement" for Attention-FA does not match the table. The table difference between AFA and AVG without CMC or CoT is 6.4 points for P→V and 7.7 for V→P, a mean of 7.05 points tab. 1 §5.
  • MIL-NCE gap does not match the table. The text says COOT beats MIL-NCE by 16.4% on P→V R@1 §5. In Table 3 the gap is 77.2 − 61.9 = 15.3 points, or about 25% relative tab. 3. Use the table values.
  • Two versions of Table 2 in the source. results.tex contains an \iffalse block with older COOT numbers (61.3 and 60.6 R@1). The active table has 60.8 and 60.9 tab. 2 tab. 1. The active numbers match the ablation's full model.
  • Margin mismatch. The text sets \mu = 0.2, but eq. 3 uses \eta, which is never given a value §4 eq. 3. The code uses 0.2 for all terms anet_coot.yaml.
  • Margin scale. Because the code averages over B^2 pairs, the alignment losses are on a different scale from the paper's summed form. Copy the code's normalisation, or retune λ, if you want the same balance.
  • Temperature. With the paper's squared distance over 384 features, the cycle softmax is much sharper than with the code's mean. An implementation that follows the paper literally will put much more weight on the nearest clip.
  • Features are external. The ActivityNet features come from the HSE work, and YouCook2 option A follows Miech et al. 2019 §4. YouCook2 option B uses a video-text embedding network pretrained on HowTo100M by Miech et al. 2020 §4. None of these extractors is part of this paper. An implementer must obtain the features, and the paper does not describe their extraction in full.
  • Sampling makes results noisy. The cycle term draws random positions, so results vary with the seed loss_fn.py. The number of runs behind the ± values is not stated tab. 2 tab. 3.
  • Hyperparameters were searched per dataset. The authors used manual search plus BOHB, and the three final configurations are listed in tab. 7 app. A.1. The paper does not test them on other data.
  • Dropout is also applied in pooling. The code applies dropout inside the attention pooler, which the paper's equations do not show poolers.py.
  • Frame noise is a regulariser for YouCook2 only. Gaussian noise (0.01) on frame features is used for the ResNet/ResNeXt features, and is zero elsewhere tab. 7 yc2_2d3d_coot.yaml.
  • Training cost. The paper says the full model trains in under 3 hours on two GTX 1080 Ti GPUs, not counting data I/O §5 (author-reported).
  • Retrieval of clips uses intermediate embeddings. For clip-sentence retrieval, use the clip and sentence embeddings before the contextual transformer, not the final video and paragraph embeddings §4.

Lineage

  • Builds on: Zhang, Hu and Sha, "Cross-modal and hierarchical modeling of video and text" (HSE). The alignment losses and the ActivityNet features are taken from that work §2 §4. No arXiv id is recorded here; it is not in this wiki.
  • Compared against: LSTM-YT, No Context, DENSE, VSE, FSE and HSE on ActivityNet tab. 2; MIL-NCE, ActBERT, Miech et al. 2019 and HGLMM on YouCook2 tab. 3; MART, TransformerXL and VTransformer as captioners tab. 4 tab. 5.
  • Superseded by: not stated in the paper.

Source and license

Adapted from “COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning” by Simon Ging, Mohammadreza Zolfaghari, Hamed Pirsiavash, Thomas Brox (arXiv:2011.00597), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/gingsi/coot-videotext at commit 5b15397e03d6 (Apache-2.0). Any quoted code is shown under that license.