wikillm

Concept · 2 papers · updated 2026-11-15

Cycle consistency

Penalise a round trip that does not return to its start. COOT maps each sentence to a soft nearest clip, maps back to a soft position and penalises the index error, adding about 0.5 to 1.0 R@1 in its ablation. Weakly Supervised Dense Event Captioning uses a caption-to-segment cycle as its core loss, so event timing is learned from captions alone.

consistency-losscross-modalvideo-text-retrievalsequence-alignmentweak-supervisiondense-captioning

Markdown9 min read

AI-compiled overview, updated 2026-11-15, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

The invariant is a round trip: an item sent to a second set of items and back should land on the item it came from. A loss that penalises the distance between the start and the end of the trip can be built from the pairing that already exists. In COOT, it needs no labels beyond the clip-sentence pairing 2011.00597 §3. In Weakly Supervised Dense Event Captioning, the cycle links a caption and a time span, so the model learns from captions without timestamps 1812.03849 §1 1812.03849 §3.1.

The two papers use the cycle for different jobs, so the variants below are different designs, not interchangeable settings. Cards that use a cycle loss for image translation do not list this slug and are not covered here.

Variants

Soft nearest-neighbour cycle on ordered sequences (COOT)

Two ordered sequences describe the same content, here the clips of a video and the sentences of a paragraph. For each sentence embedding, the method takes a softmax-weighted average of the clip embeddings, using negative squared distance as the score. That average is then matched against the sentence sequence with the same kind of softmax, which gives a soft position. The loss is the squared gap between the sentence's own index and that soft position 2011.00597 §3 eq. 6 eq. 7 eq. 8. Card: COOT.

The paper applies the cycle in both directions, starting from clips and from sentences, and sums the two terms 2011.00597 §3. It is one of several terms in the total loss, next to the contrastive alignment losses at three levels 2011.00597 §2 eq. 9.

Reported effect, from the ablation on ActivityNet-captions val1: for paragraph to video R@1, adding the cycle term moved COOT with attention-aware pooling from 59.0 to 59.5 without the contextual transformer, and from 59.8 to 60.8 with it. For HSE, the same term moved R@1 from 45.6 to 46.6 2011.00597 tab. 1. The paper reports ± values as printed and does not state the number of runs, so single differences of this size are not established 2011.00597 tab. 1.

Caption-segment cycle without timestamps (WSDEC)

A sentence localizer l(V, C) gives a time segment for a caption, and a caption generator g(V, S) writes a caption for a segment 1812.03849 §3.1. The two form a cycle in both directions. Reconstructing the caption through a segment gives a caption loss L_c. Reconstructing the segment through a generated caption gives a segment loss L_s. The total is L_c + λ_s L_s 1812.03849 §3.1 eq. 3 eq. 5 eq. 6. Card: Weakly Supervised Dense Event Captioning.

The cycle is the core of training, but not all of it. The anchor classifier is trained separately, against the anchor with the best METEOR score for each caption 1812.03849 §3.2 eq. 16. Training alternates between the cycle losses and the anchor loss in periodic epochs 1812.03849 §4.

Test time does not run the full cycle. Random candidate segments are each passed once through the localizer on their generated caption, S' = l(V, g(V, S)) 1812.03849 §3.1 eq. 4. The paper states that the conditions for this fixed point are not guaranteed for these networks, so the step should be read as a heuristic 1812.03849 §3.1.

Reported results, author-reported on ActivityNet Captions. The model uses no ground-truth segments 1812.03849 §4. For dense captioning, it reaches CIDEr 18.77 and METEOR 6.30. The two supervised baselines score CIDEr 17.29 and 16.08, and METEOR 4.82 and 7.71, so the model is best on CIDEr and below one baseline on METEOR 1812.03849 tab. 1. For sentence localisation with no timestamps, R@1 at IoU 0.3 is 41.98. That is above the supervised CTRL result (28.70) and below the supervised ABLR result (55.67) and the paper's own fully supervised localizer (52.89) 1812.03849 tab. 2. Tables 1 and 2 have no ± values on the card. Over five seeds at N_r of 10 to 20, the standard deviation of CIDEr is at most 0.14 1812.03849 tab. 3.

The table has no row that removes the cycle loss alone. Its "Pretrained only" row is not described on the card as a cycle ablation, so this page does not use it as one 1812.03849 tab. 1.

When to use which

SituationUseWhy
Two ordered sequences that describe the same content, with a known pairing to alignThe soft nearest-neighbour cycle as an extra loss termThe paper's ablation shows a gain with and without its contextual transformer 2011.00597 tab. 1
One of the two sides has no meaningful order or indexNot coveredThe loss is defined on the index of each item in both sequences 2011.00597 §3 eq. 8; the paper tests only clip-sentence and video-paragraph data 2011.00597 §4
Limited budget for tuning, COOT settingStart from a published weight, then sweep itThe paper sets the weight per dataset, 0.01 on ActivityNet and 0.001 on YouCook2, found by manual search and BOHB 2011.00597 app. A.1. A new dataset needs its own sweep; the two values do not transfer by themselves
Comparing COOT runs, or expecting reproducible numbersRepeat runs over several seeds before reading a small gainThe cycle term draws random positions, so results vary with the seed; the number of runs behind the reported ± values is not stated 2011.00597 tab. 1 2011.00597 §4
Dense captioning with only captions per video, no segment boundariesThe caption-segment cycle (WSDEC)The model is trained without timestamps and scores above two supervised baselines on CIDEr 1812.03849 tab. 1
Sentence localisation with no timestamps, as a baselineWSDEC's localizer, with its test-time stepIts unsupervised R@1 at IoU 0.3 is 41.98, below the supervised ABLR result of 55.67 1812.03849 tab. 2
Caption model not pretrained on the whole videoPretrain the caption generator firstThe cycle needs a meaningful caption model to start from; the paper pretrains on the full-range segment (0, 1) 1812.03849 §4
Videos with many overlapping events, or events with no captionExpect the one-to-one assumption to failThe paper's own qualitative results show missed event starts and two or three captions per video 1812.03849 §4
Re-implementing WSDEC from its official repositoryImplement from the paper and the cardThe repository has no licence file, so no code is quoted; see the WSDEC card for the differences found in the code

Implementation sketch

Both variants use the same round-trip idea but different operations. The COOT sketch vectorises its equations over a sequence. Run it once per direction and add the results, with a weight λ.

Pseudocode
# One direction of the cycle  [arxiv:2011.00597 §3] [eq. 6] [eq. 7] [eq. 8]
# seq: (N, d) items to cycle; other: (M, d) items of the other sequence

def cycle_loss(seq, other):
    # Soft nearest neighbour of each item in the other sequence  [eq. 6]
    weights = softmax(-sq_dist(seq, other), axis=1)  # (N, M)
    nearest = weights @ other  # (N, d)
    # Cycle back: soft position of each neighbour within seq  [eq. 7]
    back = softmax(-sq_dist(nearest, seq), axis=1)  # (N, N)
    soft_index = back @ arange(N)  # (N,)
    # Squared gap between the true index and the soft index  [eq. 8]
    return mean((arange(N) - soft_index) ** 2)

# Total: alignment terms plus lam times both directions  [eq. 9]
# loss = align_terms + lam * (cycle_loss(clips, sents)
#                             + cycle_loss(sents, clips))

The WSDEC sketch is one training step. The caption is the only input; the video is held fixed.

Pseudocode
# One training step of the caption-segment cycle
#   [arxiv:1812.03849 §3.1] [eq. 3] [eq. 5] [eq. 6]
# video: (T, d) frame features; caption: tokens of one event, no timestamps

def caption_segment_cycle(video, caption):
    # Localise the caption as a segment (centre, width) in [0, 1]
    #   [§3.2] [eq. 10]
    segment = localizer(video, caption)
    # Perturb the segment with noise before captioning  [§3.1] [eq. 5]
    noisy = segment + noise()  # noise family and scale: see Contradictions
    # Caption the perturbed segment; compare with the original caption
    #   [§3.2] [eq. 3] [eq. 14]
    logits = captioner(video, noisy)
    loss_caption = cross_entropy(logits, caption)
    # Decode, then localise the generated caption  [§3.1] [eq. 5]
    # Gradient flow through decode: not stated [§3.1]
    segment_back = localizer(video, decode(logits))
    # Segment reconstruction loss  [eq. 15]
    loss_segment = squared_error(segment_back, segment)
    # Total for this step  [eq. 6]
    return loss_caption + lambda_s * loss_segment
Pseudocode
# Test-time captioning with one refinement round  [§3.1] [eq. 4] [§4]
# Candidate count N_r = 15 in the paper; overlap rules: see Contradictions

def dense_caption(video, n_rand):
    # Draw random (centre, width) candidates  [§4]
    guesses = uniform(0, 1, shape=(n_rand, 2))
    # One fixed-point step: localise each candidate's caption  [eq. 4]
    refined = localizer(video, [captioner(video, g) for g in guesses])
    # Keep and merge refined segments by the overlap rules
    segments = filter_and_merge(guesses, refined)
    return [(captioner(video, s), s) for s in segments]

Where variants differ:

  • COOT, distance scale. The paper writes the squared L2 distance over the full feature vector eq. 6. The scale of that distance sets how sharp the softmax is, and so how much weight the nearest item gets. See Contradictions.
  • COOT, sampling. The paper's equations treat every item. It reports that one clip per video and one sentence per paragraph gave the best results 2011.00597 §4.
  • COOT, weight. λ is set per dataset 2011.00597 app. A.1.
  • WSDEC, loss weights. The paper sets λ_s to 0.1 and λ_a to 0.1 1812.03849 §4. The code scales the segment term differently; see Contradictions.
  • WSDEC, candidates. The paper uses random candidates and 15 of them 1812.03849 §4. Over N_r of 10, 15 and 20, CIDEr rises from 17.75 to 19.14 1812.03849 tab. 3, so the candidate count is a small lever.

Contradictions

  • COOT, distance scale in the official code. The paper's soft nearest-neighbour weights use the full squared distance eq. 6. The official repository, which the card records, averages over feature dimensions instead of summing. That changes the softmax temperature by about the feature dimension (384 for COOT's hidden size). Both forms are on the card's Paper vs code section, COOT card.
  • COOT, averaging. The paper's per-item loss eq. 8 is averaged over items in the code, over the batch, and over positions with one random index per sequence (see the card). The paper does not state the averaging used.
  • COOT, input normalisation. The contrastive terms use L2-normalised embeddings. The paper does not say whether the cycle term uses normalised embeddings. The official code feeds it un-normalised ones (see the card).
  • COOT, weight scale. The paper's λ values were chosen for the paper's summed contrastive form, and the code's contrastive losses are scaled differently. λ values are therefore not interchangeable between the paper and the code 2011.00597 app. A.1.
  • WSDEC, segment-loss scale. The paper sets λ_s to 0.1 on the squared error of centre and width 1812.03849 §4 eq. 15. The official code multiplies the mean squared error by 1/100 and adds it with weight 1, as the card's Paper vs code section records WSDEC card. Both forms are recorded; this page does not choose one.
  • WSDEC, noise. The paper uses Gaussian noise on the localizer's output, with σ not stated 1812.03849 §3.1 eq. 5. The official code uses uniform noise in (−0.01, 0.01) on centre and width, per the card.
  • WSDEC, test-time overlap rule. The paper says to keep segments with high overlap between a candidate and its refinement 1812.03849 §4. The official code skips a refined segment when its overlap with its guess is above 0.2, so it keeps low-overlap segments, per the card. Both sides are stated; this page does not resolve them.
  • WSDEC, ablation claims. The text says removing either component decreases accuracy 1812.03849 §4. In tab. 1, the no-regression variant beats the final model on BLEU-3 (2.69 against 2.62) and BLEU-4 (1.44 against 1.27), while scoring lower on METEOR, CIDEr and ROUGE-L 1812.03849 tab. 1.
  • WSDEC, table mismatch. The N_r = 15 row of tab. 3 (METEOR 6.29, CIDEr 18.65) does not match the "Ours" row of tab. 1 (6.30, 18.77). The paper does not say why 1812.03849 tab. 1 1812.03849 tab. 3.

Papers

  • COOT: defines the soft nearest-neighbour cycle on clips and sentences, applies it in both directions, and reports it as an extra term in the ablation.
  • Weakly Supervised Dense Event Captioning: trains a caption generator and a sentence localizer as a caption-segment cycle, so dense captioning needs captions but no timestamps.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.