---
kind: concept
slug: cycle-consistency
title: Cycle consistency
summary: >-
  Penalise a round trip that does not return to its start. COOT maps each sentence to a soft nearest clip, maps back to a soft position and penalises the index error, adding about 0.5 to 1.0 R@1 in its ablation. Weakly Supervised Dense Event Captioning uses a caption-to-segment cycle as its core loss, so event timing is learned from captions alone.
tags: [consistency-loss, cross-modal, video-text-retrieval, sequence-alignment, weak-supervision, dense-captioning]
papers: ["2011.00597", "1812.03849"]
related: [cross-modal-retrieval, contrastive-learning, cross-attention, consistency-regularization, weakly-supervised-learning]
status: draft
updated_at: 2026-11-15
---

## Overview

The invariant is a round trip: an item sent to a second set of items and back
should land on the item it came from. A loss that penalises the distance
between the start and the end of the trip can be built from the pairing that
already exists. In COOT, it needs no labels beyond the clip-sentence pairing
[arxiv:2011.00597 §3]. In Weakly Supervised Dense Event Captioning, the cycle
links a caption and a time span, so the model learns from captions without
timestamps [arxiv:1812.03849 §1] [arxiv:1812.03849 §3.1].

The two papers use the cycle for different jobs, so the variants below are
different designs, not interchangeable settings. Cards that use a cycle loss
for image translation do not list this slug and are not covered here.

## Variants

### Soft nearest-neighbour cycle on ordered sequences (COOT)

Two ordered sequences describe the same content, here the clips of a video and
the sentences of a paragraph. For each sentence embedding, the method takes a
softmax-weighted average of the clip embeddings, using negative squared
distance as the score. That average is then matched against the sentence
sequence with the same kind of softmax, which gives a soft position. The loss
is the squared gap between the sentence's own index and that soft position
[arxiv:2011.00597 §3] [eq. 6] [eq. 7] [eq. 8]. Card: [COOT](/papers/2011.00597).

The paper applies the cycle in both directions, starting from clips and from
sentences, and sums the two terms [arxiv:2011.00597 §3]. It is one of several
terms in the total loss, next to the contrastive alignment losses at three
levels [arxiv:2011.00597 §2] [eq. 9].

Reported effect, from the ablation on ActivityNet-captions val1: for paragraph
to video R@1, adding the cycle term moved COOT with attention-aware pooling from
59.0 to 59.5 without the contextual transformer, and from 59.8 to 60.8 with it.
For HSE, the same term moved R@1 from 45.6 to 46.6 [arxiv:2011.00597 tab. 1].
The paper reports ± values as printed and does not state the number of runs,
so single differences of this size are not established [arxiv:2011.00597 tab. 1].

### Caption-segment cycle without timestamps (WSDEC)

A sentence localizer l(V, C) gives a time segment for a caption, and a caption
generator g(V, S) writes a caption for a segment [arxiv:1812.03849 §3.1]. The
two form a cycle in both directions. Reconstructing the caption through a
segment gives a caption loss L_c. Reconstructing the segment through a generated
caption gives a segment loss L_s. The total is L_c + λ_s L_s
[arxiv:1812.03849 §3.1] [eq. 3] [eq. 5] [eq. 6]. Card:
[Weakly Supervised Dense Event Captioning](/papers/1812.03849).

The cycle is the core of training, but not all of it. The anchor classifier is
trained separately, against the anchor with the best METEOR score for each
caption [arxiv:1812.03849 §3.2] [eq. 16]. Training alternates between the cycle
losses and the anchor loss in periodic epochs [arxiv:1812.03849 §4].

Test time does not run the full cycle. Random candidate segments are each
passed once through the localizer on their generated caption,
S' = l(V, g(V, S)) [arxiv:1812.03849 §3.1] [eq. 4]. The paper states that the
conditions for this fixed point are not guaranteed for these networks, so the
step should be read as a heuristic [arxiv:1812.03849 §3.1].

Reported results, author-reported on ActivityNet Captions. The model uses no
ground-truth segments [arxiv:1812.03849 §4]. For dense captioning, it reaches
CIDEr 18.77 and METEOR 6.30. The two supervised baselines score CIDEr 17.29 and
16.08, and METEOR 4.82 and 7.71, so the model is best on CIDEr and below one
baseline on METEOR [arxiv:1812.03849 tab. 1]. For sentence localisation with no
timestamps, R@1 at IoU 0.3 is 41.98. That is above the supervised CTRL result
(28.70) and below the supervised ABLR result (55.67) and the paper's own fully
supervised localizer (52.89) [arxiv:1812.03849 tab. 2]. Tables 1 and 2 have no ±
values on the card. Over five seeds at N_r of 10 to 20, the standard deviation
of CIDEr is at most 0.14 [arxiv:1812.03849 tab. 3].

The table has no row that removes the cycle loss alone. Its "Pretrained only"
row is not described on the card as a cycle ablation, so this page does not use
it as one [arxiv:1812.03849 tab. 1].

## When to use which

| Situation | Use | Why |
|---|---|---|
| Two ordered sequences that describe the same content, with a known pairing to align | The soft nearest-neighbour cycle as an extra loss term | The paper's ablation shows a gain with and without its contextual transformer [arxiv:2011.00597 tab. 1] |
| One of the two sides has no meaningful order or index | Not covered | The loss is defined on the index of each item in both sequences [arxiv:2011.00597 §3] [eq. 8]; the paper tests only clip-sentence and video-paragraph data [arxiv:2011.00597 §4] |
| Limited budget for tuning, COOT setting | Start from a published weight, then sweep it | The paper sets the weight per dataset, 0.01 on ActivityNet and 0.001 on YouCook2, found by manual search and BOHB [arxiv:2011.00597 app. A.1]. A new dataset needs its own sweep; the two values do not transfer by themselves |
| Comparing COOT runs, or expecting reproducible numbers | Repeat runs over several seeds before reading a small gain | The cycle term draws random positions, so results vary with the seed; the number of runs behind the reported ± values is not stated [arxiv:2011.00597 tab. 1] [arxiv:2011.00597 §4] |
| Dense captioning with only captions per video, no segment boundaries | The caption-segment cycle (WSDEC) | The model is trained without timestamps and scores above two supervised baselines on CIDEr [arxiv:1812.03849 tab. 1] |
| Sentence localisation with no timestamps, as a baseline | WSDEC's localizer, with its test-time step | Its unsupervised R@1 at IoU 0.3 is 41.98, below the supervised ABLR result of 55.67 [arxiv:1812.03849 tab. 2] |
| Caption model not pretrained on the whole video | Pretrain the caption generator first | The cycle needs a meaningful caption model to start from; the paper pretrains on the full-range segment (0, 1) [arxiv:1812.03849 §4] |
| Videos with many overlapping events, or events with no caption | Expect the one-to-one assumption to fail | The paper's own qualitative results show missed event starts and two or three captions per video [arxiv:1812.03849 §4] |
| Re-implementing WSDEC from its official repository | Implement from the paper and the card | The repository has no licence file, so no code is quoted; see the [WSDEC card](/papers/1812.03849) for the differences found in the code |

## Implementation sketch

Both variants use the same round-trip idea but different operations. The COOT
sketch vectorises its equations over a sequence. Run it once per direction and
add the results, with a weight λ.

```pseudocode
# One direction of the cycle  [arxiv:2011.00597 §3] [eq. 6] [eq. 7] [eq. 8]
# seq: (N, d) items to cycle; other: (M, d) items of the other sequence

def cycle_loss(seq, other):
    # Soft nearest neighbour of each item in the other sequence  [eq. 6]
    weights = softmax(-sq_dist(seq, other), axis=1)  # (N, M)
    nearest = weights @ other  # (N, d)
    # Cycle back: soft position of each neighbour within seq  [eq. 7]
    back = softmax(-sq_dist(nearest, seq), axis=1)  # (N, N)
    soft_index = back @ arange(N)  # (N,)
    # Squared gap between the true index and the soft index  [eq. 8]
    return mean((arange(N) - soft_index) ** 2)

# Total: alignment terms plus lam times both directions  [eq. 9]
# loss = align_terms + lam * (cycle_loss(clips, sents)
#                             + cycle_loss(sents, clips))
```

The WSDEC sketch is one training step. The caption is the only input; the
video is held fixed.

```pseudocode
# One training step of the caption-segment cycle
#   [arxiv:1812.03849 §3.1] [eq. 3] [eq. 5] [eq. 6]
# video: (T, d) frame features; caption: tokens of one event, no timestamps

def caption_segment_cycle(video, caption):
    # Localise the caption as a segment (centre, width) in [0, 1]
    #   [§3.2] [eq. 10]
    segment = localizer(video, caption)
    # Perturb the segment with noise before captioning  [§3.1] [eq. 5]
    noisy = segment + noise()  # noise family and scale: see Contradictions
    # Caption the perturbed segment; compare with the original caption
    #   [§3.2] [eq. 3] [eq. 14]
    logits = captioner(video, noisy)
    loss_caption = cross_entropy(logits, caption)
    # Decode, then localise the generated caption  [§3.1] [eq. 5]
    # Gradient flow through decode: not stated [§3.1]
    segment_back = localizer(video, decode(logits))
    # Segment reconstruction loss  [eq. 15]
    loss_segment = squared_error(segment_back, segment)
    # Total for this step  [eq. 6]
    return loss_caption + lambda_s * loss_segment
```

```pseudocode
# Test-time captioning with one refinement round  [§3.1] [eq. 4] [§4]
# Candidate count N_r = 15 in the paper; overlap rules: see Contradictions

def dense_caption(video, n_rand):
    # Draw random (centre, width) candidates  [§4]
    guesses = uniform(0, 1, shape=(n_rand, 2))
    # One fixed-point step: localise each candidate's caption  [eq. 4]
    refined = localizer(video, [captioner(video, g) for g in guesses])
    # Keep and merge refined segments by the overlap rules
    segments = filter_and_merge(guesses, refined)
    return [(captioner(video, s), s) for s in segments]
```

Where variants differ:

- **COOT, distance scale.** The paper writes the squared L2 distance over the
  full feature vector [eq. 6]. The scale of that distance sets how sharp the
  softmax is, and so how much weight the nearest item gets. See Contradictions.
- **COOT, sampling.** The paper's equations treat every item. It reports that
  one clip per video and one sentence per paragraph gave the best results
  [arxiv:2011.00597 §4].
- **COOT, weight.** λ is set per dataset [arxiv:2011.00597 app. A.1].
- **WSDEC, loss weights.** The paper sets λ_s to 0.1 and λ_a to 0.1
  [arxiv:1812.03849 §4]. The code scales the segment term differently; see
  Contradictions.
- **WSDEC, candidates.** The paper uses random candidates and 15 of them
  [arxiv:1812.03849 §4]. Over N_r of 10, 15 and 20, CIDEr rises from 17.75 to
  19.14 [arxiv:1812.03849 tab. 3], so the candidate count is a small lever.

## Contradictions

- **COOT, distance scale in the official code.** The paper's soft
  nearest-neighbour weights use the full squared distance [eq. 6]. The official
  repository, which the card records, averages over feature dimensions instead
  of summing. That changes the softmax temperature by about the feature
  dimension (384 for COOT's hidden size). Both forms are on the card's Paper vs
  code section, [COOT card](/papers/2011.00597).
- **COOT, averaging.** The paper's per-item loss [eq. 8] is averaged over items
  in the code, over the batch, and over positions with one random index per
  sequence (see the card). The paper does not state the averaging used.
- **COOT, input normalisation.** The contrastive terms use L2-normalised
  embeddings. The paper does not say whether the cycle term uses normalised
  embeddings. The official code feeds it un-normalised ones (see the card).
- **COOT, weight scale.** The paper's λ values were chosen for the paper's
  summed contrastive form, and the code's contrastive losses are scaled
  differently. λ values are therefore not interchangeable between the paper and
  the code [arxiv:2011.00597 app. A.1].
- **WSDEC, segment-loss scale.** The paper sets λ_s to 0.1 on the squared error
  of centre and width [arxiv:1812.03849 §4] [eq. 15]. The official code
  multiplies the mean squared error by 1/100 and adds it with weight 1, as the
  card's Paper vs code section records [WSDEC card](/papers/1812.03849). Both
  forms are recorded; this page does not choose one.
- **WSDEC, noise.** The paper uses Gaussian noise on the localizer's output,
  with σ not stated [arxiv:1812.03849 §3.1] [eq. 5]. The official code uses
  uniform noise in (−0.01, 0.01) on centre and width, per the card.
- **WSDEC, test-time overlap rule.** The paper says to keep segments with high
  overlap between a candidate and its refinement [arxiv:1812.03849 §4]. The
  official code skips a refined segment when its overlap with its guess is above
  0.2, so it keeps low-overlap segments, per the card. Both sides are stated;
  this page does not resolve them.
- **WSDEC, ablation claims.** The text says removing either component decreases
  accuracy [arxiv:1812.03849 §4]. In tab. 1, the no-regression variant beats the
  final model on BLEU-3 (2.69 against 2.62) and BLEU-4 (1.44 against 1.27),
  while scoring lower on METEOR, CIDEr and ROUGE-L [arxiv:1812.03849 tab. 1].
- **WSDEC, table mismatch.** The N_r = 15 row of tab. 3 (METEOR 6.29, CIDEr
  18.65) does not match the "Ours" row of tab. 1 (6.30, 18.77). The paper does
  not say why [arxiv:1812.03849 tab. 1] [arxiv:1812.03849 tab. 3].

## Papers

- [COOT](/papers/2011.00597): defines the soft nearest-neighbour cycle on clips and sentences, applies it in both directions, and reports it as an extra term in the ablation.
- [Weakly Supervised Dense Event Captioning](/papers/1812.03849): trains a caption generator and a sentence localizer as a caption-segment cycle, so dense captioning needs captions but no timestamps.
