Paper · 2018-12-10 · cs.CV
Weakly Supervised Dense Event Captioning in Videos
Dense event captioning without temporal segment labels. A sentence localizer and a caption generator are trained as dual tasks with a cycle reconstruction loss and an anchor classifier. Test-time random segments are refined by one fixed-point step. On ActivityNet Captions the author-reported CIDEr is 18.77, the best of the compared methods, and METEOR is 6.30.
Weakly-supervised learningCycle consistencyVisual groundingCross-attentionvideodense-captioningweak-supervisiontemporal-localization
arXivPDFCodeMarkdown12 min read
AI-compiled by claude-haiku-5-5 on 2026-11-13, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Describing every event in a long video usually needs a person to mark when each event starts and ends, as well as writing what happens. That marking is slow and expensive. This paper asks whether a model can learn the task from captions alone, with no timestamps.
The trick is that a caption and its time span point at each other. A model that finds the span for a sentence, and another that writes the sentence for a span, can check each other: write a sentence, find its span, then rewrite the sentence from that span, and compare with the start. Random spans are then nudged towards the spans that the caption model can describe well. At test time the model starts from random spans and produces a caption for each span it keeps. The same localizer also finds the span for a given sentence, without any timestamps.
TL;DR
Dense event captioning is trained with only per-video caption text, using a caption generator and a sentence localizer that are inverses of each other. Test-time candidates come from random segments, refined once by the localizer. Author-reported results on ActivityNet Captions are comparable to two fully-supervised baselines on METEOR and the best of the three on CIDEr. Use this card if you need weak supervision for temporal captioning or an unsupervised sentence-localization baseline.
Problem
Dense event captioning finds all events in a video and writes one sentence per event. Standard methods learn from event boundaries and captions together §1 §2. Collecting boundaries is what the paper calls source-consuming §1. The paper's setting gives only a paragraph or a set of sentences per video §1. The assumption is one-to-one correspondence: each caption describes one segment and each segment has one caption §1 §3.1.
A proposal model cannot be trained from captions alone, because event proposals are one-to-many and too noisy to learn weakly §3.1. The paper avoids the proposal model.
Method
Formulation §3.1. A video is V with frames v_t. A segment is S = (m, w), with centre m and width w, both normalised to [0, 1] §3.1. A caption is C_i. Two functions are defined:
- sentence localizer l(V, C) gives a segment §3.1;
- event captioner g(V, S) gives a caption for a segment §3.1.
The caption and segment of one event are fixed points of the nested maps eq. 1 eq. 2. The caption reconstruction loss L_c eq. 3 is a caption auto-encoder that never sees the segment §3.1. A segment reconstruction loss L_s eq. 5 uses Gaussian noise on the localizer's output, which the paper motivates as a denoising auto-encoder §3.1. The total is L = L_c + λ_s L_s eq. 6.
Test-time fixed-point step §3.1. Candidate segments are sampled at random. Each is passed once through S' = l(V, g(V, S)) eq. 4. The paper uses one round, not many. Refined segments then give captions C = g(V, S') §3.1. Candidates that converge to the same segment are merged, and the rest are kept §4 helper_function.py. The paper states that the fixed-point conditions are not guaranteed for these networks, and the Gaussian smoothing does not give Lipschitz continuity in theory §3.1.
Sentence localizer §3.2. Two attention blocks read the video and the sentence:
- f_c attends over the caption features with the video's final hidden state as query eq. 7;
- f_v attends over the video features with the caption's final hidden state as query eq. 8.
The two are fused as in eq. 9, which cites a fusion layer of Gao et al. §3.2. The paper writes the fusion with an undefined symbol f_s eq. 9. The code uses the caption context there sentence_localizer.py. The fused feature is turned into a segment by two heads. The first is a classification over anchor segments, trained to pick the anchor with the best METEOR score §3.2. The second is a regression offset from that anchor §3.2 eq. 10. The paper says a direct regression is easily stuck in local minima, which motivates the anchors §3.2.
Caption generator §3.2. A soft mask over frames replaces hard clipping, so that the step is differentiable eq. 11. The mask is a difference of two sigmoids with scale K. The frame features are averaged with the normalised mask eq. 12. A recurrent decoder then writes the caption, starting from the hidden state at the segment's end eq. 13 caption_generator.py.
Losses §3.2. Caption loss L_c is a token-level cross-entropy eq. 14. Segment loss L_s is the squared error on (m, w) eq. 15. Anchor loss L_a is a softmax cross-entropy over anchors against the best-METEOR anchor eq. 16. The total loss is L = L_c + λ_s L_s + λ_a L_a eq. 17.
Training schedule §4 train_final.py. The caption generator is first pretrained on the whole video §4. Then training alternates: most epochs minimise L_c and L_s, and periodic epochs minimise L_a. The paper states two stages, L_c with L_s and then L_a §4.
# Cycle training step, L_c + L_s [§3.1] [eq. 3] [eq. 5]
# Input: video features (T, d); caption tokens of one event
# Output: scalar loss, gradients to localizer and generator
def cycle_step(video, caption):
# 1. Localize the caption on the video [§3.2] [eq. 10]
segment = localizer(video, caption) # (c, w) in (0, 1)
# 2. Add small Gaussian noise to the segment [§3.1] [eq. 5]
noisy = segment + normal(0, sigma) # sigma not stated
# 3. Caption the video under the soft mask [§3.2] [eq. 11] [eq. 12]
caption_prob, caption_pred = generator(video, noisy)
# 4. Re-localize the generated caption [§3.1] [eq. 5]
segment_back = localizer(video, caption_pred.detach())
# 5. Caption loss and segment loss [eq. 14] [eq. 15] [eq. 6]
loss_caption = caption_nll(caption_prob, caption)
loss_segment = squared_error(segment_back, segment)
return loss_caption + lambda_s * loss_segment# Test-time dense captioning [§3.1] [§4 Testing] [eq. 4]
# Input: video features (T, d); n_rand; one refinement round
# Output: list of (caption, segment) pairs
def dense_caption(video, n_rand=15):
# 1. Draw random (c, w) guesses in (0, 1) [§4 Testing]
guesses = uniform(0, 1, shape=(n_rand, 2))
# 2. Caption each guess [§3.2] [eq. 13]
captions = generator(video, guesses)
# 3. One fixed-point step: localize the generated captions [eq. 4]
refined = localizer(video, captions)
# 4. Keep refined segments whose IoU with their guess is at most 0.2
# [code: utils/helper_function.py#L117]
# 5. Merge refined segments whose IoU is above 0.9
# [code: utils/helper_function.py#L129]
segments = merge_and_filter(guesses, refined)
# 6. Caption the kept segments [§3.1]
return [(generator(video, s), s) for s in segments]Hyperparameters
Values are as stated in the paper, then the code default where they differ or where the paper is silent.
| Name | Value | Where | Notes |
|---|---|---|---|
| Vocabulary size | 6000 | §4 | Code default is 6000 train_final.py |
| GRU hidden and output size | 512 | §4 | Encoders are in the supplement, which is not in the source; code: hidden 512 train_final.py |
| Video feature dimension | 500 | not stated in paper | C3D features, 500-d in code train_final.py |
| Sentence embedding dimension | 512 | not stated in paper | train_final.py |
| RNN layers | 2 | not stated in paper | train_final.py |
| Dropout (fc, rnn) | 0.3, 0.3 | not stated in paper | train_final.py |
| Max caption length | 20 | not stated in paper | train_final.py |
| λ_s (segment loss weight) | 0.1 | §4 | Code uses 1/100 times the mean squared error instead; see Paper vs code |
| λ_a (anchor loss weight) | 0.1 | §4 | Code uses weight 1 on cross-entropy in the anchor phase train_final.py |
| Optimiser | SGD | §4 | train_final.py |
| Initial learning rate | 0.01 | §4 | Same in code train_final.py |
| Momentum | 0.9 | §4 | Code default 0.8 train_final.py |
| Weight decay | not stated | paper | Code default 1e-4 train_final.py |
| LR decay | ×0.1 at epochs 1, 2, 20 | not stated in paper | MultiStep, gamma 0.1 train_final.py |
| Training epochs | 25 | not stated in paper | train_final.py |
| Alternation period | every 5 epochs | §4 "two stages" | Code switches to anchor loss when step % 5 == 0 train_final.py |
| Batch size (caption) | 32 | not stated in paper | train_final.py |
| Batch size (localizer) | 6 | not stated in paper | Hard-coded train_final.py |
| Anchors | 15 | not stated in paper ("multiple scales") | global_config.py |
| Regression range | ±0.15 around the anchor | not stated in paper | sigmoid times 0.3, minus 0.15 sentence_localizer.py train_final.py |
| Soft-mask scale K | 0.1 | not stated in paper | Code default train_final.py |
| Noise on segment | Gaussian, σ not stated | §3.1 | Code: uniform in (-0.01, 0.01) train_final.py |
| Random test segments N_r | 15 | §4 | Default evaluate.py |
| Random seed (test) | 233 | not stated in paper | Code default evaluate.py |
| Refinement rounds | 1 | §3.1 §4 | evaluate.py |
| IoU merge threshold | not stated in paper | §4 | Code default 0.9 evaluate.py |
| Test-time IoU filter | 0.2 | not stated in paper | Hard-coded helper_function.py |
| Evaluation tIoU thresholds | 0.3, 0.5, 0.7, 0.9 | §4 | Scores averaged over these thresholds |
Results
Author-reported. Setup: ActivityNet Captions, 20,000 videos with about 3.65 events each, split 50/25/25 for train, validation and test §4. Scores use the official densevid_eval scripts, averaged over tIoU 0.3, 0.5, 0.7 and 0.9 §4. "Ours" is weakly supervised: it uses no ground-truth segments §4.
Dense event captioning tab. 1.
| Model | ws | METEOR | CIDEr | ROUGE-L | B@1 | B@2 | B@3 | B@4 |
|---|---|---|---|---|---|---|---|---|
| Krishna et al. | False | 4.82 | 17.29 | not reported | 17.95 | 7.69 | 3.86 | 2.20 |
| Yao et al. | False | 7.71 | 16.08 | 13.27 | 17.50 | 9.62 | 5.54 | 3.38 |
| Pretrained only | True | 4.58 | 10.45 | 9.27 | 8.70 | 3.39 | 1.50 | 0.69 |
| Ours, no classification | True | 6.08 | 15.10 | 12.25 | 11.85 | 4.67 | 1.90 | 0.80 |
| Ours, no regression | True | 6.11 | 17.66 | 12.40 | 11.98 | 5.45 | 2.69 | 1.44 |
| Ours | True | 6.30 | 18.77 | 12.55 | 12.41 | 5.50 | 2.62 | 1.27 |
Robustness to random segments tab. 3 (Ours, mean ± std over 5 seeds).
| N_r | METEOR | CIDEr | B@1 | B@2 | B@3 | B@4 |
|---|---|---|---|---|---|---|
| 10 | 6.13 ± 0.03 | 17.75 ± 0.12 | 12.10 ± 0.06 | 5.33 ± 0.05 | 2.50 ± 0.03 | 1.20 ± 0.01 |
| 15 | 6.29 ± 0.01 | 18.65 ± 0.14 | 12.39 ± 0.04 | 5.49 ± 0.02 | 2.58 ± 0.03 | 1.23 ± 0.02 |
| 20 | 6.34 ± 0.01 | 19.14 ± 0.05 | 12.52 ± 0.02 | 5.57 ± 0.02 | 2.61 ± 0.02 | 1.25 ± 0.02 |
Sentence localization tab. 2 (unsupervised for "Our Final"; R@1 at IoU thresholds 0.1, 0.3 and 0.5, and mIoU).
| Model | Unsupervised | R@1 IoU 0.1 | R@1 IoU 0.3 | R@1 IoU 0.5 | mIoU |
|---|---|---|---|---|---|
| CTRL (Gao et al.) | False | 49.09 | 28.70 | 14.00 | 20.54 |
| ABLR (Yuan et al.) | False | 73.30 | 55.67 | 36.79 | 36.99 |
| Fully supervised (paper's own) | False | 70.01 | 52.89 | 37.61 | 40.36 |
| Ours, final | True | 62.71 | 41.98 | 23.34 | 28.23 |
Recall of detected events fig. 2 is plotted against tIoU; the paper reports it as higher than the random-proposal model, and the final model above both baselines in general. No numbers are given in the text.
Qualitative fig. 3 shows captions for two videos. The paper notes that the model often misses the start of an event and tends to output two or three captions per video §4.
Paper vs code
Code: https://github.com/XgDuan/WSDEC, commit
64abf5ee041ac6ad138de31880dd6cad73c70b00. The owner is the first author's
GitHub account, so this is treated as the official repository. The README
calls it "the official repo" [README.md#L2]. The paper gives no URL; it says
the details are in "our Github repository" §4. The repository has no LICENSE
file, so no code may be quoted and code_license is recorded as none.
- Momentum: paper says 0.9 §4; code default 0.8 train_final.py.
- Segment-loss weight: paper says λ_s = 0.1 §4; code multiplies the mean squared error by 1/100 train_final.py and adds it with weight 1 train_final.py.
- Anchor-loss weight: paper says λ_a = 0.1 §4; code trains only the cross-entropy during anchor epochs, with weight 1 train_final.py.
- Schedule: paper says two stages, L_c with L_s first, then L_a §4; code alternates the two phases every 5 epochs over 25 epochs after a separate pretraining train_final.py train_cg_pretrain.py.
- Noise: paper says Gaussian §3.1 eq. 5; code uses uniform noise in (-0.01, 0.01) on (c, w) train_final.py.
- Test-time filter: paper says keep segments with high IoU between the candidate and its refinement §4 Testing. Code skips a refined segment when its IoU with its guess is above 0.2, so it keeps low-overlap segments helper_function.py. The code's rule is the one to implement; it is the opposite of the paper's wording.
- Test-time candidates: paper says random segments §4; code draws c and w each uniformly in [0, 1), so a candidate can have w near 1 helper_function.py.
- Regression head: paper says an FC layer gives ΔS §3.2 eq. 10. Code bounds it to ±0.15 with a sigmoid sentence_localizer.py.
- Regression loss: the localizer's own L1 regression loss exists sentence_localizer.py, but train_final.py never calls it. The regression head only learns from the cycle loss, through forward_diff train_final.py.
- Fusion and heads: paper writes a plain FC after the fusion eq. 9. Code adds batch norm, ReLU and a two-layer combiner before the anchor and regression heads sentence_localizer.py sentence_localizer.py.
- Soft mask sign: the paper's mask eq. 11 is written with the opposite sign to the code, and the text says the mask is zero inside the segment. The code gives one inside attention.py. The sign cancels after normalisation eq. 12 only if the attention weights are normalised after masking; check this when implementing.
- Caption generator context: the paper's decoder start is the hidden state at the segment end eq. 13. The code gets it from the segment end index caption_generator.py.
- Localizer batch size: 6, hard-coded train_final.py; the paper says nothing.
- Gradient clipping: the config comment says the clip value is "not used" train_final.py, but clip_grad_norm is called on the localizer during cycle epochs train_final.py.
- External data: the code's output JSON marks "external_data" as used, with the provided C3D features train_final.py. The paper says the cycle method uses no external proposal model §4. The C3D features themselves come from the ActivityNet Captions release, as the README shows [README.md#L34-L41].
- Table 3 vs Table 1: the N_r = 15 row of Table 3 (METEOR 6.29, CIDEr 18.65, B@4 1.23) does not match the "Ours" row of Table 1 (6.30, 18.77, 1.27). The paper does not say why.
Pitfalls
- No pretrained proposal model is allowed. The paper rejects SST for candidate generation because it uses external temporal data §4. Random candidates are the intended source.
- Pretraining is needed. The caption generator is pretrained on the whole video, with the full-range segment (0, 1) §4 train_cg_pretrain.py. Without it the cycle has no meaningful caption model to start from.
- The one-to-one assumption is load-bearing. Videos with many overlapping events, or events with no caption, break it. The paper's own qualitative results show missed event starts and two or three captions per video §4.
- Fixed-point convergence is not guaranteed. The paper gives a Lipschitz condition §3.1 and says the network does not satisfy it, then uses one round §3.1. Treat the test-time step as a heuristic.
- Random-segment sensitivity. The paper shows small variance across seeds at N_r = 10 to 20 tab. 3. The gain from N_r is small: CIDEr rises by about 1.4 from 10 to 20 tab. 3.
- Ablation claims are not uniform. The text says removing either component decreases accuracy §4. In tab. 1, the no-regression variant beats the final model on BLEU-3 (2.69 against 2.62) and BLEU-4 (1.44 against 1.27), while scoring lower on METEOR, CIDEr and ROUGE-L. Do not cite the ablation as uniform.
- Hard anchor choice. Localization uses argmax over anchors at test time sentence_localizer.py sentence_localizer.py. The predictor gets gradients only from the anchor phase, so the cycle losses cannot change which anchor is picked.
- Anchor labels cost one caption pass per anchor. The anchor label is the best METEOR over all 15 anchors, so each training step runs the generator 15 times and a METEOR scorer per video train_final.py. Expect this to dominate training time.
- Environment. The code targets Python 2.7, PyTorch 0.3.1 and CUDA 9.0 [README.md#L17-L22]. CPU is not supported: the config asserts CUDA and the code calls
.cuda()throughout global_config.py. - Seeds. Global seeds are fixed to 233 global_config.py.
- Missing details. The supplement is referenced for the video and sentence encoders and for proposals §3.2 §4, but it is not in the source. The encoder details above come from the code and are labelled as such.
Lineage
- Builds on: the anchor-based localizer follows the multi-modal fusion of Gao et al. (CTRL) and the regression idea of Yuan et al. (ABLR) §3.2. The caption generator is similar to S2VT, the recurrent video captioner of Venugopalan et al. §2 §3.2. Neither is compiled here.
- Compared against: Krishna et al. (ActivityNet dense captioning, DAP and context S2VT) and Yao et al. (ActivityNet Challenge 2017 entry) tab. 1 §2.
- Superseded by: not known from this wiki.
Source and license
Adapted from “Weakly Supervised Dense Event Captioning in Videos” by Xuguang Duan, Wenbing Huang, Chuang Gan, Jingdong Wang, Wenwu Zhu, Junzhou Huang (arXiv:1812.03849), used under CC0 1.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/XgDuan/WSDEC at commit 64abf5ee041a (none (no LICENSE file in the repository)). Any quoted code is shown under that license.