MovieNet: A Holistic Dataset for Movie Understanding

In plain words

MovieNet is a dataset for understanding a whole film, whose plot runs across many scenes. For each movie it bundles many kinds of material: the trailer, still photos, a plot summary, the script, subtitles and facts such as genre and cast. On top of that, people labelled the film. They drew boxes around characters and named them, marked where each scene starts and ends, tagged what happens and where in each stretch, and tagged how the camera is framed and moves.

The authors use these labels to define tasks: find the people in a frame, name the characters, find where scenes change, guess the genres, recognise the camera style, and match a plot summary to the right part of a film. They run standard models on each task. The results show that some tasks, such as finding people, are close to solved, while the story-level tasks are still hard. The paper's text is under a free licence, but that licence covers the paper, not the films or the labels, so check the access terms before using the data.

TL;DR

A dataset paper, not a method. MovieNet pairs 1,100 movies with multi-modal material and manual labels (characters, scenes, actions and places, camera style, synopsis alignment) and sets up nine benchmark tasks with baselines. Use it to find a large movie dataset and its task definitions, or to check which baseline numbers exist. Every number is author-reported. Access to the movies and labels is governed by terms outside the paper's CC BY 4.0 licence.

Problem

Story-based long video, such as film, is hard for current models §1. Earlier movie datasets were small, covered one modality, or focused on short clips or single images §2, §3.1, tab. 1, tab. 2. Genre work used small image or trailer sets §4.1. Character recognition work used other data sources such as COCO, surveillance and ReID sets, or small movie benchmarks §4.3. Scene segmentation datasets were a few hundred scenes app. C.5, tab. C4. The paper targets all of these at once: scale, several modalities, and several kinds of annotation on the same films §1.

Method

The paper has two parts: how the dataset was built (§3, app. A and B), and how each benchmark was set up and run (§4, app. C). Nothing here is a new model except the MS-LSTM scene model and the cinematic-style-guided attention module, which are described in §4.4 and §4.5 and app. C.5 and C.8.

Data collection §3.1 app. A

1. Movies: about 1,100 films bought as copies, chosen to be in colour, longer than one hour, and spread across genres, years and countries §3.1.
2. Metadata: collected from IMDb and TMDb. The fields cover basic facts about the film (its name, dates, places, genre, rating, length), its people (the director and cast), and its plot summary. The paper reports 28 genres, 805K genre tags and 375K metadata items in total §3.1 app. A, tab. A1.
3. Trailers: downloaded from YouTube using the links listed on IMDb and TMDb, not from a title search. The paper reports 60K trailers for 33K movies §3.1.
4. Subtitles: taken from the embedded subtitle stream where present, otherwise crawled from a subtitle site for movies with no English subtitle; all were checked by hand for alignment §3.1.
5. Scripts: about 2K from two script archives, aligned to the film timeline by matching dialogue to subtitles §3.1 (algorithm in app. A, alg. 1).
6. Synopses: 11K plot summaries from IMDb, each with more than 50 sentences, aligned to movie segments by hand §3.1 §3.2.
7. Photos: 3.9M from IMDb and TMDb, including posters, still frames and publicity photos §3.1.

Annotation §3.2 app. B

• Characters: four steps. (1) Keyframes are selected for box labelling (758K keyframes). (2) A detector is trained on these labels. (3) The detector proposes more boxes, which people clean by hand. (4) People label identities. Only the top ten credited cast members by IMDb order are named; all other people are labelled "others". The paper reports 1.1M instances of 3,087 credited identities plus 364K "others" §3.2.
• Cinematic style: each of 47K shots (from films and trailers) gets one view scale tag (five classes: long, full, medium, close-up, extreme close-up) and one camera-movement tag (four classes: static, pan or tilt, zoom in, zoom out) §3.2.
• Scene boundaries: two levels are used, shot and scene. A scene is a run of related shots. The paper reports 42K annotated scenes. The annotation workflow is in app. B and is not summarised here §3.2.
• Action and place tags: each scene is cut into clips and clips are tagged with several places and, in sub-clips that contain characters, several actions. Tags that carry little story meaning, such as "stand" and "talk", were removed. Tags were kept with a minimum frequency of 25, leaving 80 action and 90 place classes. The result is 42K segments with 19.6K place tags and 45K action tags §3.2.
• Synopsis alignment: three annotators aligned synopsis paragraphs to segments in a coarse-to-fine procedure, giving 4,208 paragraph-segment pairs §3.2.
• Trailer-to-movie shot alignment: an automatic method maps trailer shots onto the film app. B. See Pitfalls.

Benchmarks §4 app. C

Each task uses a 3:1:1 split of movies unless the task states otherwise app. C, intro.

• Genre classification (multi-label, 21 genres). Image input: 1.6M photos of four types. Video input: 32K trailers. Loss: binary cross-entropy §4.1 app. C.1. Long-film genre analysis uses a shot-level visual model trained on trailers, followed by a temporal model over the film's shot features, to produce a genre-response curve §4.1.
• Cinematic style prediction (two tasks: scale and movement). Baselines are TSN and I3D; TSN+R3Net adds a subject branch. R3Net gives a saliency map for each frame; a two-branch network takes the whole frame and the subject region for scale, and the whole frame and background for movement. Branch features are concatenated and passed to a fully connected layer §4.2 app. C.2.
• Character detection. Faster R-CNN, RetinaNet and Cascade R-CNN with a ResNeXt-101 backbone and feature pyramid, trained on MovieNet and compared with models trained on COCO, CalTech and CSM §4.3 app. C.3.
• Character identification. Query: one portrait per credited cast member, taken from the cast member's homepage. Search each instance in the movie by face and body features. Baselines: body-only ResNet-50 softmax, a two-step method (face first, then body set matching), and PPCC §4.3 app. C.4.
• Scene segmentation. Baselines: Grouping, Siamese, and MS-LSTM. MS-LSTM is the authors' Bi-LSTM model over per-shot audio, character, action and place features §4.4 app. C.5.
• Action and place tagging (multi-label). Action: TSN, I3D and SlowFast. Place: TSN and I3D §4.4 app. C.6, app. C.7.
• Movie-to-synopsis segment retrieval. A VSE model embeds a segment and a paragraph in a joint space. Extra nodes (appearance, subtitle, action, character) are added one at a time. Character nodes are matched by a Kuhn-Munkres assignment. Training uses a pairwise ranking loss §4.5 app. C.8:

\mathcal{L}(S;\theta) = \sum_i \sum_{j \neq i} \max(0,\ S(Q_j,P_i) - S(Q_i,P_i) + \alpha) + \sum_i \sum_{j \neq i} \max(0,\ S(Q_i,P_j) - S(Q_i,P_i) + \alpha) app. C.8

Here Q are segments, P are paragraphs, S is the similarity and \alpha is the margin. A graph-matching variant (MovieSynAssociation) adds event-flow and character-interaction modules §4.5 app. C.8.

Dataset-building procedure for script alignment. The paper formulates dialogue-to-subtitle matching as a monotone assignment and solves it with dynamic programming app. A. The pseudocode below is built from the constraint stated in app. A, eq. opt, not copied from alg. 1 (see Pitfalls):

# Monotone dialogue-to-subtitle matching  [app. A, eq. opt]
# Built from the constraint i_(j-1) <= i_j; not copied from alg. 1
# Input:  sim (M, N): TF-IDF cosine, dialogue snippets x subtitles
# Output: match (N,): snippet index for each subtitle, nondecreasing

def match_monotone(sim):
    M, N = sim.shape
    # score[i]: best total for subtitles 0..j, subtitle j -> snippet i
    score = sim[:, 0].copy()                 # (M,)
    back = zeros((N, M), dtype=int)
    for j in range(1, N):
        # Best previous score over snippets <= i (prefix maximum)
        prefix = running_max(score)          # (M,)
        back[j] = running_argmax(score)      # (M,)
        score = prefix + sim[:, j]           # (M,)
    # Backtrack from the best final match
    match = zeros(N, dtype=int)
    match[N - 1] = argmax(score)
    for j in range(N - 1, 0, -1):
        match[j - 1] = back[j][match[j]]
    return match

Trailer shot localisation. Each trailer shot is located in the film by the window of film frames whose summed frame similarity is largest [app. B, eq. trailer_align]. Frame similarity is the sum of two cosine similarities: one on GIST features and one on ResNet-50 pool5 features pretrained on ImageNet app. B.

# Locate a trailer shot in the film  [app. B]
# Input:  shot (M frames), movie (N frames), frame_sim(a, b) in [-2, 2]
# Output: best start index and its summed similarity

def locate_shot(shot, movie):
    best_start = None
    best_score = float("-inf")
    for start in range(len(movie) - len(shot) + 1):
        # Sum similarity over the window of the same length as the shot
        score = 0.0
        for i in range(len(shot)):
            score += frame_sim(shot[i], movie[start + i])
        if score > best_score:
            best_start, best_score = start, score
    return best_start, best_score

Hyperparameters

Values are from the paper's appendix unless marked otherwise. Where the paper says nothing, the cell says "not stated". Dataset-building parameters are in the rows above.

Results

Author-reported. Setup is given per row, and each row's source is in the last column; the benchmark sections are §4.1 to §4.5 and app. C.1 to app. C.8. The paper gives no statistical test or variance for these numbers.

Per-genre AP for the ResNet-50 model ranges from 9.13 (Biography) to 86.45 (Animation) app. C.1, tab. C1.

Paper vs code

The official toolbox is movienet/movienet-tools at code_commit. The project page links it from the MovieNet GitHub organisation [project page]. The repository is the toolbox only.

• Scope. The repository has crawlers, shot detection, feature extractors (action, place, audio, person detection), a data parser and model configs . It has no training or evaluation code for the seven benchmarks in §4 and app. C. The paper says benchmark code would be released with the toolbox app. D. This card's benchmark hyperparameters are therefore not verified against code.
• No script or trailer alignment code. A search for alignment, TF-IDF and GIST in the repository found only ROI-align operators and shot-detection files. Algorithm 1 and the trailer-alignment method are paper-only.
• Shot detection. The paper cites a shot detector §3.2. The code uses ContentDetectorHSVLUV with threshold 30.0, minimum shot length 15, HSV gap 10, LUV threshold 40 and HSV weight 5 [code: movienet/tools/shotdetect/shotdetect/content_detector_hsv_luv.py#L39-L46], registered in shotdetector.py. The paper states none of these values.
• Person detection inference. The detection script keeps boxes with confidence at least 0.5 by default detect_person.py. The paper does not state this threshold. The model config has two classes (character and background) cascade_rcnn_x101_64x4d_fpn.json, is consistent with the paper's Cascade R-CNN with ResNeXt-101 and FPN app. C.3; the paper does not give the 64x4d variant name.
• Licence. setup.py declares "Apache License 2.0" setup.py but the repository has no LICENSE file. The code_license field records Apache-2.0 from setup.py only.
• Dataset release. The paper says the dataset "will be published in compliance with regulations" §1, abstract. The project page describes the annotations and features as downloadable under a user agreement, with full movies "coming soon" [project page, fetched 2026-11-01].

Pitfalls

• Licence scope. The card is under CC BY 4.0, which covers this paper's text. It does not license the films, trailers, photos or labels. The project page describes access through a user agreement and says movies are released under strict conditions [project page]. Check those terms before any commercial use of the data.
• Count conflicts in the paper. These do not reconcile, so use the number the task needs and cite its location:
  • Trailers: 60K for 33K movies §3.1, app. A vs 68K in the genre table tab. 3.
  • Scene boundaries: 42K in §3.2 and app. C.5, but "40K" in the introduction §1.
  • Aligned description sentences: 2.5K in the abstract, 12K in the introduction §1, and 25K in the comparison table tab. 2.
  • Character identities: the text says 3,087 identities §3.2. App. C.4 says 3K identities split 2,088 / 821 / 876, which sum to 3,785. The instance split (639.9K + 336.6K + 364.2K = 1,340.7K) also does not match the 1.1M headline.
  • Action vs place splits use different unit counts: 41,259 action clips and 13,700 place clips app. C.6, app. C.7.
  • Cinematic TSN+R3Net scale accuracy: 87.50 tab. 4 vs 87.58 [app. C.2, tab. C2].
• Trailer shot localisation equation. The sum runs over the movie length N, but it should run over the shot length M. The formula also writes s_N^T where it means the movie frame sequence app. B, eq. trailer_align. The threshold for "low similarity" marking misaligned shots is not stated app. B.
• Script alignment algorithm. The recurrence in the text (L(i,j) = max{L(i,j-1), L(i-1,j) + s_ij}) and the loop in alg. 1 use different neighbours and a different backtracking pointer app. A. Implement from the monotone constraint, as the pseudocode above does, and test on small matrices. This card did not run either version.
• Benchmark splits. The 3:1:1 movie split applies "unless specified" app. C, intro. Several tasks use other split rules; check each one.
• Character identification queries. The paper says portraits come from the cast members' homepages app. C.4 but does not name the source sites.
• Scene segmentation model. MS-LSTM is described as a Bi-LSTM over multi-modal shot features §4.4 app. C.5. Layers, hidden size, window and fusion are not stated. The Bi-LSTM is cited to earlier work §4.4, but the card does not verify those details.
• Story retrieval baselines. The main table compares the authors' VSE variants with MovieSynAssociation, a graph-matching method by Xiong et al. tab. 9. The paper says it implements that method with this paper's features, which differ from the original app. C.8. The numbers are not a direct comparison with the original work.
• Character detection claim. The paper states that character detection "can be well solved" with MovieNet and current detectors §4.3. That is the authors' conclusion from one split and one metric.
• Movie-level data. Each benchmark is split by movie. Action and place splits are random by clip but state "without overlapping movies" app. C.6, C.7. Verify this for any new split you build.
• Not summarised here. The scene-boundary annotation workflow and the quality checks in app. B are not condensed on this card.
• Snapshot. The paper is a 2020 arXiv preprint (v1). The project page and the dataset may have changed since.

Lineage

• Builds on: image and video backbones and detectors cited in the benchmarks: TSN app. C.2, I3D app. C.2, SlowFast app. C.6, Faster R-CNN, RetinaNet and Cascade R-CNN app. C.3, and VSE app. C.8.
• Compared against: MovieQA, LSMDC, MovieGraphs and AVA for data and annotation tab. 1, tab. 2; MGCD, LMTD and MovieScope for genre tab. 3; Lie 2014, Sports 2007, Context 2011 and Taxon 2009 for cinematic style tab. 4; CSM, Market and CUHK03 for character recognition tab. 5, tab. 6; OVSD and BBC for scene segmentation tab. 7, tab. 8.
• Methods the benchmarks reuse: trailer-based genre learning [§4.1, cites Huang et al. 2018]; the PPCC identification method app. C.4; and the MovieSynAssociation graph matching method §4.5, app. C.8.
• Superseded by: not stated in the paper, and no later card in this wiki supersedes it.