ViMax: Agentic Video Generation

In plain words

Today's AI video tools are good at making a few seconds of footage, but a long story needs many shots. Each shot is generated on its own, so a character's face, clothes or a room's layout drifts from one shot to the next, and nothing plans the story as a whole.

ViMax handles a video the way a small film crew would. Separate AI roles write the screenplay, plan each shot, prepare reference pictures and check them. A long story is broken into smaller planning pieces, and the system looks up the original text so each piece stays tied to the whole story.

To keep things consistent across shots, the shots are grouped by camera angle and linked into a tree, so a shot can start from a picture that comes from an earlier camera. When two camera angles look at the same place, a short transition video is made between them so the place stays the same. The authors report that this gives more consistent multi-shot videos than the baselines they tested.

TL;DR

• ViMax is an agentic pipeline: screenwriter, storyboard planner, keyframe generator and video generator, with VLM judges §2.1.
• Story planning uses hierarchical decomposition plus retrieval over the source text §2.2.
• Cross-shot consistency uses a camera dependency tree, plus transition videos between camera angles in one scene §2.3.
• The authors' ViMax-Bench (35 stories) reports the best global consistency among the compared systems (author-reported) tab. 1.
• Implement the planning and consistency logic from this card. The reported numbers depend on proprietary models and are not reproducible from the repository alone [Limitations].

Problem

Video generators work on one shot at a time and have no memory of earlier shots. Character appearance, environment design and spatial layout therefore drift across shots §1. Streaming and single-clip methods extend one clip but do not handle multi-scene narratives §1.

Long narratives also exceed what a language model can plan in one pass. Dozens of shots need coordinated character arcs, plot and theme, and scene-level decisions must stay tied to the whole story §1 §2.2.

The method assumes access to strong proprietary language, image, video and VLM models. The authors note that performance is sensitive to model updates [Limitations].

Method

The system has three parts: a multi-agent production workflow §2.1, a hierarchical planning step with retrieval §2.2, and a visual-consistency layer built on a camera dependency tree and transition videos §2.3. The paper's figure of the architecture is fig. 2.

Screenwriting and shot planning §2.1.1. The user's input \mathcal{I} (a sentence, a full novel or a screenplay draft) is routed to one of several categories. The category selects a prompt, and the screenwriter LLM writes a screenplay \mathcal{S} eq. 1. A storyboard agent then turns \mathcal{S} into an ordered list of shot specifications, each with camera angle, movement, lighting, character positions and duration eq. 2.

Keyframes and videos §2.1.1. Images come first: character profiles, scene backgrounds and shot keyframes. Some images use other images as references, so a keyframe may depend on a character portrait eq. 3. Each shot video is then generated from its specification and its keyframes eq. 3.

Best-of-k quality control §2.1.2. For each generation task, k candidates are sampled in parallel. A VLM judge scores each one against the shot requirements on visual fidelity, narrative consistency and adherence to the shot specification. The highest-scoring candidate moves on eq. 4. The paper's operating point is k=2 §3.7.

# Best-of-k keyframe selection  [§2.1.2] [eq. 4]
# Paper default: k = 2  [§3.7]
def best_of_k(shot, references, k):
    # Sample k candidates in parallel  [§2.1.2]
    candidates = [gen_image(shot, references) for _ in range(k)]
    # The VLM judge scores each candidate against the shot spec  [eq. 4]
    scores = [vlm_judge(candidate, shot) for candidate in candidates]
    # Keep the highest-scoring candidate  [eq. 4]
    return candidates[argmax(scores)]

\text{score}_j = \text{VLM-Judge}(C_j, S_i), \qquad C^* = \arg\max_{C_j} \text{score}_j \qquad [\text{eq. 4}]

Hierarchical story decomposition §2.2.1. The screenplay is split recursively: major events, then scenes within each event, then shot specifications within each scene. Each node carries a text description T_{d,j} eq. 5. The authors argue this keeps each planning step small enough for the model §2.2.1.

\{N_{d,1}, \ldots, N_{d,k_d}\} = \text{Decomp}(N_{d-1,i}) \qquad [\text{eq. 5}]

Retrieval-augmented context §2.2.2. Decomposition tends to keep only the scope of each node and drop motivations, plot links and themes. The source text is therefore indexed once. For each node, the description is used to retrieve matching source passages, which are added to the node description eq. 6.

\mathcal{R} = \text{Index}(\mathcal{S}), \quad C_{d,j} = \text{Retrieve}(\mathcal{R}, T_{d,j}), \quad T'_{d,j} = \text{Enrich}(T_{d,j}, C_{d,j}) \qquad [\text{eq. 6}]

# Retrieval-enriched planning node  [§2.2.2] [eq. 6]
# The index is built once over the full source text  [eq. 6]
def plan_node(node_text, source_index):
    # Retrieve source passages that match the node description  [§2.2.2]
    passages = source_index.retrieve(node_text)
    # Enrich the node description with the retrieved context  [eq. 6]
    return enrich(node_text, passages)

Camera dependency graph §2.3.1. Shots in a scene are assigned to cameras. A dependency graph \mathcal{G} over the cameras is built from the shot specifications. During generation, the graph is traversed in topological order. Cameras whose dependencies are satisfied run in parallel. A dependent camera receives a visual anchor from its prerequisite eq. 7.

\mathcal{G} = \operatorname{BuildCameraGraph}(\{S_i\}_{i=1}^{n}), \quad \mathcal{K}_i = \operatorname{GenConditioned}(S_i, A_i) \qquad [\text{eq. 7}]

Transition videos §2.3.2. For two camera positions C_i and C_j in the same scene, a transition video V_{trans} is generated that moves the camera from one to the other. Views for both cameras are then extracted from this video, so both shots share the same spatial layout eq. 8.

V_{trans} = \text{GenTransition}(C_i, C_j, S), \quad V_i, V_j = \text{ExtractViews}(V_{trans}, C_i, C_j) \qquad [\text{eq. 8}]

The per-camera scheduling follows §2.3.1 and §2.3.2. A child camera waits for its parent's first frame, makes a transition, and then uses the view from it as the anchor for its own keyframe.

# Frames for one camera, in dependency order  [§2.3.1] [§2.3.2]
# Child cameras wait for the parent shot's first frame  [§2.3.1]
async def frames_for_camera(camera, shots, k):
    # The first shot of the camera is the anchor for the rest  [§2.3.1]
    first = camera.shot_idxs[0]
    anchor = None
    if camera.parent_shot is not None:
        # Wait until the parent shot's first frame exists  [§2.3.1]
        await first_frame_ready(camera.parent_shot)
        # Transition video between the two camera positions  [eq. 8]
        clip = gen_transition(parent_frame(camera), shots[first])
        # Extract this camera's view from the transition  [eq. 8]
        anchor = extract_view(clip, camera)
    # Keyframe for the first shot, using the anchor  [§2.3.1] [eq. 7]
    first_frame = best_of_k(shots[first], anchor, k)
    return first_frame

Hyperparameters

The paper reports no training of its own models. The values below are the ones it states, plus values the code sets. Values the paper does not state are marked "not stated".

Results

Author-reported. Setup: ViMax-Bench (35 stories) app. A.2. Global consistency (GC) and cross-scene consistency (CC) are ViCLIP cosine similarities across shot pairs; higher means more stable features app. A.2. Full table in tab. 1; selected rows below.

The authors state that ViMax-Gemini-3-Pro-Veo 3.1 is best in all nine Medium, Long and Overall settings §3.2.

Human preference (author-reported), win rate of ViMax over each baseline, overall tab. 2:

Novel-to-video narrative planning, 50 novels, judged by GPT-5.4 on a 1 to 5 scale tab. 3:

Column meanings are in app. A.2. "Avg." is the mean of the six columns app. A.2. The authors note that removing RAG slightly raises Avg. but lowers NF §3.4.

Scheduling and cost (author-reported, 35 stories, API cost only) §3.5 §3.7:

Best-of-k with k=2 costs $17.500 per task at 405 s. k=3 costs $17.770 and 415 s; k=4 costs $18.040 and 425 s §3.7.

Best-of-k (author-reported) §3.7 fig. 5: overall GC is 0.5571 at k=2, 0.5041 at k=3 and 0.5194 at k=4. The authors attribute the drop to more candidates adding selection noise in identity and scene state §3.7.

Language model backbone (author-reported) tab. 4:

The authors conclude GPT-5.4 gives the best trade-off §3.6. The GPT-5.4 row equals the ViMax row of tab. 3.

Paper vs code

The official repository is HKUDS/ViMax, linked from the abstract, MIT-licensed. Cloned at commit fd4b72e7731be6e5f88486206d25ea59849b0285. Read: agents/camera_image_generator.py, pipelines/script2video_pipeline.py, pipelines/novel2movie_pipeline.py, agents/best_image_selector.py, utils/image_selection.py, agent_runtime/config.py, and the configs and tests cited below.

Models differ from the appendix. The paper's default language backbone is Gemini-3-Flash app. A.4 app. A.5. The shipped script2video config uses google/gemini-2.5-flash-lite-preview-09-2025 script2video.yaml. The agent default is gpt-5.5 config.py. The video default is Veo via Google's API script2video.yaml, and veo3.1-fast config.py. The paper uses Wan2.2 for shot video app. A.4.

Reranker differs. The paper uses Qwen3-Rerank app. A.4. The code default is BAAI/bge-reranker-v2-m3 config.py.

Best-of-k selection is not score-based. The paper picks the candidate with the highest VLM score eq. 4. The code asks the selector for best_image_index and a reason, with no per-candidate score best_image_selector.py. The code also skips the VLM step when only one candidate is valid script2video_pipeline.py, and rejects candidates that are not landscape script2video_pipeline.py. The paper states neither.

Transition views. Eq. 8 extracts views for both cameras from the transition eq. 8. The code instead takes one "new camera" image from the transition, which is the first frame of the second detected scene, or the last frame if no second scene is found camera_image_generator.py script2video_pipeline.py. The parent's view is the existing first frame of the parent shot script2video_pipeline.py. The transition prompt is a generic two-shot cut prompt that asks for a consistent style, and includes both shot descriptions camera_image_generator.py. The paper does not state its prompt.

Missing information triggers a reference step. If the parent fully covers the child, the new-camera image is copied as the child's first frame. Otherwise, reference selection and generation run, and the missing elements are passed as text script2video_pipeline.py script2video_pipeline.py script2video_pipeline.py. The paper describes only the anchor path §2.3.1.

Camera tree built by an LLM. The paper describes a graph build eq. 7. The code asks the chat model for a parent camera, parent shot, reason, coverage flag and missing info for each camera camera_image_generator.py camera_image_generator.py. Cycles are rejected camera_image_generator.py. The system prompt encodes shot-size rules, including a ban on direct long-shot-to-close-up parent-child transitions unless unavoidable camera_image_generator.py.

Shots are grouped into cameras by a field from the storyboard. The cam_idx field on each shot decides the grouping script2video_pipeline.py shot_description.py. The paper says shots are assigned to cameras during planning §2.3.1.

Priority index uses the wrong key. The pipeline builds priority_shot_idxs from camera.parent_cam_idx (camera indices) script2video_pipeline.py. It then compares them with shot indices script2video_pipeline.py. A helper that builds the list from parent_shot_idx exists script2video_pipeline.py and is covered by a test test_wrong_output_guards.py, but the pipeline does not call it. The effect is on task order inside one camera. Dependencies are still enforced by the wait on the parent's first-frame event script2video_pipeline.py.

Last frame only for some shots. The code generates a last frame, and feeds it to the video model with the first frame, only when the shot is "medium" or "large" in variation script2video_pipeline.py script2video_pipeline.py. The paper does not describe this.

Narrative RAG. The code chunks the novel at 512 with overlap 128 novel2movie_pipeline.py. For each event query it takes 10 nearest chunks, reranks them and keeps the top 10, summing scores across queries novel2movie_pipeline.py. Retrieved chunks are passed to scene extraction novel2movie_pipeline.py. The paper's Eq. 6 names an Enrich step eq. 6 but does not state chunk sizes or the number of retrieved passages.

Keyframe resolution. The code requests 1600x900 keyframes script2video_pipeline.py camera_image_generator.py. The paper does not state a resolution.

Searched for code. Step 1 found a link in the abstract to github.com/HKUDS/ViMax. The paper itself links it, which meets the rule for official code, so no GitHub search was run and no third-party repository was considered.

Pitfalls

• Consistency metric. GC and CC are cosine similarities between ViCLIP shot features app. A.2. They measure feature stability, not narrative correctness. The authors' own limitations say that representation-based similarity scales well but handles fine-grained narrative intent and multi-person cases poorly [Limitations].
• Table 1 names. The paper does not say which model each name part stands for. The best row ("ViMax-Gemini-3-Pro-Veo 3.1") has the same overall CC/IC/GC (0.5500 / 0.5772 / 0.5571) as the "Gemini-3-Flash" row of tab. 4 and the best-of-2 default in §3.7. The appendix names Gemini-3-Flash as the language model and Gemini-3-Pro-Image as the keyframe model app. A.5. Treat the mapping as inferred.
• Which LLM produced Table 3. The appendix says the narrative-planning runs use GPT-5.4 app. A.4. The default in app. A.5 is Gemini-3-Flash. The GPT-5.4 row of tab. 4 equals the ViMax row of tab. 3.
• Best-of-k is not monotone. GC drops at k=3 (0.5041) and rises again at k=4 (0.5194), both below k=2 §3.7.
• Undefined ablation name. §3.4 calls "ViMax w/o PC" the removal of the "Event-Aware Process Chain". The method section never defines this component. Implement only what §2.2 describes, and treat PC as unspecified.
• Cost and latency. Costs cover API usage only, measured end to end §3.5. They are not hardware costs. Cost rises with k §3.7; the paper does not give a per-shot cost.
• Model dependence. The system depends on proprietary models, and the authors say results are sensitive to model updates [Limitations].
• Known generator failure modes. The keyframe generator struggles in crowded scenes with contact, and the video generator may not respect keyframe semantics [Limitations]. Errors cascade downstream [Limitations].
• Rate limits. The shipped config allows 2 image and 2 video requests per minute and 50 per day each script2video.yaml script2video.yaml. Full runs take many requests.
• Resumable state. The pipeline caches frames, transition clips and selection records on disk and skips existing files script2video_pipeline.py camera_image_generator.py. A stale cache can mix results from different configs. The code checks a request fingerprint for image selection script2video_pipeline.py.
• Appendix labels. The appendix section labels (A.1 to A.6) are read from the LaTeX source. The compiled PDF was not checked.
• Scope. Audio, dialogue sync, shot-level pacing and interactive revision are not covered [Limitations].

Lineage

• Builds on: none stated.
• Related systems named in the paper: FilmAgent, Kubrick and VideoGen-of-Thought (multi-agent video production); StoryDiffusion, ConsiStory and STAGE (consistency); StreamingT2V and FreeLong (streaming) §1.
• Compared against: Wan2.2, Veo 3.1, StoryDiffusion, IC-LoRA, HoloCine, MovieAgent, AniMaker §3.1 app. A.3; RAG-Anything for narrative planning §3.1.
• Superseded by: not known from wiki content.