NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

In plain words

A robot is given a route in words, such as "walk past the sofa and stop by the door", and has to follow it in a house it has never seen. Many earlier systems rely on extra inputs: a depth sensor, wheel odometry to track where the robot has moved, or a floor map. NaVid tries to do the job with one ordinary camera. It watches the video from that camera, reads the instruction, and writes in plain text what the robot should do next, such as "move forward" or "turn left". A model that understands both images and text does the reading and the writing. To keep its input small, it stores only a few compressed tokens for each past frame and a richer view of the current frame. It was trained on navigation demonstrations plus general video data. The authors report that it does well in simulation and in four indoor rooms with a small real robot.

TL;DR

• Input: monocular RGB video plus a text instruction. Output: one text action per decision (stop; move forward with a distance; turn left or right with an angle) §IV-A §IV-B.
• Built on the LLaMA-VID video VLM with a Vicuna-7B language model §IV-A §IV-D.
• Training mixes 320k oracle R2R steps, 180k DAgger steps and 10k instruction-reasoning samples, 510k in all §IV-C. A 763k web-data count appears only in the abstract.
• Author-reported: R2R val-unseen SR 37.4 and SPL 35.9 with RGB only tab. I; zero-shot RxR val-unseen SR 23.8 and SPL 21.2 tab. II.
• Released code is MIT-licensed. The gaps between paper and code are large (see Paper vs code): the instruction-queried tokens are not in the released token path, and the released agent runs up to three unit actions per model call.

Problem

• Vision-and-language navigation in continuous environments (VLN-CE): the agent must follow a free-form instruction in an unseen scene, using low-level actions (forward, turn, stop) rather than teleports between graph nodes §III §I.
• The authors argue that depth, odometry and map inputs add noise and sim-to-real gaps, and that generalisation to new scenes and to the real world is the open problem §I [abstract].
• Formulation: at time t the agent sees the instruction I and frames x_0..x_t, and must pick the next low-level action from {FORWARD, TURN-LEFT, TURN-RIGHT, STOP} with its argument. The paper frames this as a POMDP §III.
• Success: an episode counts when the agent calls STOP within 3 m of the goal in simulation, or within 1.5 m in the real world §V-A.

Method

Architecture §IV-A fig. 2. A vision encoder (EVA-CLIP in the paper) turns each frame into patch tokens. A projector maps them into the language model's embedding space. A Vicuna-7B language model reads them together with the instruction tokens and writes the action as text. The paper says its main contribution is the task-specific token design on top of LLaMA-VID, not the architecture §IV-A.

Per-frame tokens §IV-B eq. 1 eq. 2 eq. 3.

1. Each 224-pixel frame gives N_x = 256 patch embeddings (a 16 x 16 grid) §IV-B.
2. A Q-Former-style query generator conditions on the instruction and produces M queries per frame eq. 1. Cross-attention between those queries and the patches gives one instruction-queried token per frame, after averaging over the queries eq. 2. The paper gives no value for M §IV-B.
3. Instruction-agnostic tokens come from grid pooling of the patches, then a projector eq. 3. The current frame gets 64 tokens (an 8 x 8 grid), and each history frame gets 4 tokens (a 2 x 2 grid). The stride is H/2 for the 2 x 2 case §IV-B app. A.3.
4. The paper says LLaMA-VID's two tokens per frame do not meet the needs of quantitative robot actions, so it keeps a finer grid for the current frame §IV-B.

Input layout §IV-B. Special tokens mark the blocks. History tokens sit between the history markers, current-frame tokens between the observation markers, and a navigation marker precedes the instruction. The paper's template is: history block, observation block, navigation marker, instruction; the answer follows. The paper names the markers HIS, OBS and NAV.

Action output §IV-B. The answer is a text sentence with an action type (FORWARD, TURN-LEFT, TURN-RIGHT or STOP) and, where needed, a distance or rotation. A regular-expression parser extracts both from the text §IV-B.

Training data §IV-C fig. 3.

• Oracle trajectories: step-wise samples (video, instruction, next oracle action) from the R2R training split on 61 MP3D scenes, 320k samples §IV-C.
• Non-oracle (DAgger-style) trajectories: a model trained on the oracle data is run in VLN-CE, and the step samples it produces are labelled with the shortest-path next action, 180k samples §IV-C.
• Instruction reasoning: given a trajectory video, the model writes the instruction, 10k trajectories §IV-C.
• Co-training with video question-answering data, following LLaMA-VID; the paper does not detail it §IV-C.
• Frames: video-caption data sampled at 1 FPS; navigation samples keep every frame, mostly fewer than 300 §IV-D.

Training configuration §IV-D. 24 A100 GPUs for about 28 hours (672 GPU-hours). EVA-CLIP, the Q-Former, BERT and Vicuna-7B start from pretrained weights. Only the LLM and the text encoder are optimised, for one epoch §IV-D. Learning rate, batch size, optimiser, schedule and loss are not stated.

Inference loop §IV-B §IV-D agent_navid.py. The released agent calls the model once per decision. It runs the answer through the parser and expands it into unit actions, which it then executes one by one.

# One decision of NaVid  [§IV-B] [code: navid/model/navid_arch.py#L200-L222]
# Input:  frames (all RGB frames so far, last = current), instruction
# Output: text answer from the language model

def decide(frames, instruction):
    # Patch embeddings for every frame  [§IV-B]
    patches = [vision_encoder(frame) for frame in frames]  # each (256, C)

    # Every frame, including the current one: 2 x 2 pool, 4 tokens each
    # [code: navid/model/navid_arch.py#L218]
    frame_blocks = [project(pool(p, grid=2)) for p in patches]  # (4, d)

    # The current frame also gets an 8 x 8 pool, 64 tokens
    # [code: navid/model/navid_arch.py#L214-L217]
    current_block = project(pool(patches[-1], grid=8))  # (64, d)

    # Video block with separators, then current block, then the
    # navigation marker and the instruction
    # [code: navid/model/navid_arch.py#L343-L361]
    tokens = assemble(frame_blocks, current_block, instruction)

    # Sampling, not greedy decoding  [code: agent_navid.py#L132-L133]
    return language_model.generate(
        tokens,
        do_sample=True,
        temperature=0.2,
        max_new_tokens=1024,
    )

# Parse the text answer  [code: agent_navid.py#L151-L175]
# Output: (action, number); action is STOP, FORWARD, LEFT, RIGHT or None

def parse(answer):
    # The released agent drops the last character first  [agent_navid.py#L273]
    text = answer[:-1]
    # "stop" anywhere in the text wins  [code: agent_navid.py#L154]
    if "stop" in text:
        return STOP, None
    # Otherwise the first verb found in this order sets the action
    options = [("forward", FORWARD), ("left", LEFT), ("right", RIGHT)]
    for verb, action in options:
        if verb in text:
            # The first integer in the text is the argument (cm or degrees)
            return action, first_integer(text)  # None if no integer
    return None, None

# Turn a parsed answer into queued unit actions
# [code: agent_navid.py#L278-L299]
# One unit is 25 cm forward or 30 degrees of turn; at most 3 units per answer

def expand(action, number):
    queued = []
    if action == STOP:
        queued.append(STOP)
    elif action == FORWARD:
        # int() truncates: under 25 cm gives zero units
        # [code: agent_navid.py#L281]
        queued += [FORWARD] * min(3, int(number / 25))
    elif action in (LEFT, RIGHT):
        # Under 30 degrees gives zero units [code: agent_navid.py#L285-L289]
        queued += [action] * min(3, int(number / 30))
    if action is None or not queued:
        # Fallback: a random forward or turn  [code: agent_navid.py#L292-L293]
        queued.append(random.randint(1, 3))
    return queued

The model is not called again until the queue is empty agent_navid.py.

Real-world system §V-A app. A.3. A Turtlebot4 with a Kinect DK camera. NaVid uses only the RGB stream. A remote server runs the model on an A100 and sends commands back to the robot. Each action takes about 1.2 to 1.5 seconds §IV-D.

Hyperparameters

Results

Author-reported. Setup: VLN-CE R2R (trained on the 10,819-episode R2R train split, tested on 1,839 val-unseen episodes) and RxR (zero-shot, tested on 1,517 val-unseen episodes), plus a real Turtlebot4 in four rooms §V-A. Only low-level-action rows are shown. Baselines marked as high-level action in tab. I are omitted. Metrics: TL, trajectory length; NE, navigation error from the goal (lower is better); OS, oracle success; SR, success rate; SPL, success weighted by path length.

VLN-CE R2R val-unseen tab. I.

The text says NaVid is state of the art on SPL, NE and OS among these low-level rows, and comparable on SR §V-B. The SR column shows WS-MGMap at 38.9 against NaVid's 37.4. The text reports a 1.6-point SPL margin over WS-MGMap.

Zero-shot VLN-CE RxR val-unseen, trained on R2R only tab. II.

Subset of 100 R2R val-unseen episodes, LLM and VLM baselines tab. III. Chosen because the baselines are expensive to run. The paper says Emu, LLaVA and LLaMA-VID give no valid actions.

Navigation-history representation tab. IV. Same training data and special-token scheme, with different ways to store history. These are the authors' own re-implementations.

Text-based history takes about 2.7 seconds per step, roughly twice the video version §V-B.

LM-Nav comparison on RxR val-unseen tab. V. LM-Nav receives the ground-truth landmarks and, in place of its visual navigator, a ground-truth shortest path. The table reports OS and SR only.

Real world, four rooms, 25 simple and 25 complex instructions each tab. VI. Cells are success rate (SR) for simple and complex instructions; NE is omitted here. Each cell is out of 25.

Seq2Seq and CMA reach 0% to 8% in every cell tab. VI. The text says NaVid completes about 84% of simple and about 48% of complex instructions. The table averages to 85% and 47%.

Ablations on R2R val-unseen tab. VII.

Visual tokens per frame tab. VIII. Average time per action as reported; hardware not stated.

Other results in the text.

• Long horizons: success stays flat from short to long episodes. About 5.91% of training episodes exceed 90 steps, and about 74.3% fall in the 35 to 75 step range §V-B fig. 4.
• Data scale: with pre-DAgger data below about 280k samples, progress is slow, and a clear jump comes at about 330k. Post-DAgger gains are small §V-D fig. 8.
• Zero-shot object-goal navigation on HM3D is reported in the appendix app. A.5 and is not tabulated here.
• Single-step prediction on samples built from R2R val-unseen: 69.4% success, 52.1% stop success, 6.82 degrees mean angle error, 16.7 cm mean distance error app. A.5. The paper ties this to 37.4% SR for the full task.

Paper vs code

Official repository jzhzhang/NaVid-VLN-CE, MIT license, checked at commit ce2f804a43fb23f1d52431ec24be1acf74c3f4f4. The owner name matches the first author, and the paper's project page links this repository (pku-epic.github.io/NaVid). The repo has the evaluation agent, Habitat extensions, the model code and a training script. The only NaVid shell scripts are evaluation scripts. No command for the NaVid training run and no script that builds the oracle or DAgger data is in the repo.

• Instruction-queried tokens are not in the released token path. The paper builds them with a Q-Former-style cross-attention eq. 1 eq. 2 §IV-B. The released token generation only grid-pools patch features and projects them navid_arch.py. The Q-Former options in the training script are commented out train.py. In vlm_attention, the prompt text only flags the navigation prompt and checks its shape; no query branch depends on it navid_arch.py.
• The current frame gets extra tokens. The paper gives 64 tokens for the current frame and 4 for each history frame fig. 2 §IV-B. In navigation mode the released code pools every frame, including the current one, to 4 tokens inside the video block navid_arch.py. It also adds a 64-token block for the last frame after the video block navid_arch.py navid_arch.py. The current frame therefore appears twice, once at 4 tokens and once at 64. I read this from the code path and did not run it.
• Special-token names and layout differ. The paper uses HIS, OBS and NAV markers §IV-B. The agent uses <video_special>, </video_special>, <image_special>, </image_special>, [Navigation] and <image_sep> agent_navid.py. It also puts <image_sep> between frame blocks agent_navid.py.
• The paper does not say how a distance or angle becomes an action. The released agent uses 25 cm and 30 degree units, at most three per answer agent_navid.py agent_navid.py. The simulator is set to 0.25 m and 30 degrees vlnce_task_navid_r2r.yaml.
• Multi-unit actions run without a new model call. The paper describes one next-step prediction per decision §IV-B. The agent queues up to three units and executes them before asking the model again agent_navid.py agent_navid.py.
• Fallback to random actions. The paper says the regular-expression parser gives 100% valid actions on R2R val-unseen §IV-D. The agent instead picks a random forward or turn when parsing fails or yields zero units agent_navid.py. The agent keeps no count of how often this fires, so the 100% claim cannot be checked from this code.
• Decoding and prompt are not in the paper. The agent samples with temperature 0.2 and uses a fixed prompt agent_navid.py agent_navid.py. Repeat runs will differ.
• The released RxR checkpoint was trained on RxR too. The paper's RxR result is zero-shot after training on R2R only §V-B. The evaluation script loads a checkpoint named r2r-rxr-training-split eval_navid_vlnce.sh, and the README says it was trained on the R2R and RxR training splits, following the Uni-NaVid recipe README.md. That checkpoint cannot reproduce the zero-shot RxR setting.
• Table V and Table II disagree on NaVid's RxR numbers. Table V gives OS 35.5 and SR 23.5 tab. V. Table II gives OS 34.5 and SR 23.8, and the text uses 23.8 tab. II §V-B.
• Real-world text and table disagree. The text gives about 84% and about 48%. The table averages to 85% and 47% tab. VI.
• Vision encoder naming. The paper says EVA-CLIP. The README names EVA-ViT-G weights README.md. The encoder code is navid/model/multimodal_encoder/eva_vit.py.
• Training hyperparameters are absent. No learning rate, batch size, schedule or loss appears in the paper or the repo. The training script exposes a flag for tuning the vision encoder, which defaults to off train.py.
• Evaluation early stopping is not in the paper. The configs stop after 25 rotations and 400 steps on R2R, and after 25 rotations and 500 steps on RxR navid_r2r.yaml navid_rxr.yaml.

Pitfalls

• Unit conversion drops small answers. Under 25 cm or under 30 degrees gives zero units, so the agent takes a random action instead. A "turn left 15 degrees" answer therefore becomes a random action agent_navid.py.
• The last character is dropped before parsing. An answer that ends without a full stop can lose its last digit, so "right 30" becomes "right 3", which then gives zero units agent_navid.py.
• Word order decides the action. Any answer containing "stop" becomes STOP, even when it also names a turn or a distance agent_navid.py.
• The whole history goes through the language model at every decision. Each call rebuilds the full input sequence, with 4 tokens per past frame plus 64 for the current frame and the text. The agent caches vision features for old frames but keeps no key-value cache across calls agent_navid.py agent_navid.py. Episodes run up to 500 steps vlnce_task_navid_r2r.yaml. Reported cost per action grows with tokens per frame: 0.87 s, 1.22 s and 2.72 s for 1, 4 and 16 tokens tab. VIII.
• Memory grows without a cap. The agent keeps every past frame in one tensor and never truncates it agent_navid.py. The paper gives no maximum history length.
• Loss and token masking are not stated. The preprocessing code sets the non-answer tokens of each target to an ignore index train.py, which suggests the loss covers only answer tokens. The paper does not say this.
• Data mixture is incomplete. The paper gives 510k samples in the body and a 763k web-data count only in the abstract. Mixing weights, sampling rules and the DAgger rollout settings are not stated.
• Baseline sources are unclear. The paper does not say whether the RGB-only Seq2Seq and CMA rows were rerun by the authors or copied. The VLN-CE baseline numbers for the other rows come from prior work tab. I.
• Token-count ablation is not the same as the main design. Table VIII varies the count for every frame, while the main design uses 64 for the current frame and 4 for history. The paper does not explain how the two relate tab. VIII.
• Real-world evidence is small. One robot, four rooms, 25 instructions per type per room, no variance reported §V-A tab. VI.
• Real-world latency is per action. The robot waits 1.2 to 1.5 s per action on a remote server. The paper suggests quantisation as a fix and reports no test of it §IV-D.

Lineage

• Builds on: LLaMA-VID, a video vision-language model whose token design and co-training recipe NaVid reuses §IV-A §IV-C. Also DAgger, the imitation method behind the non-oracle data §IV-C. LLaMA-VID has no card here, so it is named but not linked.
• Compared against: Seq2Seq and CMA (VLN-CE baselines), WS-MGMap, LM-Nav, NavGPT-style GPT-4 planners, A2Nav, GPT-4V, LLaVA, Emu and LLaMA-VID §V-B.
• Follow-up in the released repo: Uni-NaVid, arXiv:2412.06224, named in the repository README README.md. It has no card here.
• Superseded by: not known from content on this wiki.