Ground Slow, Move Fast: A Dual-System Foundation Model for Generalizable Vision-and-Language Navigation

In plain words

A robot that follows spoken directions through a building has two jobs. It has to work out where to go next from what it sees, which is slow and needs a large model. It also has to steer around chairs and people in real time, which must be quick. DualVLN handles these with two parts that run at different speeds. A large vision-language model acts as the slow planner: it looks at the camera images and the instruction, then points to a spot in the image that it wants to reach next.

A small, fast network acts as the driver. It takes the planner's chosen spot, plus a compact summary of the planner's internal state, and draws a short smooth path for the robot to follow. The planner runs now and then, and the driver keeps updating its path from the newest camera frames in between. The authors train the two parts in sequence, so the large model keeps its general knowledge while the driver learns local steering.

The paper is a report from a research lab, and it ships an official code repository. The results it reports come from simulation benchmarks and a small set of real robots, and they are the authors' own measurements.

TL;DR

• Dual-system VLN: System 2 is a Qwen2.5-VL 7B planner that outputs a pixel goal or a view-adjustment action; System 1 is a diffusion transformer that outputs a 32-waypoint trajectory §3.1 §3.2.
• System 2 gets its image-grounded goal from projected 3D trajectories; System 1 is conditioned on four learned latent tokens taken from System 2's last hidden layer and on a 32-token summary of two RGB frames §3.2 app. A.
• Training is sequential: System 2 first, then System 2 is frozen and System 1 plus the latent queries are trained on pixel-goal samples §1 app. A.
• Author-reported: 64.3 SR and 58.5 SPL on R2R-CE val-unseen with RGB only, against 56.9 SR for StreamVLN tab. 1. On VLN-PE val-unseen, 51.60 SR with no VLN-PE training, against 22.42 for NaVid tab. 2.
• Not reproducible from the paper alone for three reasons: the sampler, the trajectory scale and the sample aggregation are not stated, and they are set in code. See Pitfalls.

Problem

Most VLN-with-VLM systems map camera frames and an instruction straight to a short discrete action, such as "move forward 0.25 m" §1. The authors say this gives jerky motion, high latency because the large model runs on every step, and no clear split between high-level planning and local control, which makes dynamic obstacle avoidance hard §1.

DualVLN aims for an open-ended navigation model that keeps a VLM's generalization, runs in real time, and avoids moving obstacles §1.

Method

System 2: pixel-goal grounding with view adjustment §3.1

System 2 is a Qwen-VL-2.5 model adapted to VLN. It reads a sequence of egocentric frames and the instruction, and it predicts either a view-adjustment action or a 2-D pixel coordinate for the next waypoint fig. 2 §3.1.

• Training targets. Each VLN-CE trajectory is projected onto the egocentric frames. A point counts as visible only when its distance is not larger than the depth at its pixel. The farthest visible waypoint is the pixel-goal target §3.1 app. A.
• View adjustment. When no future waypoint is visible, for example at the start of a sequence of turns, the model predicts turn actions instead. It can turn left or right by 15 degrees, or look up or down by 15 degrees §3.1. The discrete action set is STOP, forward 25 cm, turn left 15 degrees and turn right 15 degrees; at most four turn actions are predicted per chunk app. A.
• Stop. The last step of each sequence is supervised with STOP app. A.

System 1: diffusion transformer with multimodal conditioning §3.2

System 1 generates a trajectory of 32 waypoints. Its conditions are a latent goal from System 2 and high-frequency RGB input §3.2 fig. 2.

Latent goal. After System 2 writes the pixel goal, four learnable tokens <TRAJ> are appended to the text sequence. Their final-layer hidden states are taken as the latent goal app. A §3.3 internvla_n1.py. The latent tokens are initialized randomly and trained by prompt tuning §3.2.

RGB branch. The current frame and the frame from System 2's time step are each encoded by a ViT, the DepthAnythingV2-Small backbone §3.3. The two feature sets are fused with a self-attention encoder. A Q-Former then compresses them into 32 tokens §3.2 internvla_n1_arch.py.

Diffusion transformer. The policy is a DiT with hidden size 384, 12 layers and 6 heads §3.3 nextdit_crossattn_traj.py. The latent goal is first projected from 3584 to 768 dimensions §3.3.

Asynchronous use of latents. System 2 runs at a lower rate than System 1. System 1 is conditioned on the stale latent goal from time t, together with the newest frame at time t+k, so that it can estimate the distance already travelled §3.2 fig. 1.

Flow matching. The training objective is a flow-matching loss eq. 1 eq. 2 eq. 3:

• A noisy trajectory is X_u = alpha_u X_0 + sigma_u eps, where alpha_u falls and sigma_u rises with u eq. 1.
• The network predicts the velocity of the trajectory from X_u, the time u and the concatenated conditions eq. 2.
• The loss is the mean squared error between predicted and true velocity eq. 3.

The paper does not give the noise schedule. The code uses a linear blend and a library scheduler (see Paper vs code).

# System 1 training step, flow matching  [§3.2] [eq. 1-3]
# [code: internnav/model/basemodel/internvla_n1/internvla_n1.py#L263-L286]
def system1_loss(waypoints, latents, frames, valid):
    # Draw one noise level per trajectory, u ~ U(0, 1)  [§3.2]
    u = uniform(0, 1, size=batch)                    # (B,)
    # Map u to the scheduler timestep and its sigma  [code: L263-L266]
    timestep, sigma = schedule_lookup(u)             # (B,), (B, 1, 1)
    # Draw Gaussian noise  [§3.2]
    noise = normal_like(waypoints)                   # (B, 32, 3)
    # Blend trajectory and noise  [eq. 1] [code: L270]
    noisy = (1 - sigma) * waypoints + sigma * noise
    # Embed the noisy waypoints and add their positions  [code: L271-L274]
    features = action_encoder(noisy) + position_embedding(32)
    # Predict the velocity from the time and the conditions  [eq. 2]
    predicted = dit(features, timestep, context=latents)
    predicted = action_decoder(predicted)            # (B, 32, 3)
    # Velocity target is noise minus the clean trajectory  [code: L282]
    target = noise - waypoints
    # Mean squared error over valid frames only  [eq. 3] [code: L283-L286]
    return masked_mean_square(predicted - target, valid)

In the pseudocode below, policy means internnav/model/basemodel/internvla_n1/internvla_n1_policy.py.

# System 1 inference: 32 samples, 10 Euler steps  [§3.2] [app. A]
# [code: internnav/model/basemodel/internvla_n1/internvla_n1.py#L349-L435]
# [code: internnav/model/utils/vln_utils.py#L129-L131]
def generate_actions(latents, frames):
    # Visual memory from the two frames  [§3.2]
    memory = qformer(fuse(frames))                   # (1, 32, 768)
    # Context is visual memory followed by the projected latent goal  [§3.2]
    context = concat(memory, cond_projector(latents))  # (1, 36, 768)
    # Start 32 independent samples from Gaussian noise  [code: L389-L392]
    samples = normal((32, 32, 3))                    # (N, waypoints, 3)
    # Integrate the flow over 10 steps, sigma from 1.0 down to 0.1  [code: L396]
    for step in schedule_10_steps:
        velocity = dit(samples, step, context)
        samples = euler_step(samples, velocity, step)
    # Undo the x, y scale and average the paths  [code: vln_utils L129-L131]
    paths = cumulative_xy(samples with xy divided by 4.0)
    mean_path = mean(paths, over the 32 samples)
    # Convert the mean path to discrete actions, keep the first four  [code: policy#L212-L214]
    return path_to_discrete_actions(mean_path)[:4]

Training recipe §3.3 app. A

Hyperparameters

Values from the paper are cited to the paper. Values marked "code" appear only in the released scripts and config, not in the paper.

Results

Author-reported. Each table below is copied from the paper and cited to it. The paper evaluates on R2R-CE and RxR-CE (VLN-CE, Habitat), on VLN-PE (Unitree H1, physical controller), on its Social-VLN benchmark, and on real robots (CMA, NaVid, NaVILA and StreamVLN as baselines) §5.1 §5.2.

Table 1: VLN-CE, val-unseen tab. 1. Observation columns are marked x when the method uses them: Pano = panoramic RGB, Odo = odometry, Depth = depth, S.RGB = single first-person RGB. * marks methods that use the waypoint predictor from Hong et al.

Table 2: VLN-PE, R2R tab. 2. Physical locomotion controller, Unitree H1. "Train on VLN-PE" rows were trained on VLN-PE; "zero-shot" rows were trained on VLN-CE only. Fall and stuck rates are lower-is-better. Per the paper's caption, + means trained on Habitat and then fine-tuned on VLN-PE (no such row here), and † means trained with data augmentation (the Seq2Seq and CMA zero-shot rows).

Table 3: R2R val-unseen, standard and Social-VLN tab. 3. HCR = human collision rate, lower is better. Social-VLN is the authors' benchmark (see Pitfalls).

Table 4: System 1 versus point-goal local planners, VLN-PE with flash controller tab. 4. The paper's text says these planners get oracle depth for the pixel goal. Rows show that System 1 beats iPlanner and NavDP on both splits.

Ablations of System 1 conditioning fig. 7. The paper compares three variants on one plot: training System 1 and System 2 jointly in one stage (w/o Sys.2 Train), removing the pixel-goal text before the latent queries (w/o Pixel Goal), and using only the frozen last-layer hidden states of the pixel-goal text (w/o Latent Goal). The text reports that each variant is worse and that joint training converges more slowly, but gives no numbers, so the exact values are only in the figure.

Data scaling of System 1 fig. 9. Using 1%, 10% and 50% of the System 2 trajectories: 10% is near saturation and 50% adds little. Only the figure reports the values.

Real-world fig. 5 fig. 6. Hallway, bedroom and office scenarios, 20 trials per scenario per model, measured by success rate and navigation error. The text gives no numbers. The authors describe DualVLN as achieving high SR and low NE in both static and dynamic scenes.

Paper vs code

Code: https://github.com/InternRobotics/InternNav, MIT, commit 7a5c62400ac45b313d9b709c740b64191556a242. The abstract links the repository, and its owner is the InternRobotics organisation, which the paper names as the authors' project §5.2 LICENSE.

Each difference below is a place where the released code and the paper disagree, or where the paper is silent and the code fixes a value.

• Latent projection. Paper: the 3584-dim latent is linearly projected to 768 §3.3. Code: a two-layer MLP with GELU, Linear(3584, 768) -> GELU -> Linear(768, 768) internvla_n1_arch.py.
• Noise blend. Paper: alpha_u X_0 + sigma_u eps with generic decreasing and increasing functions eq. 1. Code: (1 - sigma) X_0 + sigma eps, with sigma taken from a library scheduler internvla_n1.py.
• Time sampling. Paper: u ~ U(0, 1) §3.2. Code: u is mapped to a discrete index on the scheduler's timestep grid, floor(u * num_train_timesteps) internvla_n1.py. The scheduler is built with library defaults internvla_n1_arch.py. The paper states no schedule.
• Loss mask. Paper: plain MSE eq. 3. Code: MSE masked to valid frames and averaged over the valid count internvla_n1.py.
• Trajectory scale. Not in paper. Code multiplies x and y deltas by 4 at data preparation internvla_n1_lerobot_dataset.py and divides by 4 at inference vln_utils.py.
• Inference sampler. Not in paper. Code: 10 Euler steps with sigma from 1.0 to 0.1 internvla_n1.py internvla_n1.py.
• Guidance. Not in paper. Code: a null-condition branch with zeros and guidance scale 1.0 internvla_n1.py internvla_n1.py.
• Aggregation of 32 samples. Paper: "generates 32 trajectories in parallel" §5.2 with no rule for choosing among them. Code: the 32 paths are averaged before conversion to actions vln_utils.py.
• System 2 epochs. Paper: one epoch §3.3. Code: 2.0 epochs train_system2.sh.
• System 2 vision-tower rate. Not in paper. Code: 5e-6 train_system2.sh.
• System 1 schedule. Paper: none stated. Code: cosine with minimum 1e-5, 3 epochs train_dual_system.sh train_dual_system.sh.
• Visual memory. Paper: a self-attention module fuses the two frames, then a Q-Former outputs 32 tokens §3.2. Code: a 3-layer Transformer encoder with learned positions, then a 3-layer Q-Former internvla_n1_arch.py.
• Depth ViT checkpoint. Paper: DepthAnythingV2-Small §3.3. Code: loads the metric-Hypersim ViT-S weights internvla_n1_arch.py.
• Second System 1 type. The repository also has a NavDP-based System 1 with its own loss internvla_n1.py. The DualVLN training script selects the nextdit_async variant train_dual_system.sh, which is the one matching the paper.

Checked and consistent: the latent token count (four), the 32 waypoints, the DiT size (384 wide, 12 layers, 6 heads), the frozen System 2 during System 1 training (tune_mm_* False and pixel_goal_only True) train_dual_system.sh train_dual_system.sh, and System 1 learning rate and batch.

Pitfalls

• Scale factor 4. A reimplementation that skips the x and y scale at either end gets trajectories that are 4 times too long or too short. The paper does not mention it internvla_n1_lerobot_dataset.py vln_utils.py.
• Averaging 32 samples. The paper says the model generates 32 trajectories but does not say how one is used. The code averages the paths. Choosing one sample instead would change the behaviour vln_utils.py.
• Guidance branch. At inference the guidance scale is 1.0, so the null-condition branch has no effect internvla_n1.py. The training forward (L234-L286) shows no condition dropout, so the null branch is never trained. Adding guidance will need a training change as well.
• Scheduler defaults. Sigma values come from the library's default flow-matching scheduler, not from the paper. Pin the library version or you will get a different noise schedule internvla_n1_arch.py.
• Stale latent goal. System 1 is conditioned on the latent from an earlier System 2 step, and it must cope with the robot having moved since. The paper argues it can estimate the distance already travelled §3.2. It reports that System 1 tolerates small pixel-goal errors in the right direction, but not large or semantically wrong goals fig. 8.
• Point-goal baselines are not like-for-like. The comparison in tab. 4 gives the point-goal planners oracle depth. The paper attributes their gap partly to trajectory-distribution shift in System 2 §5.1.
• Social-VLN is not in the repository. Searches for "social", "HCR" and the human-collision pipeline found nothing in the released code. The paper describes the benchmark and a 763K-episode training pipeline built on Habitat 3.0 humanoids and 60 MP3D scenes §4. Treat tab. 3 as unverified.
• Zero-shot VLN-PE claim. DualVLN was not trained on VLN-PE §5.1. The table comparison is valid, but the baselines trained on VLN-PE are the stronger reference in that setting.
• Real-world latency. System 2 takes 0.7 s per output with KV-cache reuse and System 1 takes 0.03 s with TensorRT. These are author measurements on one RTX 4090 §5.2. The asynchronous pipeline is needed to reach the reported behaviour.
• Data and weights. The training data (InternData-N1, linked from the abstract) and the released checkpoint were not checked for licence and are not used on this card.
• Figure and equation numbers. Counted from the LaTeX source order (11 figures, 3 numbered equations, 4 tables). If the compiled PDF numbers differ, cite the section instead.

Lineage

• Builds on: StreamVLN, which supplies the System 2 data recipe §3.3. DepthAnythingV2 provides the ViT §3.3. Qwen-VL-2.5 is the planner §3.1.
• Compared against: NaVid, NaVILA, UniNaVid, StreamVLN, MapNav, ETPNav, GridMM and others on VLN-CE tab. 1; RDP, CMA, Seq2Seq and NaVid on VLN-PE tab. 2; iPlanner and NavDP as local planners tab. 4.
• Part of: the InternVLA-N1 framework, according to the paper's acknowledgments (Contributions and Acknowledgments section). The repository README lists DualVLN as one InternVLA-N1 configuration README.md README.md.
• Superseded by: not known from content already in this wiki.