---
kind: concept
slug: vision-language-navigation
title: Vision-language navigation
summary: >-
  An agent follows a natural-language instruction through a scene using camera
  views, in continuous rooms (VLN-CE) with low-level moves. ETPNav builds a
  topological map online, plans over it with a graph-aware transformer, and
  adds an escape routine for wall deadlocks. Trade-offs: it needs ground-truth
  pose, pre-training on discrete graphs, and shortest-path work that grows with
  the map.
tags: [navigation, embodied-ai, instruction-following, vision-language, habitat]
papers: ["2304.03047"]
related: [topological-map-navigation, imitation-learning, graph-transformer, vision-language-action-model, multimodal-language-model, robot-simulation]
status: draft
updated_at: 2026-11-05
---

## Overview

In vision-language navigation (VLN), an agent receives a natural-language
instruction and must move through a scene to the place it describes, using
what it sees at each step. In the continuous variant (VLN-CE), the agent does
not hop between nodes of a predefined graph. It moves with low-level actions on
a 3D mesh: a fixed forward step, a fixed turn, and STOP [arxiv:2304.03047 §3].

The shared difficulty is that a local view does not show the whole route. The
agent must remember where it has been and sometimes return to an earlier
junction. Modular methods handle this by predicting nearby waypoints and
choosing one, but local waypoints give a poor picture of the global layout
[arxiv:2304.03047 §1]. The variant below keeps a map of the whole episode.

*Coverage note:* this page currently draws on the one paper card tagged with
this concept. It will be extended as more VLN cards are tagged.

## Variants

### Online topological map with graph-aware planning (ETPNav)

The agent builds a graph of visited places and "ghost" nodes (seen but not yet
visited) as it moves. Nodes come from waypoints predicted from depth, and a
waypoint that lands near an existing node is merged into it rather than added
again [arxiv:2304.03047 §3.1]. A cross-modal transformer scores each ghost
against the instruction. Its attention adds a learned bias based on the
shortest-path distance between nodes, so the scores reflect the map's shape as
well as visual similarity [arxiv:2304.03047 §3.2] [arxiv:2304.03047 eq. 1].
The chosen ghost is reached by a rotate-then-forward controller along a
shortest path on the graph [arxiv:2304.03047 §3.3]. Card:
[ETPNav](/papers/2304.03047).

Reported results, author-reported: on R2R-CE test unseen, 55 SR and 48 SPL
against 49 SR and 45 SPL for the strongest listed prior method, Reborn
[arxiv:2304.03047 tab. 2]. On RxR-CE test unseen, 51.21 SR and 39.86 SPL
[arxiv:2304.03047 tab. 4].

## When to use which

Only one variant is covered so far, so the choice is between using this design
or not. Use the constraints below to decide whether its assumptions hold.

| Situation | Use | Why |
|---|---|---|
| Long routes where the agent must backtrack to an earlier place | Global map with graph-aware planning | On R2R-CE val unseen, global planning with graph attention reached 57.21 SR, against 53.15 SR for local planning [arxiv:2304.03047 tab. 7] |
| Sliding along walls is forbidden (larger agent radius) | Add an escape routine for stuck forward steps | Without it, the RxR-CE SDTW drops from 45.33 to 18.64 [arxiv:2304.03047 tab. 10]. The paper's escape only runs in that setting [arxiv:2304.03047 §3.3] |
| Only odometry or noisy pose is available | Not established by this paper | The method uses the simulator's ground-truth pose for mapping and control. Noisy pose is listed as future work [arxiv:2304.03047 §3] [arxiv:2304.03047 §5] |
| No pre-training data on a discrete graph | Train from scratch with a drop in accuracy | From scratch: 37.41 SR. With MLM and SAP pre-training: 57.21 SR [arxiv:2304.03047 tab. 8] |
| Many steps or a large map | Cap the map size and tune the localisation threshold | The threshold γ sets the node count. Larger γ means fewer nodes but a sharp SR loss [arxiv:2304.03047 tab. 6]. Shortest paths are recomputed over the whole graph at each step [arxiv:2304.03047 §3.2] |
| Few human-written instructions available | Expect to need synthetic instructions | The RxR-CE main rows use synthetic instructions. Without them, the paper reports lower numbers (†) [arxiv:2304.03047 tab. 4] |

Teacher rules differ between datasets. The R2R-CE teacher picks the ghost
nearest the goal, and the RxR-CE teacher follows the annotated reference path
[arxiv:2304.03047 §3.4]. Changing datasets therefore changes the training
signal, not only the data.

## Implementation sketch

The loop below follows the paper's description. Names are ours. Each step
states the section it comes from.

```pseudocode
# One VLN-CE episode with an online topological map  [arxiv:2304.03047 §3]
# Input:  instruction tokens; step limit (15 for R2R-CE, 25 for RxR-CE)
# Output: a low-level action sequence ending in STOP

def navigate(instruction, step_limit):
    # Encode the instruction once  [arxiv:2304.03047 §3.2]
    text = text_encoder(instruction)
    graph = empty_map()
    for _ in range(step_limit):
        # Predict waypoints from depth only  [arxiv:2304.03047 §3.1]
        candidates = waypoint_predictor(depth_views)
        # Merge each waypoint into a nearby node or open a new ghost  [§3.1]
        graph = localise_and_update(graph, candidates)
        # Score ghosts and STOP against the instruction  [§3.2] [eq. 1] [eq. 2]
        scores = graph_transformer(text, graph)
        if best(scores) is STOP:
            return STOP
        goal = best(scores)
        # Remove the goal from the map so it is not picked again  [§3.3]
        graph.remove(goal)
        # Drive along the graph path with a rotate-then-forward controller  [§3.3]
        follow(shortest_path(graph, goal))
    return STOP
```

When a forward step does not move the agent in a sliding-forbidden setting,
the escape routine tries a fixed set of headings with one step each and resumes
the controller once it leaves the dead end [arxiv:2304.03047 §3.3]. The paper
gives seven headings from −90° to 90° [arxiv:2304.03047 §3.3]. For the code's
version, see the [ETPNav card](/papers/2304.03047).

Training uses student forcing with a teacher that supplies the next target. The
loss is the summed negative log-likelihood of that teacher node over decision
steps [arxiv:2304.03047 §3.4] [arxiv:2304.03047 eq. 3].

## Contradictions

- **Training versus evaluation back-paths.** The paper says each planned path
  is executed by the rotate-then-forward controller [arxiv:2304.03047 §3.3].
  The released training setup teleports along back-paths, while evaluation
  uses the controller. The paper does not discuss this difference. See the
  [ETPNav card](/papers/2304.03047), "Paper vs code".
- **Teacher-forcing schedule.** The paper describes a decaying teacher
  probability with a decay every 3,000 iterations and ratio 0.75
  [arxiv:2304.03047 §4.1.3]. The code's schedule starts the first 3,000
  iterations at 0.75 [see the card, "Paper vs code"].
- **Paper-internal numbers.** The text and the table give different values for
  the teleport upper bound on RxR-CE NDTW (64.43 against 64.33) and for
  Reborn's test SDTW (38.43 against 38.42) [arxiv:2304.03047 §4.3.3]
  [arxiv:2304.03047 tab. 10]. These are small, but compare with the table.

## Papers

- [ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments](/papers/2304.03047): online topological map, graph-aware planner, and a rotate-then-forward controller with a wall-escape routine for VLN-CE.