wikillm

Concept · 1 paper · updated 2026-11-05

Vision-language navigation

An agent follows a natural-language instruction through a scene using camera views, in continuous rooms (VLN-CE) with low-level moves. ETPNav builds a topological map online, plans over it with a graph-aware transformer, and adds an escape routine for wall deadlocks. Trade-offs: it needs ground-truth pose, pre-training on discrete graphs, and shortest-path work that grows with the map.

navigationembodied-aiinstruction-followingvision-languagehabitat

Markdown4 min read

AI-compiled overview, updated 2026-11-05, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

Overview

In vision-language navigation (VLN), an agent receives a natural-language instruction and must move through a scene to the place it describes, using what it sees at each step. In the continuous variant (VLN-CE), the agent does not hop between nodes of a predefined graph. It moves with low-level actions on a 3D mesh: a fixed forward step, a fixed turn, and STOP 2304.03047 §3.

The shared difficulty is that a local view does not show the whole route. The agent must remember where it has been and sometimes return to an earlier junction. Modular methods handle this by predicting nearby waypoints and choosing one, but local waypoints give a poor picture of the global layout 2304.03047 §1. The variant below keeps a map of the whole episode.

Coverage note: this page currently draws on the one paper card tagged with this concept. It will be extended as more VLN cards are tagged.

Variants

Online topological map with graph-aware planning (ETPNav)

The agent builds a graph of visited places and "ghost" nodes (seen but not yet visited) as it moves. Nodes come from waypoints predicted from depth, and a waypoint that lands near an existing node is merged into it rather than added again 2304.03047 §3.1. A cross-modal transformer scores each ghost against the instruction. Its attention adds a learned bias based on the shortest-path distance between nodes, so the scores reflect the map's shape as well as visual similarity 2304.03047 §3.2 2304.03047 eq. 1. The chosen ghost is reached by a rotate-then-forward controller along a shortest path on the graph 2304.03047 §3.3. Card: ETPNav.

Reported results, author-reported: on R2R-CE test unseen, 55 SR and 48 SPL against 49 SR and 45 SPL for the strongest listed prior method, Reborn 2304.03047 tab. 2. On RxR-CE test unseen, 51.21 SR and 39.86 SPL 2304.03047 tab. 4.

When to use which

Only one variant is covered so far, so the choice is between using this design or not. Use the constraints below to decide whether its assumptions hold.

SituationUseWhy
Long routes where the agent must backtrack to an earlier placeGlobal map with graph-aware planningOn R2R-CE val unseen, global planning with graph attention reached 57.21 SR, against 53.15 SR for local planning 2304.03047 tab. 7
Sliding along walls is forbidden (larger agent radius)Add an escape routine for stuck forward stepsWithout it, the RxR-CE SDTW drops from 45.33 to 18.64 2304.03047 tab. 10. The paper's escape only runs in that setting 2304.03047 §3.3
Only odometry or noisy pose is availableNot established by this paperThe method uses the simulator's ground-truth pose for mapping and control. Noisy pose is listed as future work 2304.03047 §3 2304.03047 §5
No pre-training data on a discrete graphTrain from scratch with a drop in accuracyFrom scratch: 37.41 SR. With MLM and SAP pre-training: 57.21 SR 2304.03047 tab. 8
Many steps or a large mapCap the map size and tune the localisation thresholdThe threshold γ sets the node count. Larger γ means fewer nodes but a sharp SR loss 2304.03047 tab. 6. Shortest paths are recomputed over the whole graph at each step 2304.03047 §3.2
Few human-written instructions availableExpect to need synthetic instructionsThe RxR-CE main rows use synthetic instructions. Without them, the paper reports lower numbers (†) 2304.03047 tab. 4

Teacher rules differ between datasets. The R2R-CE teacher picks the ghost nearest the goal, and the RxR-CE teacher follows the annotated reference path 2304.03047 §3.4. Changing datasets therefore changes the training signal, not only the data.

Implementation sketch

The loop below follows the paper's description. Names are ours. Each step states the section it comes from.

Pseudocode
# One VLN-CE episode with an online topological map  [arxiv:2304.03047 §3]
# Input:  instruction tokens; step limit (15 for R2R-CE, 25 for RxR-CE)
# Output: a low-level action sequence ending in STOP

def navigate(instruction, step_limit):
    # Encode the instruction once  [arxiv:2304.03047 §3.2]
    text = text_encoder(instruction)
    graph = empty_map()
    for _ in range(step_limit):
        # Predict waypoints from depth only  [arxiv:2304.03047 §3.1]
        candidates = waypoint_predictor(depth_views)
        # Merge each waypoint into a nearby node or open a new ghost  [§3.1]
        graph = localise_and_update(graph, candidates)
        # Score ghosts and STOP against the instruction  [§3.2] [eq. 1] [eq. 2]
        scores = graph_transformer(text, graph)
        if best(scores) is STOP:
            return STOP
        goal = best(scores)
        # Remove the goal from the map so it is not picked again  [§3.3]
        graph.remove(goal)
        # Drive along the graph path with a rotate-then-forward controller  [§3.3]
        follow(shortest_path(graph, goal))
    return STOP

When a forward step does not move the agent in a sliding-forbidden setting, the escape routine tries a fixed set of headings with one step each and resumes the controller once it leaves the dead end 2304.03047 §3.3. The paper gives seven headings from −90° to 90° 2304.03047 §3.3. For the code's version, see the ETPNav card.

Training uses student forcing with a teacher that supplies the next target. The loss is the summed negative log-likelihood of that teacher node over decision steps 2304.03047 §3.4 2304.03047 eq. 3.

Contradictions

  • Training versus evaluation back-paths. The paper says each planned path is executed by the rotate-then-forward controller 2304.03047 §3.3. The released training setup teleports along back-paths, while evaluation uses the controller. The paper does not discuss this difference. See the ETPNav card, "Paper vs code".
  • Teacher-forcing schedule. The paper describes a decaying teacher probability with a decay every 3,000 iterations and ratio 0.75 2304.03047 §4.1.3. The code's schedule starts the first 3,000 iterations at 0.75 [see the card, "Paper vs code"].
  • Paper-internal numbers. The text and the table give different values for the teleport upper bound on RxR-CE NDTW (64.43 against 64.33) and for Reborn's test SDTW (38.43 against 38.42) 2304.03047 §4.3.3 2304.03047 tab. 10. These are small, but compare with the table.

Papers

Related

ConceptTopological map navigationMoves an agent over a graph of places it has seen. LM-Nav matches route landmarks to nodes and finds the cheapest walk through them. SAME learns a policy over a graph. ETPNav builds its graph online in continuous rooms. NoMaD samples local moves from a diffusion policy. Trade-offs: LM-Nav needs a map and recognisable landmarks; SAME and ETPNav need training data; NoMaD takes photo goals.ConceptImitation learningLearning a policy from recorded behaviour instead of a hand-written reward: behaviour cloning, nearest-neighbour lookup, return-conditioned sequence models, imitation terms in offline RL, vision-language-action and tactile policies, planning from logged video, imitating solver branching, expert hints for LLM agents and style imitation, with the trade-offs in data, compute and evaluation.ConceptGraph transformerA graph transformer runs self-attention over a graph's nodes plus structure: attention biases (Graphormer, ETPNav), positional vectors (LSPE), linear-cost latent-graph attention (NodeFormer), a predicted relation graph (D3G), typed grid attention (GENCO), overlap weights (TransMOT), pooled patches (GTP) or a denoiser over agent graphs (GTD). Full attention grows with the square of node count.ConceptVision-language-action modelA policy that takes camera images and a language instruction and outputs robot actions, usually by building on a pretrained vision-language backbone. Covers action heads (tokens, regression, diffusion, flow matching), memory, cross-embodiment and generalist training, fine-tuning recipes, joint video-action models, failure detection, supervision and rollout evaluation.ConceptMultimodal language modelA language model that reads images, video, audio or other signals next to text and answers in text. Designs differ in how the non-text input enters the model, which parts train, and whether the output can also be masks or boxes: a frozen LLM behind a small bridge, a natively multimodal decoder, a fully fine-tuned generalist, or one token vocabulary for everything.ConceptRobot simulation for learningTraining and testing robot policies in a simulated world before real hardware. Orbit and Isaac Gym run many physics copies in parallel on one GPU. Replica supplies indoor scenes, and many papers instead score policies on simulated benchmarks such as LIBERO, RoboTwin or RLBench. Choose a simulator by rendering needs, deformable objects and GPU budget.

Source and license

Compiled from the papers listed on this page, all published under CC BY, CC BY-SA or CC0, with citations. See each paper's page for its authors and license.