End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering

In plain words

A robot with a camera has to walk to an object in a house. The usual approach splits the job into parts: one module builds a map, another plans a path, a third moves the wheels. This paper skips most of that. The software looks at the depth sensor and finds directions the robot can drive in without hitting a wall. It then draws a few numbered red arrows onto the camera picture, one per direction. A vision-language model, a model that reads pictures and text, is shown the marked picture and asked which numbered arrow to take. The robot drives that way, and the model is asked again.

Nothing is trained for this task. The authors use a general-purpose model, and the main work is in how the picture is marked and how the question is worded. A second question, asked on each step, decides when the robot has arrived. The authors report that this beats a plain prompting baseline and an earlier visual-prompting method on object-search benchmarks. Specialised trained systems still do better on the harder benchmark.

TL;DR

• A VLM is used as an end-to-end navigation policy with no fine-tuning. Each step it chooses one of several numbered actions drawn on the RGB image §3 §3.4.
• Candidate actions come from a depth-based navigability test, a top-down map that marks explored and unexplored areas, and spacing rules that favour unexplored directions §3.1 §3.2.
• Stopping uses a separate VLM question on the un-annotated image. The episode ends after two consecutive "stop" answers §3.5.
• Author-reported: ObjectNav SR 50.4% and SPL 0.210 against 24.6% and 0.106 for PIVOT; GOAT SR 16.3% against 8.3% for PIVOT, but below the two specialised systems in the paper §4.1 §4.2.
• The released code and the paper differ in several numeric details. Read "Paper vs code" before implementing.

Problem

Modular navigation stacks need a map module and a planner, and they add complexity and task specialisation §1. VLMs answer questions about images well but struggle with fine spatial output such as obstacle distances and coordinates. Feeding them a task text plus an observation-action history does not reliably produce usable navigation §1. Earlier zero-shot prompting methods such as PIVOT sample actions from a Gaussian, need many VLM calls per action, and ignore depth when judging whether an action is feasible §2.

The method assumes a simulator with ground-truth depth and pose, and collision-free sliding along walls (the allow_slide setting). The authors say that disabling sliding makes performance drop sharply §4 §4.1 §4.2.

Method

The agent runs four components on each observation: navigability, action proposer, projection and prompting §3. A second thread runs termination checks §3.5. The code uses one VLM call for the action and one for stopping, both on every step agent.py.

Navigability and the voxel map §3.1

The depth image is turned into a height map, and pixels close to the floor form the navigability mask agent.py. Directions are sampled across a sweep that is 1.5 times half the field of view, in 60 steps agent.py. Along each direction, a 2 m ray is traced through the image until it leaves the navigable pixels agent.py. The traced distance gives a candidate action (radius, angle) §3.1.

The voxel map is a top-down grid with 100 pixels per metre ObjectNav.yaml. Each candidate marks a thick "unexplored" line out to its radius minus 0.5 m, and an "explored" line out to a shorter distance agent.py.

# Navigability: find collision-free rays  [§3.1] [code: src/agent.py#L200-L233]
def navigable_actions(depth, pose, fov_deg):
    # Pixels whose height is near the floor are navigable  [code: src/agent.py#L382]
    mask = floor_mask(depth, pose)
    # Sweep angles over 1.5 times half the field of view, 60 steps  [code: src/agent.py#L215-L217]
    sweep = radians(fov_deg / 2) * 1.5
    actions = []
    for theta in linspace(-sweep, sweep, 60):
        # Walk a 2 m ray until it reaches a blocked pixel  [code: src/agent.py#L408-L438]
        radius = walk_ray(mask, depth, pose, theta, length=2.0)
        if radius is None:
            continue
        # Record the ray in the voxel map  [§3.1] [code: src/agent.py#L227-L230]
        update_voxel_map(radius, theta, pose)
        actions.append((radius, theta))
    return actions

Action proposer §3.2

Each candidate is classified as explored or unexplored by looking up the top-down map at a point 0.8 times its length along its angle agent.py. Candidates are shortened to 2/3 of their length, and very short ones are dropped agent.py agent.py.

Selection then has two passes. The first adds unexplored candidates, starting from the longest, with an angular gap of at least 0.9 units. The second adds explored candidates with a larger gap of 4 units (explore_bias). One unit is fov / 360 radians agent.py agent.py. Finally, lengths are capped at 1.7 m ObjectNav.yaml.

# Action proposer: spaced, exploration-biased actions  [§3.2]
# [code: src/agent.py#L235-L323]
def propose_actions(actions, voxel_map, fov_deg):
    # One angular unit, in radians  [code: src/agent.py#L237]
    unit = fov_deg / 360
    candidates = []
    for radius, theta in actions:
        # Shorten to two thirds of the length  [§3.2] [code: src/agent.py#L262]
        length = 0.66 * radius
        # Unexplored if the map is not yet explored there  [§3.2] [code: src/agent.py#L257-L262]
        unexplored = is_unexplored(voxel_map, 0.8 * radius, theta)
        if length > 0.75:  # [code: src/agent.py#L268]
            candidates.append((length, theta, unexplored))
    # Pass 1: unexplored, longest first, gap at least 0.9 units  [code: src/agent.py#L273-L287]
    chosen = spaced_pick(candidates, unexplored_only=True, gap=0.9 * unit)
    # Pass 2: any ray not yet picked, gap at least 4 units  [code: src/agent.py#L296-L298]
    chosen = add_spaced(chosen, candidates, gap=4 * unit)
    # Fallback when nothing was chosen: longest ray, gap of 1 unit  [code: src/agent.py#L301-L317]
    if not chosen:
        chosen = spaced_pick(candidates, unexplored_only=False, gap=unit)
    # Cap each length at 1.7 m, sort by angle
    # [code: src/agent.py#L282] [code: src/agent.py#L322-L323]
    return sorted((min(r, 1.7), t) for r, t in chosen)

The code also returns a fixed set of four default arrows, at 1.5 m, when there is no usable action and the turn-around cooldown has not expired agent.py agent.py.

Projection §3.3

Each action is drawn as a red arrow from the camera's image centre to its endpoint, with a numbered white circle at the tip. Numbers start at 1 and are given in the order of the drawn arrows. An endpoint is skipped if it falls within 4% of the image edge agent.py agent.py. The turn-around option, labelled 0 with the words TURN AROUND, is drawn on the left edge of the image, not as an arrow agent.py. Turn-around is offered when the turn-around cooldown (3 steps) has passed ObjectNav.yaml.

Prompting §3.4

The image and a text prompt go to the VLM in one call. The system instruction says the model is an embodied assistant with an RGB sensor, whose text output is turned into movement, and that it cannot move through closed doors agent.py. The action prompt gives the goal, asks the model to describe the view and any leads, asks for a general direction, and asks for a final choice in a dictionary with the key action. It also tells the model to choose 0 to turn around or when no arrow looks good agent.py. The same structure is shown in fig. 1. The model is not given the navigability map or any history by default; context_history is 0 ObjectNav.yaml.

The response is parsed by taking the last {...} block as a Python literal. If parsing fails, the step uses the default forward move of 0.2 m agent.py agent.py ObjectNav.yaml.

# Choose one action with the VLM  [§3.4] [code: src/agent.py#L345-L371]
def choose_action(image, actions, goal):
    # Draw numbered arrows; keep the projected list in number order  [§3.3]
    marked, drawn = draw_arrows(image, actions)
    # One VLM call with the marked image and the action prompt  [§3.4]
    reply = vlm.call(marked, action_prompt(goal, len(drawn)))
    # Parse the last {...} block; fall back to a short forward move  [code: src/agent.py#L647-L657]
    number = parse_action_number(reply)
    if number == 0:
        # Turn around  [code: src/agent.py#L639-L640]
        return (0.0, pi)
    if 1 <= number <= len(drawn):
        radius, theta = drawn[number - 1]
        # The executed yaw is the negated arrow angle  [code: src/agent.py#L637-L638]
        return (radius, -theta)
    return (0.2, 0.0)  # default action  [code: config/ObjectNav.yaml#L15]

Termination §3.5

Each step, a separate VLM call sees the raw un-annotated image and answers whether the agent is close to the goal. The reply is parsed for a done flag agent.py. When the flag is set once, the agent turns off navigability and explore bias, and draws the four default arrows at 1.5 m agent.py agent.py. When it is set on two consecutive steps, the episode ends with a stop action agent.py agent.py. The authors give this as the reason for using two calls: annotations add clutter, and separating the two questions avoids interference §3.5.

Hyperparameters

Values in the code are given in the config files. Where the paper states a value, it is cited to the paper. Otherwise the table says "not stated".

Results

Author-reported. Setup: Habitat ObjectNav and GOAT, scenes from HM3D §4. The VLM is Gemini Flash. Baselines are PIVOT, "Ours w/o nav" (static evenly spaced arrows) and "Prompt Only" (no drawn arrows) §4. The paper does not give the seed count or episode number.

ObjectNav, val split tab. 1:

GOAT, val unseen tab. 2:

GOAT against specialised systems, with sliding off for the authors' method tab. 3:

Design study, ObjectNav §4.3:

The field-of-view study (82°, 100°, 115°, 131°) is only given as a plot. The text says wider FOV gives higher SR and SPL §4.3.1 fig. 6. The numbers are not in the text.

Paper vs code

Searched: the arXiv LaTeX source, the abstract and the project page (https://jirl-upenn.github.io/VLMnav/). The project page links github.com/Jirl-upenn/VLMnav as its code. Its owner is the group that hosts the project and is linked from the paper's page, so it is used here. Commit 82d59684c15a6694ea73b969e76a57157c8740b8 was read. The repository has no LICENSE file, so no code is quoted. Citations are by path and line only.

Differences:

• Explored area. Paper: areas within 2 m of the agent are explored §3.1. Code: the explored line runs to 0.8 times the radius, capped at 1.7 m agent.py agent.py. The unexplored line runs to the radius minus 0.5 m agent.py.
• Spacing thresholds. Paper: θ_δ and θ_Δ are symbols with no values §3.2. Code: unit = fov / 360 radians, about 0.364 rad (about 21°) for 131° agent.py. The first pass uses 0.9 units agent.py agent.py. The second pass uses explore bias (4) units, about 83° agent.py. The fallback uses 1 unit agent.py agent.py.
• Second pass. Paper: explored actions are added in the second pass §3.2. Code: the second pass loops over every candidate, including unexplored ones that were skipped in the first pass agent.py.
• Bookkeeping bug. In the second pass the code adds the loop variable theta (the last angle from an earlier loop) to the set of chosen angles, not theta_i agent.py. This changes the spacing test for later candidates. Paper: no equivalent detail.
• Exploration flag. Paper: e_i is 1 for unexplored and 0 otherwise §3.2. Code: e_i is score < 3, where the score counts explored pixels in a short window around the endpoint agent.py. The constant explore_threshold = 3 agent.py is not referenced in the code read.
• Navigable directions. Paper: all directions θ in the FOV §3.1. Code: 1.5 times half the FOV, about ±98° agent.py.
• Turn around. Paper: a special action (π, 0) when stuck §3.2. Code: option 0 is offered after the cooldown and executes a 180° turn agent.py agent.py. The four default arrows of agent.py are a separate fallback. Their logic at agent.py is hard to follow: they are returned when no action passes the 0.5 m minimum and the cooldown has not expired.
• Prompt count. The code passes the number of drawn arrows, which excludes option 0 agent.py agent.py. The prompt then says that many minus one arrows are present agent.py. As read, the count is one less than the drawn arrows. This was not run.
• System instruction. Code: includes "cannot move through closed doors" agent.py. The system box in fig. 1 does not show it.
• Navigability threshold. Paper: no height threshold §3.1. Code: pixels within 0.2 m of the floor height (agent height minus 0.04 m) are navigable agent.py ObjectNav.yaml.
• Segmentation model. Paper: Segformer for the floor mask §4.3.3. Code: Mask2Former with a Swin-small ADE20K checkpoint, using floor and rug classes vlm.py vlm.py.
• Pixel-to-distance. Paper: pixel count times a constant factor §4.3.3. Code: an exponential in the pixel index plus an offset agent.py. The constants are code-only.
• Depth estimator. Paper: ZoeDepth §4.3.3. Code: the KITTI-trained checkpoint Intel/zoedepth-kitti vlm.py. The checkpoint is not named in the paper.
• Stop threshold. Paper: ObjectNav success within 1.2 m §4.1; GOAT within 1 m §4.2. Config: 0.3 for ObjectNav ObjectNav.yaml and 0.25 for GOAT GOAT.yaml. The config is what the environment reads env.py. Results at different thresholds are not comparable.
• Episode length. The paper does not state a step cap. The ObjectNav config caps episodes at 10 steps ObjectNav.yaml. The GOAT config caps at 10000 steps with 30 per subgoal GOAT.yaml.
• Camera pitch. Paper: 25° §4. Config: -0.45 rad, about -25.8° ObjectNav.yaml.
• No-navigability baseline. Paper: a static set of evenly spaced arrows §4. Code: five fixed angles at -0.36π, -0.28π, 0, 0.28π and 0.36π, which are not evenly spaced agent.py.
• Depth ablation text. Paper: ZoeDepth gives a 10% drop in SR §4.3.3. The table gives 50.4% - 39.1% = 11.3 points tab. 5.
• Stopping rule. Consistent: after the first stop, navigability and explore bias are turned off §3.5 agent.py.
• Sampling and cost. Not in the paper: the Gemini sampling settings vlm.py. Cost is tracked per call vlm.py, but the paper gives no cost figure.

Pitfalls

• Numbering. Only arrows that pass the edge and visibility tests are numbered, in order. Turn-around is always 0 and is not an arrow agent.py agent.py. Reproducing the number-to-action map requires the same filtering.
• Sliding. Results depend on allow_slide. Without it, ObjectNav SR falls from 50.4% to 12.9% tab. 1. The paper says that the navigability module ignores the robot's size and shape, so the robot can get stuck with no way to back up §4.1. Do not assume the method transfers to a real robot without a collision-aware controller; the authors say so §5.
• Explored-area radius. Using the paper's 2 m figure instead of the code's 0.8 r (capped at 1.7 m) changes the exploration signal.
• Spacing units. The code uses degrees divided by 360 as radians. A reimplementation that converts the FOV to radians first gets a different spacing.
• Stop at arrival. The stop rule needs two consecutive positive answers §3.5. The authors report that in 13.9% of GOAT runs the VLM stopped 1 to 1.5 m from the object, which counts as failure under the 1 m threshold §4.2.
• Depth noise. Depth from ZoeDepth lowers SR, and an RGB-only mask does nearly as well as the depth sensor §4.3.3 tab. 5. The mask method has no documented threshold for its floor class beyond the names in the code.
• History. Adding history hurt performance §4.3.2. The "no history" row in tab. 4 has exploration bias off, so it is not comparable with tab. 1.
• Model dependence. Results use Gemini Flash. The code also has an OpenAI path vlm.py. Other models are untested by the authors.
• Per-step cost. Two VLM calls run every step. The stopping call always sees the raw image agent.py.
• Memory. The two 5000 × 5000 uint8 maps take about 150 MB per agent agent.py.
• Figure and table numbers. Numbers here follow the order the figures and tables appear in the LaTeX source. The compiled PDF was not checked.

Lineage

• Builds on: visual prompting that marks actions on images, such as set-of-mark §2 §3.
• Compared against: PIVOT (iterative Gaussian action sampling) §2 §4; Modular GOAT and SenseAct-NN on GOAT §4.2; Prompt Only and Ours w/o nav as internal ablations §4.
• Superseded by: not stated.