Paper of the day, 2026-10-31

End-to-End Navigation with Vision Language Models: Transforming Spatial Reasoning into Question-Answering

Listen to the episode, or read on. The paper: arXiv:2411.05755.

Vision-language models describe a kitchen happily, but ask one how many centimetres it is to the fridge and it falls apart. VLMnav works around that weakness instead of fixing it. Plain geometry finds the safe directions, they are drawn as numbered arrows on the camera image, and the model only has to pick a number. If you are connecting a VLM to anything that moves, this trick is worth a look this week.

The problem in one picture

Drawing made for wikillm on 2026-10-31 by the AI editor. It is not taken from the paper.

How it works

It is Halloween, so picture a robot creeping through a dark house, looking for one particular object and trying hard not to walk into the furniture. VLMnav does not train anything for this. It takes an off-the-shelf vision-language model, Gemini Flash in the paper, and asks it the right question at each step.

Every step goes through four stages.

Navigability. The depth image becomes a height map, and pixels that sit near floor level count as drivable. In the released code, the robot sweeps 60 directions across one and a half times half its field of view. Along each direction it traces a ray of up to 2 m until the ray leaves the floor. Each ray gives a candidate move as a distance and an angle.

Action proposer. A top-down grid remembers where the robot has already looked. Each candidate is labelled explored or unexplored and shortened to two thirds of its length, and the very short ones are thrown away. Selection then runs twice. The first pass picks unexplored directions, longest first, and keeps them a small angle apart. The second pass may add other directions, but only if they sit much further from everything already chosen. The result leans toward new territory. The code caps each move at 1.7 m.

Projection. Each surviving move becomes a red arrow on the camera image with a white numbered circle at its tip. A separate option 0, a turn-around, sits at the image's left edge.

Prompting. The marked image goes to the VLM with the goal. The model is asked to describe what it sees, reason about a general direction, and then name one arrow number. By default it gets no history and never sees the map. If the reply cannot be parsed, the robot just moves forward 0.2 m.

A second VLM call runs on every step to decide when to stop. It looks at the clean, unmarked image and answers one question: am I close? The episode ends after two yes answers in a row. The authors keep the two questions apart because the arrows clutter the image and the two tasks could interfere.

So the geometry stays with the code, which is good at geometry, and the meaning stays with the model, which is good at meaning. Neither has to fake the other's skill.

Try it yourself

In this exercise you rebuild the action proposer on a toy top-down floor plan. Three round obstacles stand in for furniture, and a blue disc marks the area the robot has already explored. The code sweeps rays, filters and shortens them, runs both spacing passes, and draws the numbered arrows a VLM would be shown. The real system works on image pixels. This version works in a flat top-down room, so it is a sketch of the procedure, not a copy of it.

Python

import numpy as np, matplotlib.pyplot as plt
walls = [(1.2, 0.3, 0.4), (0.5, 1.4, 0.35), (-0.4, 1.0, 0.3)]  # obstacles: x, y, radius (m)
fov = 131; unit = fov / 360          # spacing unit, the code's degrees-over-360 quirk
sweep = np.radians(fov / 2) * 1.5    # sweep 1.5x half the field of view
pt = lambda d, t: (d * np.sin(t), d * np.cos(t))   # robot at origin, facing +y
hit = lambda x, y: any(np.hypot(x - a, y - b) < r for a, b, r in walls)
seen = lambda x, y: np.hypot(x + 1.0, y - 0.3) < 0.9   # area explored earlier
def ray(t, L=2.0):
    return next((d for d in np.linspace(0.05, L, 80) if hit(*pt(d, t))), L)
cands = []
for t in np.linspace(-sweep, sweep, 60):
    r = ray(t)
    if 0.66 * r > 0.75: cands.append((0.66 * r, t, not seen(*pt(0.8 * r, t))))
cands.sort(reverse=True)                                   # longest first
chosen = []
far = lambda t, gap: all(abs(t - c) >= gap for _, c in chosen)
for L, t, new in cands:                                    # pass 1: unexplored only
    if new and far(t, 0.9 * unit): chosen.append((min(L, 1.7), t))
for L, t, new in cands:                                    # pass 2: anything, wide gap
    if far(t, 4 * unit): chosen.append((min(L, 1.7), t))
chosen.sort(key=lambda a: a[1])                            # number left to right
fig, ax = plt.subplots(figsize=(6, 5))
for a, b, r in walls: ax.add_patch(plt.Circle((a, b), r, color="grey"))
ax.add_patch(plt.Circle((-1.0, 0.3), 0.9, color="skyblue", alpha=0.3))
for L, t, _ in cands: ax.plot(*zip((0, 0), pt(L, t)), color="0.85", lw=0.8)
for i, (L, t) in enumerate(chosen, 1):
    ax.annotate("", pt(L, t), (0, 0), arrowprops=dict(color="red", width=2))
    ax.text(*pt(L, t), str(i), ha="center", va="center", bbox=dict(boxstyle="circle", fc="white"))
ax.set_aspect("equal"); ax.set_title("Numbered arrows a VLM would choose from"); plt.show()
print(f"rays swept: 60, kept after filtering: {len(cands)}, arrows offered: {len(chosen)}")
print(f"pass-1 gap: {np.degrees(0.9*unit):.1f} deg, pass-2 gap: {np.degrees(4*unit):.1f} deg")

What to expect. You get a fan of faint grey rays, with gaps where obstacles cut rays short enough to be dropped. On top are a handful of red numbered arrows, mostly pointing away from the blue explored disc. The printout shows how many of the 60 rays survived and the two spacing gaps, roughly 19° and 83°. That second gap is why the explored side gets at most one arrow.

Explore further. Change 4 * unit in the second pass to 1 * unit. The arrows that point back into explored ground multiply, so the menu gets longer and the push toward new territory gets weaker. For a second experiment, convert fov to radians before dividing by 360 and see how the spacing collapses. The card warns about exactly this trap.

What the authors report, and what they leave open

On Habitat ObjectNav with HM3D scenes, the authors report a 50.4% success rate and an SPL of 0.210. PIVOT, an earlier visual-prompting method, reaches 24.6% and 0.106. The ablations show where the gain comes from. Fixed arrows without the navigability step reach 33.2%, and a bare prompt with no arrows reaches 29.8%. Turning off sliding along walls drops the method to 12.9%, which is the most sobering number in the paper.

On the harder GOAT benchmark, the method gets 16.3% against PIVOT's 8.3%. Two specialised systems do better: an RL skill chain at 29.5% and a modular GOAT system at 24.9%. With sliding off, the method falls to 6.9%. Adding history made results worse rather than better (46.8% with none, 40.4% with 15 steps, both without exploration bias). Swapping the depth sensor for an RGB-only Segformer floor mask costs little (47.2%), while estimated ZoeDepth depth costs more (39.1%). In 13.9% of GOAT runs the model stopped 1 to 1.5 m from the target, which counts as a failure.

The paper does not state the seed count, the number of episodes, the exact Gemini version or the cost. The card also lists gaps between the paper and the code. The success distances in the configs (0.3 and 0.25) differ from the paper's 1.2 m and 1 m, the code has a step cap of 10 for ObjectNav, and the spacing thresholds appear only as symbols in the paper. Real robots are untested, other VLMs are untested, and the code repository has no license.

Ask yourself

Sliding along walls more than triples the success rate. Is the result about VLM reasoning, or about a forgiving simulator? How would you separate the two?
Why might more history hurt a VLM navigator? What would you give it instead of raw past frames?
The stop question needs two yes answers in a row. Would you trade that for a distance check from depth, and what would you lose?