Embodied Agents Take Control: Minimal-Interface Zero-Shot Agents Rival Industrial-Scale Policies in Vision-and-Language Navigation

In plain words

Picture giving a robot a sentence such as "walk down the hall, turn left at the kitchen and stop by the sofa", and then asking it to find the route using only what its camera shows. The usual ways to build this either train a dedicated navigation model, or put a language model inside a fixed set of helper steps written by people.

This paper asks a simpler question: can a general-purpose AI coding agent run the whole loop by itself? The agent gets a camera, four moves (forward, turn left, turn right, stop) and the instruction. It decides when to look, when to move and when to stop. It has no map, no navigation memory and no planner of its own, and the program around it does not decide the order of actions.

On 100 test routes in photorealistic 3D building scans, the authors report success rates from about 68 to 78 percent. An optional tool that proposes nearby waypoints raised the success rate further and used fewer simulator steps per episode. The paper is clear about the weak spots: the agent often stops confidently in the wrong place, and its context grows so much on long routes that episodes become slow and costly.

TL;DR

• Zero-shot vision-and-language navigation on R2R-CE with three coding-agent harnesses (Claude Agent SDK, mini-swe-agent, Codex CLI), a monocular 512x512 RGB view and four discrete actions, with no navigation-specific training, map, search or memory §1 §3.1 app. A.
• Replicated default-effort opus-5 runs (Claude Agent SDK): 70.7 ± 3.5 SR; fable-5 at max effort: 78 SR §4.2 tab. 2.
• The model choice produces the widest spread (5 to 72 SR across models); harness gaps are 1.7 to 7.3 SR and treated as descriptive §4.3.
• An optional waypoint tool alongside primitives reaches 76.7 ± 0.6 SR at default effort, with about half the environment steps of max-effort primitives §4.4 tab. 5.
• All numbers are author-reported on n = 100 with mostly single runs; the code is MIT-licensed on GitHub.

Problem

Trained navigation policies need large in-domain data, and zero-shot systems usually fix the control flow in code: a planner calls a controller, or a workflow decides when to observe §1 §2.1. The authors ask whether the model can take over the interaction loop itself (when to observe, act, recover and stop), so that reasoning and action stay aligned without a handoff §1. The setting is zero-shot R2R-CE, where a text instruction must be followed to an endpoint and a STOP must be issued within 3 m of the goal app. A.

Method

Agentic embodied control. The reasoning model directly chooses every tool call. The harness only delivers the prompt, runs the tool calls, returns the results and keeps the session. Nothing navigation-specific sits between the model and the environment §3.1. Each episode starts a fresh session, and no state is carried across episodes §3.1 app. A.

Minimal interface. Two tools only §3.2 app. A:

1. observe() returns one 512 x 512 egocentric RGB frame with a 90° horizontal field of view. It advances nothing.
2. step(actions) runs up to 50 primitives in order: 0 = STOP (ends the episode), 1 = forward 0.25 m, 2 = turn left 15°, 3 = turn right 15°. The result reports how many primitives ran and the remaining budget, but no image and no collision signal §3.2 app. A.

The agent gets no pose, odometry, depth, panorama, map, waypoint or privileged simulator state. The model has to infer landmarks, progress and the stopping point from its own observation history §3.2.

Briefing. One fixed system prompt, identical across harnesses, describes the robot, the tools, the task instruction, the 500-step budget and the success rule. The first user message is fixed at "Begin navigating. Call observe() first to see where you are." app. A

Harnesses §4.3 app. A:

• Claude Agent SDK (closed, v0.2.110): subscription authentication, the briefing replaces the system prompt, built-in tools disabled, only the bridge exposed through MCP, hard cap of 200 turns.
• mini-swe-agent (open, v2.4.5): a plain ReAct loop with no context management. The full history, including every image, is re-sent on every call. Model calls go through LiteLLM.
• Codex CLI (closed, v0.142.0 to 0.145.0): ChatGPT-subscription authentication and a read-only sandbox. It exposes no LLM-call cap, so it is bounded only by the action budget and the timeout.

Success and measurement. Success requires the agent to issue STOP on its own while within 3 m of the goal, measured as geodesic distance. An episode that runs out of budget without stopping scores zero. SR, SPL, NE and OSR are computed by the driver after the session ends, and the agent cannot see its own score app. A.

Episode loop (as implemented in the paper's setup).

# Minimal-interface episode  [§3.1] [§3.2] [app. A]
# Input:  instruction (str)
# Output: success (bool), scored after the session ends

def run_episode(instruction, agent, env):
    # Fresh session, nothing carried over from other episodes  [§3.1]
    session = agent.new_session(briefing(instruction, budget=500))
    # Hard cap of 200 model turns per episode  [app. A]
    for turn in range(200):
        # The model chooses every call; the harness only runs it  [§3.1]
        call = session.next_call()
        if call.tool == "observe":
            # Pure read: one 512x512 RGB frame, 90 deg hfov  [§3.2] [app. A]
            session.reply(env.render_rgb(width=512, hfov_deg=90))
        else:
            # Up to 50 primitives, executed in order  [§3.2] [app. A]
            # 0 STOP, 1 forward 0.25 m, 2 left 15 deg, 3 right 15 deg
            result = env.run_primitives(call.actions)
            session.reply(result.counts_and_budget)
        # Stop at the 500-step budget or when the agent issues STOP  [app. A]
        if env.steps_taken >= 500 or session.stopped:
            break
    # Success needs the agent's own STOP within 3 m of the goal  [app. A]
    return session.stopped and env.distance_to_goal() <= 3.0

Optional waypoint interface (hybrid). The primitives stay available, and a trained waypoint module is added as three tools §4.4 app. A:

• observe_waypoints() renders a 12-view RGB-D panorama, passes it to a trained candidate-waypoint predictor, and returns a four-view strip (Left, Front, Right, Back) with up to five numbered candidates, plus text listing each candidate's direction and distance.
• goto(k) turns toward candidate k and walks to it with forward primitives. Each goto consumes simulator steps from the shared 500-step budget, and at most 30 goto calls are allowed per episode. The 30th ends the episode.
• stop() ends the episode.

Moving makes the candidate numbering stale, so the agent must look again before its next goto app. A. The waypoint predictor is a trained navigation module, so these cells are not zero-shot in the strict sense app. A.

Waypoint-only arm (for the interface ablation). The waypoint tools replace the primitives. The observation becomes the four-view strip rather than a single front view, so the deltas measure the combined change app. A §4.3.

Reasoning effort. Each model runs at its vendor default effort. The effort ablation changes only the vendor-defined effort label for a fixed model and harness §4.3 app. A.

Repetitions. Each default-effort Claude SDK cell was run three times and reported as mean and sample standard deviation. Most other cells are single runs, and the paper treats differences of only a few points as descriptive §4.1.

Hyperparameters

Results

All results are author-reported. Setup: R2R-CE rand100 (100 episodes), model and harness as named, baseline as named. Rows marked with an asterisk are the mean of three runs tab. 1 tab. 2.

Headline comparison tab. 1. External rows come from other papers and are not directly comparable: they use different models, serving paths and sometimes the full val-unseen split.

Visual input: M = monocular, P = panorama, D = depth. tab. 1

Full main board tab. 2. Harness and model, then SR, SPL, NE (navigation error, m) and OSR (oracle success rate). Lower NE is better.

Source: tab. 2. The paper's Codex row for gpt-5.6 is served as gpt-5.6-sol on ChatGPT accounts app. A.

Reasoning effort tab. 3. Each variant is paired with its default run. Time is the per-episode mean and median in seconds.

The paper concludes that effort effects are model-specific rather than uniform §4.3. Full metrics appear in app. A.

Interface ablation tab. 4. Primitive versus waypoint arm, same model, harness and episodes.

On VLNVerse the collision rate rose from 6 to 35 for sonnet-5 and from 7 to 27 for fable-5 §4.3. Only the R2R-CE rows with an asterisk are replicated means; the rest are single runs.

Hybrid interface tab. 5. fable-5, Claude SDK, R2R-CE. Steps, time and calls are per-episode medians.

Asterisked SR values are means over three runs, and the ± values are sample standard deviations where the paper gives them tab. 5. The agent in the hybrid arm used waypoints early to cover distance and primitives near the target, with about eight waypoint moves and five to six primitive bursts per episode §4.4.

Long horizon tab. 6. Matched fable-5 Claude SDK runs at default effort, n = 100 English episodes each. Time and context are per-episode medians (seconds, and final-call input tokens).

Across the R2R-CE bare cells, the final-turn context has a median of 33k tokens and a maximum of 169k. Logged episodes (4,700 in total) take a median of 206 s of wall time, and 18.6% of them exceed ten minutes §4.5.

Failure taxonomy tab. 7. The 30 failures of the strongest of three fable-5 default runs (SR 70). d_goal is the median final goal distance in metres.

Of the 30 failures, 26 end in a voluntary STOP, and 23 claim success while the final distance is 3.1 to 34.1 m. Only 5 of 30 trajectories enter the 3 m success region at any point. In at least 15 episodes the reasoning states the correct doubt and still commits to the wrong endpoint §4.6.

Other benchmarks app. B. The same frozen loop, unchanged, reaches 84 SR on VLNVerse fine-grained instructions (n = 100 subset) and 76.2% answer accuracy on the full 500-question HM-EQA set (median 185 s per question). The paper's comparison values are 63.75 for Qwen-RobotNav on VLNVerse and 76.7 for its deployed agentic system on HM-EQA. The paper calls both comparisons contextual.

Physical robot §4.7. A Unitree Go2 quadruped ran the same minimal agent in an office building across 31 exploratory episodes. The paper does not report a success rate because the robot gives no ground-truth pose or success label. Its finding is that reasoning transferred while body awareness and persistent spatial memory did not.

Paper vs code

Checked against the official repository jianzhou0420/MIP at code_commit 2ebeea8f5478b03197de35418b3aa6cf02e8acbc (MIT, copyright Jian Zhou, 2026). The paper links this repository in its abstract block. Discrepancies and unverified points:

• Default effort. The paper says the Claude runs send no effort parameter, "which resolves to high" app. A. The code does send nothing when the seat word is default; the effort then falls to the CLI's own per-model default cc.yaml claude_sdk.py. The code's comment says the CLI's per-model table is sent as output_config.effort. The "high" value is therefore set by the CLI, not by this repository, and could not be confirmed here.
• Thinking. The code uses adaptive thinking with summarized output for Claude 4.6+ and 5 models, and a fixed budget only when one is set explicitly claude_sdk.py. The paper says only that extended reasoning is on. Adaptive thinking means the model chooses how much it spends, so budgets cannot be compared across runs without the vendor defaults.
• Turn-budget warnings. The code can tell the agent how many tool calls remain and warns when fewer than 20 remain, saying that ending without a STOP scores zero view.py. Under bare: true, which all reported minimal and hybrid configs set, the budget broadcast is off episode.py std_mip_r2r_minimal.yaml. The paper does not mention these warnings, and the agent in reported runs did not get them. The move warning in the hybrid is separate and is on panorama.py.
• Simulator version. The paper names Habitat-Sim 0.1.7 app. A. The install notes for the env server specify habitat-sim 0.3.3 INSTALL.md. Results may depend on which version ran.
• Waypoint RGB branch. The paper says the predictor's RGB branch is unwired and zeroed, so prediction is depth-driven app. A. The repository sends the panorama's RGB bytes to the predictor along with depth panorama.py. The zeroing would happen in the predictor service, which is not in this repository, so it is unverified.
• Success check and scene split. The 3 m success test and the rand100 split files are in the EmbodiedScore-envs submodule. The submodule directory is empty in the cloned tree, so the success radius was checked only against the paper and the configs.
• Waypoint predictor. The predictor (smartway_waypoint__predict) is called over HTTP at http://127.0.0.1:9210 std_mip_r2r_hybrid.yaml std_mip_r2r_hybrid.yaml, and its implementation is not in this repository.
• Consistent items. The 200-turn cap, 500-step budget, 2400 s timeout, 512 px render, the 30-goto cap and the 384 px strip tiles match the paper std_mip_r2r_minimal.yaml std_mip_r2r_hybrid.yaml. The briefing rule about alternating observation and steps matches the shared fragment text standard.yaml.

Pitfalls

• Few replicated cells. Only the default-effort Claude SDK cells have three runs. The sample standard deviations there run from 1.2 to 3.5 SR points, so the paper treats differences of a few points as noise §4.1. Most conclusions in the ablations rest on single runs of 100 episodes.
• Harness comparisons are not controlled. Serving paths, defaults and versions differ, and Codex CLI changed version during the campaign §4.3 app. A. The paper calls the harness comparison descriptive and says none of its conclusions rest on the GPT runs app. A.
• Effort is not comparable across vendors. Labels such as max and xhigh are vendor-specific app. A. Fix an explicit effort in your own runs if you need repeatability, since the default depends on the CLI.
• Context grows without bound in two harnesses. mini-swe-agent and the Claude Agent SDK re-send the full history with every image on every call. The SDK's backstop is automatic compaction at about 167k tokens, which replaces images with a placeholder claude_sdk.py §4.5. Long routes therefore cost more per call as the episode goes on.
• Silent failure. The tool reports how many primitives ran, not what the robot did. A blocked move looks like a completed move, and the agent must infer the block from the next image §3.2 §4.6. The physical robot showed the same problem, where an under-rotation silently corrupts the heading §4.7.
• Waypoint cap. The 30th goto ends the episode, and then stop() can no longer be called. An agent that spends its moves scores zero app. A.
• The waypoint arm is not zero-shot. The predictor is trained, so the waypoint cells fall outside the zero-shot claim app. A. The hybrid result depends on that trained module.
• Forced versus optional waypoints. On R2R-CE, forced waypoints gained +17 to +38 SR for the Qwen and GPT rows. Sonnet-5 and opus-4.8 gained about 9 points, while fable-5 and opus-5 gained 0.7 to 1.3 tab. 4. On VLNVerse forcing cost 4 to 6 SR §4.3. The abstract's statement that forced waypoints "can hinder stronger ones" refers to those VLNVerse rows, which are single runs.
• Harness gap number. The text gives 1.7 to 7.3 SR for harness gaps §4.3. The figure caption gives 2 to 7 SR fig. 3. Use the text figure, which matches the table data.
• Naming. The paper writes gpt-5.6 in the board and gpt-5.6-sol in the interface table tab. 2 tab. 4. The Codex runs use the -sol variant app. A.
• Data and licence. The Matterport3D scenes and R2R-CE data are not in the repository. The README says the scene datasets are licensed separately by their owners README.md. Their licences apply to any reuse of the environment.
• Keep the minimal condition intact. The minimal condition gives the model no pose, odometry, depth or map §3.2. Adding any of them changes what the benchmark measures, so label such runs as different from the paper's.

Lineage

• Builds on: NavGPT, an early agentic zero-shot navigation system with a ReAct loop §2.1 §2.2; the ReAct organization of an agent as an observe-reason-act loop §2.2; coding-agent harnesses such as mini-swe-agent §2.2.
• Contrasts with: Voyager, which combines an LLM with a curriculum, skill library and verification loop §2.2; zero-shot systems that fix control flow in code (MapGPT, Open-Nav, SmartWay) §2.1.
• Compared against: AgenticNav (zero-shot agentic control with map, depth and memory tools) §2.2 tab. 1; Qwen-RobotNav (trained policy under an upper-level planner) §2.2 tab. 1; ABot-N1 and InternVLA-N1 / DualVLN (trained dual-system workflows) §2.1 tab. 1. The paper notes that trained systems in its comparison were trained on millions of samples and were evaluated on a different split §1.
• Superseded by: not stated.