TANGO: Training-free Embodied AI Agents for Open-world Tasks

In plain words

A robot that has to find things in a house usually needs a lot of training for each job: one for finding a chair, another for finding a cup, another for answering questions about a room. This paper asks whether a language model can plan the job instead. The robot already knows one reliable skill, walking to a given point on the floor. The authors add a few more skills: spot an object in the camera image, keep a memory of where things were seen, and answer a question about what the camera sees.

The language model reads the user's request and writes a short list of these skills to run in order, much like a recipe. A few example recipes in the prompt show it how to do this. Nothing is trained for the task itself, so the same setup can be pointed at different jobs. Each step of the recipe is also readable, which makes it easier to see why the robot went the wrong way.

On three benchmarks the approach lands close to the best zero-shot systems. The paper's own text describes it as on par on one benchmark and second on another, and it leads on success rate on the third.

TL;DR

• Zero-shot embodied agent: an LLM writes a short program of navigation and perception primitives from 15 in-context examples §3.
• The navigation core is a PointGoal policy pretrained on HM3D, plus a memory-augmented exploration map that extends VLFM to several sequential targets §3.
• Evaluated on open-vocabulary ObjectNav (OVON), multi-modal lifelong navigation (GOAT-Bench) and active embodied QA (OpenEQA), with no fine-tuning §4.1 tab. 1 tab. 2 tab. 3.
• Author-reported: on par with VLFM on OVON, second on OpenEQA (0.9 points below the leader), and +2.6 SR on GOAT-Bench over the best prior row §4.2.
• The released evaluation code does not call the LLM; it uses fixed per-task program templates (see Paper vs code).

Problem

Modular navigation systems build maps and planners, and each task needs its own tuning §1. End-to-end reinforcement learning needs large training budgets per task §1. Program-composition systems such as VisProg and ViperGPT compose vision models with in-context examples, but were evaluated on static images, not on agents that act §1 §2.

The paper assumes that a pretrained PointGoal policy is good enough as a primitive, that an LLM can sequence primitives from a few examples, and that the detectors and value map are accurate enough for the target objects §3 §4.3.

Method

Overview. TANGO is a set of modules that the LLM composes into programs. An LLM receives a natural-language instruction and a fixed prompt of 15 in-context examples, and outputs a step-by-step program with comments §3 fig. 2. Each program line calls one module. Each module parses its arguments, runs in the environment, and writes its output to a named variable §3. The paper uses GPT-4o as the LLM §3.

Modules. The paper lists primitives such as detect, answer, match, count, is_found, navigate_to, explore_scene and turn §3 fig. 3. Those written as Python subroutines are counting and logic. Others use pretrained vision models §3 fig. 3.

Navigation. The foundational primitive is a PointGoal navigation policy. It uses egocentric depth, plus the relative distance and heading to the goal point. The paper reports that the policy reaches about 99% success on standard PointGoal evaluation, with a forward pass taking a fraction of a second §3. The policy is pretrained on the HM3D training split and was not trained on the test scenes §4.1.

Exploration with memory. Exploration follows VLFM §3. It builds an occupancy map from depth, finds frontiers, and keeps a language-grounded value map from RGB frames and a vision-language model §3. TANGO extends this in two ways. It stores a feature vector per map pixel, and it updates that feature map every step. When a new target arrives, the agent scores every stored pixel against the target embedding by cosine similarity §3. If the best score passes a threshold, the agent treats the target as remembered and goes there. Otherwise it keeps exploring §3. The paper states that the similarity is recomputed only when the target changes §3.

# Memory-augmented target search  [§3]
# Input:  feature_map (H, W, d) per map pixel; target_embedding (d,)
# Output: a map pixel to navigate to, or None if not remembered

def remembered_target(feature_map, target_embedding, threshold):
    # Cosine similarity of each stored pixel feature with the target  [§3]
    # Recomputed only when the target changes, not at every step  [§3]
    similarities = cosine(feature_map, target_embedding)  # (H, W)
    best_pixel = argmax(similarities)
    best_value = similarities[best_pixel]
    # Navigate when the best value passes the memory threshold  [§3]
    if best_value < threshold:
        return None
    return best_pixel

The block above uses the code's comparison. The paper says the best value must "exceed" the threshold, but the code rejects only values strictly below it, so a tie navigates interpreter.py.

Object detection and classification. For general classes the detector is OWLv2. For classes inside the COCO label set, DETR is used §3 §4.1. Detections go to a CLIP-based classifier that separates the target from similar classes such as "armchair" or "couch" §4.1. The paper says this filter was needed because the authors saw many false positives in synthetic 3D scenes §4.1.

Initial scan. At the start of navigation the agent makes a full 360° turn, following VLFM, to initialise frontiers §3.

Tasks. For OVON, the program searches with explore_scene, runs detect on the target, and navigates to it when found §3.1 §4.1. For GOAT-Bench, a target can be named by category, description or image, and the LLM turns a multi-target instruction into an ordered list of subgoals §3.1 §4.2 fig. 5. For OpenEQA, the program searches for the object, navigates there, and calls a BLIP-2 answer module §3.1 §4.1. The paper says the answer module is BLIP-2 §3.1.

Actions. The paper lists the agent's actions as forward, move left, move right, look up, look down and stop §4.1.

Hyperparameters

Paper values are listed first. Values found only in the released code are marked as code-only and are not claims of the paper.

Results

Author-reported. Setup: HM3D-OVON val unseen §4.1; GOAT-Bench val unseen §4.1; OpenEQA active-EQA episodes with a 500-step limit §4.1. OpenEQA score is the mean of (σ − 1)/4 × 100 on a 1–5 LLM judge scale eq. 1.

OVON, HM3D val unseen tab. 1

OpenEQA, score tab. 2

GOAT-Bench, val unseen tab. 3

The text reports +2.6 SR over the first row and −0.7 SPL against the best SPL §4.2. Figures are described, not reproduced here: a success example for GOAT-Bench memory fig. 5 and OpenEQA successes and failures fig. 4.

Failure analysis §4.3 fig. 6. A failure is an answer scored below 3. In a manually annotated subsample of EQA failures, the main cause is the detect module, such as stopping at a wrong object or ignoring the goal object §4.3. Exploration fails in about 10% of cases §4.3. The LLM writes incorrect pseudo-code in about 18% of cases: 6.9 points come from ambiguous or wrong targets and 11.2 points from wrong primitive order or usage §4.3. The sample size is not stated.

Paper vs code

Repository: filippoziliotto/TANGO, owned by the first author. It is a fork of the habitat-lab monorepo with the TANGO code added under habitat-baselines/habitat_baselines/rl/ppo/. Its LICENSE is MIT, with a Meta copyright line inherited from habitat-lab LICENSE. Commit e9cceb2af9a1f322c43e4fd9c64d298cd01e45ac. Earlier run names in the configs say "NavProg" zs_goat.yaml.

• The LLM is not in the evaluation path. The evaluator builds its generator with debug=True habitat_evaluator.py. In debug mode the generator returns a fixed program per task, not an LLM output code_generator.py. The paper says the LLM composes the program §3.
• The target name comes from the episode. The OVON template inserts goals[0].object_category from the episode utils.py. The GOAT template inserts the episode's category goat.py. The OpenEQA template reads the object name for each question from a dataset file open_eqa.py. The paper does not say that the target label is given to the agent.
• Matcher not in the active interpreter. The evaluator imports only interpreter.py habitat_evaluator.py. That file defines no match method. FeatureMatcher is called only in interpreter_goat.py and interpreter_orig.py, which nothing imports interpreter_goat.py. The paper describes SuperGlue matching for image goals §3.1.
• Value-map encoder differs. The paper says BLIP-2 §3. Code uses CLIP for OVON and EQA zs_ovon.yaml and BLIP for GOAT zs_goat.yaml.
• Detector version differs. The paper cites OWLv2 §4.1. The GOAT and EQA configs use OWL-ViT v1 zs_goat.yaml zs_eqa.yaml utils.py.
• OVON geometry and budget. The paper gives 0.88 m height, 640×480 images at 79°, and 500 steps §4.1. The OVON config has 1.2 m height, 256×256 at 90°, and 250 steps zs_ovon.yaml zs_ovon.yaml zs_ovon.yaml.
• GOAT geometry and turn. The paper gives 1.31 m camera height, 360×640 images, and 30° turns §4.1. The config uses 1.2 m, 256×256, and 20° zs_goat.yaml zs_goat.yaml.
• Actions. The paper lists move-left and move-right §4.1. The zero-shot configs register turn_left and turn_right and no strafing action zs_goat.yaml.
• OpenEQA metric. The score formula matches the paper's Eq. 1 llmscorer.py eq. 1. The in-environment measure open_eqa_answer_accuracy is an exact string match open_eqa.py, which is not the reported score. The judge model is not stated in the paper.
• PointNav checkpoint. Consistent: the paper says HM3D training split §4.1; the config loads an HM3D depth checkpoint zs_goat.yaml.
• Initial 360° scan. Consistent with §3: the code turns 360 // turn_angle times at the first subtask interpreter.py.

Pitfalls

• Test the LLM yourself. The released evaluator does not exercise the LLM, so its numbers do not test the paper's main claim. The LLM branch is broken as released. With use_LLM enabled, code_generator.py reads self.config, which the class never sets code_generator.py. The request also uses client.ChatCompletion.create, the pre-1.0 OpenAI SDK call, on a client built with OpenAI() code_generator.py code_generator.py. Write your own LLM call and keep the prompt format from the example file.
• Privileged target names. Templates insert the episode's target name and, for OpenEQA, the object name from the dataset utils.py. Any comparison with a system that only sees the instruction needs this removed.
• Errors are caught. Each episode runs inside try/except and failures go to handle_errors habitat_evaluator.py. The GOAT image-goal template calls match(target) goat.py, which the active interpreter does not define. The paper does not say how errors are scored. Check handle_errors before trusting the numbers.
• Memory threshold is unstated. The paper gives no number in the main text. The GOAT config uses 0.3 zs_goat.yaml and the class default is 0.4 models.py. The paper's supplementary ablation is not compiled into main.tex (X_suppl is commented out), so this card does not use it.
• Threshold comparison direction. The paper says the best value must "exceed" the threshold. The code treats only values strictly below it as misses interpreter.py. A tie navigates.
• Memory is off by default. The memory branch runs only when save_image_embed is true interpreter.py. OVON, ObjNav and EQA set it to false zs_ovon.yaml.
• Fusion rule. The paper does not describe how a new observation merges into the value map. The configs use a confidence-weighted average value_map.py.
• Fallback is unstated. The main text does not say what happens when the remembered point does not contain the target. After reaching the remembered point, the code turns in place 360 // turn_angle times and checks for the target interpreter.py.
• OpenEQA judge bug. In the local-model branch, diff is referenced even when the count already matches, so the code can fail with a NameError llmscorer.py. The GPT branch keeps every integer 1–5 in a response, so a response that mentions several numbers adds several scores.
• "State-of-the-art" overstates the tables. The abstract claims state-of-the-art. The text says "on par" on OVON and "second" on OpenEQA §4.2. On OVON, DAgRL+OD has higher SR and SPL. On OpenEQA, the frame-caption baseline has a higher score.
• Compute is not reported. Each step may run OWLv2, and DETR for COCO classes, plus a CLIP classifier per detection and a value-map encoder. The paper reports timing only for the PointNav policy §3. Model load time is not reported.
• LLM costs are not reported. The paper does not report LLM call counts, latency or prices. The code's generation path caps output at 200 tokens code_generator.py, which bounds program length.

Lineage

• Builds on: VisProg and ViperGPT for composing vision modules with in-context examples §1; VLFM for the frontier-based exploration and value map §3; a PointGoal navigation policy from MOPA and VLFM §3. No arXiv ids are given here because the source does not give them.
• Compared against: OVON baselines (RL, BCRL, DAgRL, DAgRL+OD, VLFM) tab. 1; OpenEQA baselines (Blind LLMs, Socratic LLMs with frame or scene-graph captions) tab. 2; GOAT-Bench baselines (SenseAct-NN, Modular GOAT, Modular CLIP on Wheels) tab. 3.
• Superseded by: not known from content already in this wiki.