InstruGen: Automatic Instruction Generation for Vision-and-Language Navigation Via Large Multimodal Models

In plain words

A robot or agent that follows spoken directions, such as "go through the kitchen and stop at the sink", learns from pairs of a route and a sentence that describes it. Writing those pairs by hand is expensive, and template-based generators produce stiff, repetitive sentences that often do not match the route.

InstruGen takes a different route. It starts from ordinary house-tour videos posted on YouTube, cuts them into a path through the rooms, and describes each room and each turn. A large multimodal model then writes a natural instruction for that path. Because such models sometimes invent objects or turns, the pipeline checks the written instruction against the path it was built from, regenerates it on mismatch, and cleans leftover artifacts with simple text rules.

The generated pairs are used to pretrain a navigation agent before it is fine-tuned on standard benchmarks. The authors report that this helps the agent in rooms it has not seen during training.

TL;DR

• Generates route-and-instruction pairs from YouTube house-tour videos with a large multimodal model (ChatGPT-4V), instead of templates. §1
• Adds a consistency check that extracts (room, action) pairs from the text and compares them with the route, plus regular-expression clean-up. §3.2
• Pretrains a ViLBERT-like agent with three pretext tasks. §3.3
• Author-reported: R2R val-unseen SR 71.10 and RxR zero-shot val-unseen SR 29.35, both with the best results in their tables. tab. 3 tab. 5
• The linked repository contains only a README. Nothing can be re-run from it.

Problem

Vision-and-language navigation (VLN) agents generalize poorly to unseen buildings, which the authors attribute to too few realistic training scenes and too few high-quality route-instruction pairs §1. Earlier data methods either change visual features in simulators, use web image captions, or build instructions from fixed templates or rules §1 §2.2. The authors argue that templates do not align instructions with what the agent sees at each step §1. Using YouTube house tours as scenes avoids the cost of building simulated scenes but raises the problem that the video has no route or action labels, so these must be estimated §3.1.

Method

The pipeline has three stages: trajectory generation, instruction generation and trajectory judgment §3 fig. 2. Section numbers follow the compiled paper: §3.1 is Trajectory Generation, §3.2 Instruction Generation, §3.3 Trajectory Judgment.

§3.1 Trajectory generation.

1. Take 3879 YouTube house-tour videos as indoor scenes §3.1. Of these, 3679 are used for training and 200 are held out §4.1.
2. Sample trajectories in the style of Lily, where each path is a sequence of room nodes and transition nodes §3.1.
3. Label each room node with its room type and salient objects, using CLIP. Transition nodes are not labelled §3.1.
4. Estimate the action between consecutive room nodes with an action inverse model (Kumar et al., 2019), instead of annotating video by hand §3.1.

§3.2 Instruction generation.

1. Build a prompt that gives the ChatGPT-4V model the sequence of (room image, room label, action) triplets, with a transition image between rooms and the last node marked "Stop" §3.2 fig. 2.
2. State the requested granularity: concise, or detailed with environment descriptions and key objects. Tell the model not to describe anything beyond the given labels, and include one example of the output format §3.2 fig. 2.
3. Consistency module: ask the model to extract (room, action) pairs from its own instruction by proximity §3.2 fig. 3. Compare these pairs with the route's labels. If they differ, regenerate §3.2.
4. Regex module: remove stray special characters and fragments of the prompt, and fix wording with rules §3.2.
5. The paper's ablation adds a manual review step for the best row only tab. 1.

# Instruction generation with a consistency check  [§3.2] [fig. 2]
# Input:  triplets (room image, room label, action) for one route
# Output: instruction text

def generate_instruction(triplets, granularity):
    # Build the prompt from the objective, output format and triplets  [§3.2]
    prompt = build_prompt(triplets, granularity)
    # Regenerate until the extracted pairs match the route  [§3.2]
    # Retry limit and match rule: not stated
    while True:
        # Ask the LMM for a candidate instruction  [§3.2]
        draft = lmm(prompt)
        # Extract (room label, action) pairs by proximity  [§3.2] [fig. 3]
        extracted = lmm(extraction_prompt(draft))
        if matches(extracted, ground_truth(triplets)):
            break
    # Remove special characters and prompt fragments with regex rules  [§3.2]
    return regex_cleanup(draft)

§3.3 Trajectory judgment (pretraining). The model follows a ViLBERT-like design, adapted from Lily and Airbert §3.3. The visual input groups region features for each node, with a separator token per node eq. 1. The text input is a [CLS] token, the instruction tokens and a [SEP] token eq. 2. Three pretext tasks are used:

• Masked language modeling and masked vision modeling: replace random words or image regions with [MASK] and predict them §3.3.
• Path-instruction judgment: the generated route with its instruction is a positive example. The same instruction with the route's node order shuffled is a negative example §3.3.
• Path ranking: for several candidate routes from the agent, the model scores how well each one matches the instruction, and the highest-scoring route is chosen §3.3.

# Path-instruction judgment examples  [§3.3]
# Input:  one generated route and its instruction
# Output: two labelled examples

def make_judgment_examples(route, instruction):
    # The generated route is a positive example  [§3.3]
    positive = (route, instruction, 1)
    # Shuffling the node order gives a negative example  [§3.3]
    negative = (shuffle_nodes(route), instruction, 0)
    return [positive, negative]

Instruction granularity. R2R# is the R2R route set re-described with InstruGen. R2R* is a mixture of R2R# and the original R2R instructions. The mix ratio is not stated §4.2 tab. 2.

Hyperparameters

Results

Author-reported throughout. Setup: R2R has 90 buildings and over 21,000 instructions; RxR is a multilingual extension §4.1. Each row below is transcribed from the compiled paper.

Verification ablation on R2R val-unseen tab. 1

Instruction granularity, pretraining and fine-tuning sets on R2R tab. 2

Testing is on R2R in every row.

Val-unseen columns for rows 1 to 6.

Comparison on R2R tab. 3

The paper does not say whether the baseline rows were re-run under this paper's setup or copied from earlier papers tab. 3.

One-shot: success rate with one training environment tab. 4

Zero-shot on RxR val-unseen (model trained on R2R* only) tab. 5

Figures 5 and 6 compare pretraining curves with Lily on the same sampled data, and show navigation paths on real scenes. No numbers are read from the figures here fig. 5 fig. 6.

Paper vs code

The abstract links https://github.com/yanyu0526/InstruGen. I checked it on the default branch (main) at commit d57181d820444eb48d81eccc11dfc26c0d49ed14. The repository has one commit and only a README, which reads # InstruGen README.md. It has no LICENSE file, no generation or verification code, no prompts, no configs, no training or evaluation code, no model weights and no dataset.

So there are no code-versus-paper discrepancies to report, and nothing in the paper's method can be checked against code. The paper's claim that a dataset is contributed §5 has no matching release. The one-shot and zero-shot setups can only be checked against the paper text.

Pitfalls

• Nothing to run from the release. The only released artifact is the README. Prompts appear only as figures fig. 2 fig. 3, so exact wording, output parsing and the regex rules must be rebuilt from descriptions.
• Best row uses manual review. The 71.10 SR headline includes a manual review step (row 4 of tab. 1). The automatic-only row reaches 70.9 SR, which is the figure to compare when you cannot review by hand. How much manual review was done, and on which samples, is not stated.
• The check cannot catch label errors. The consistency module compares the instruction with the route's own labels §3.2. Those labels come from CLIP and an estimated action model §3.1. An error in either passes through the check unless it breaks the instruction's logic. This follows from the method as written, not from a stated test.
• Match rule and retry limit are unstated. The paper says a mismatch triggers regeneration. It does not say how pairs are compared (exact match, set match, or tolerance), how many retries are allowed, or what happens after the last retry §3.2. Cost per accepted pair therefore cannot be estimated.
• Yield and size of the generated set are unstated. The number of instructions generated, the number accepted, and the number of instructions per route are not stated §3.1 §4.1. The 3879 videos are the only size given.
• Hallucination reduction is measured indirectly. The paper does not report an instruction-level error rate. The evidence is the ablation tab. 1, one qualitative example fig. 4, and agent success rates.
• Split naming is ambiguous. The 200 held-out videos are "unseen environments" in the pretraining data §4.1. The "Val-Unseen" columns are R2R's validation split. Do not treat them as the same split.
• Model selection is not described. It is not stated whether checkpoints or settings were chosen on the R2R val-unseen split that the tables report.
• Baselines may not be comparable. Table 3 mixes methods with different pretraining data and features. Whether the baseline numbers were copied or re-run is not stated tab. 3.
• Variance is not reported. The one-shot result averages five random sets, with no spread tab. 4. Some margins are under one point (for example 51.25 against 50.86 for Lily). Treat them as indicative only.
• Zero-shot baselines have gaps. DUET and Lily have no TL or NE values in the RxR table tab. 5. Whether they were run under the same protocol is not stated.
• Training recipe is incomplete. Learning rate, optimizer, warm-up, loss weights between the three pretext tasks, and the region feature extractor are not stated. The batch of 4 is spread over 4 GPUs, one sample each §4.1.
• Route format limits use. The paper says it samples discrete, R2R-like routes, which may not suit continuous navigation §5.
• Figure and table numbers differ from the LaTeX source. The source numbers figures differently from the compiled PDF. This card uses the compiled numbers: tables 1 to 5, figures 1 to 6, equations 1 and 2.

Lineage

• Builds on: Lily (Lin et al., ICCV 2023), which uses YouTube house tours with route-instruction pairs for pretraining §2.2. Lily's arXiv id is not given in this paper.
• Builds on: the ViLBERT-style backbone, and the design of Airbert and Lily §3.3.
• Compared against: Follower, Speaker and Speaker-Follower (Fried et al.), ProbES, VLNbert, Airbert, Lily tab. 3. For RxR zero-shot: DUET and Lily tab. 5.
• Superseded by: not known from the content of this wiki.