Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models

In plain words

Sometimes a humanoid robot has to put a hand or a foot on a nearby surface to keep its balance while it reaches for something far away. Deciding where that support should go is hard to automate, because a good spot depends on things a robot cannot easily model, such as whether a surface is fragile or not allowed. This work lets a person say where in ordinary language, for example "place your right hand on the book", and shows the chosen point on the robot's camera image.

The system splits the job across several models. A language model reads the instruction and works out whether it names an object or a place next to one. For a named object, a vision-language model finds it and the point is taken from that object's region. For a place next to something, a detector finds the objects and the language model writes a small arithmetic formula over their positions, which the system then computes. If the point is wrong, the person can say something like "move a bit to the right", and the system adjusts it. Once the person confirms, the robot moves a hand to that point.

TL;DR

Words2Contact is an instruction-to-contact pipeline for humanoids: an LLM classifies each utterance, a VLM grounds the named object or box, and a generated formula or a segmentation gives a pixel target. Users refine it by corrections before confirmation, and the Talos robot then executes the contact. Author-reported, the best LLM+VLM mix reaches a median overall success of 0.61 on a 78-sample set, against 0.13 for random points tab. 1. The official code (BSD-3-Clause) covers only the LLM/VLM part; the controller and the interaction loop are not in that repository.

Problem

Multi-contact whole-body controllers need the contact locations as input, and a planner usually supplies them from perception and environment models §1. Choosing support surfaces needs understanding the world that is hard to model, such as fragile, slippery or forbidden surfaces, so a human expert often has to decide §1. Language is a natural channel for that, but it is open-ended: people describe the same object in many ways, which rules out keyword matching §1. LLMs and VLMs offer a route to this, yet there was no agreed way to connect perception, language and robot actions for this task §1. Earlier language approaches for robots relied on rigid rules or fixed vocabularies §2. The paper's assumptions are that a remote operator can see the robot's camera image and confirm the final point visually §5, and that the objects named in the instruction can be detected in the image §3.3.

Method

The paper describes an interactive pipeline. The first instruction gives a predicted point. The operator can then correct it or give a new instruction until they confirm; the confirmed point goes to the controller §3 fig. 2. Each instruction is put into one of three categories: prediction, correction or confirmation §3.2. The function names in the pseudocode below are ours; the paper gives no algorithm box.

1. One LLM, several system prompts. A single LLM is used, and its system prompt changes at each step. Output is forced to JSON: grammar-constrained sampling for open models and the provider's JSON mode for GPT-3.5 §3.1 fig. 3.
2. Module selector. Five-shot classification into the three categories, with a logit bias toward the three label words §3.2.
3. Prediction module. A prompt analyser LLM returns a chain of thought, the object names and whether the position is absolute (on an object) or relative (to one or more objects) §3.3 fig. 4.
   • Absolute: a language-grounded segmentation model (CLIPSeg or CLIP Surgery) produces a heatmap for the named object. The contact point is the heatmap's maximum, given by the argmax below §3.3.
   • Relative: an open-set detector (GroundingDINO or Florence-2) gives boxes. Each box is written into the prompt as text, and the LLM writes two arithmetic expressions for the target's x and y. The expressions are parsed and computed outside the LLM. In preliminary experiments this beat having the LLM compute the number itself by about 10% §3.3.
4. Correction module. The LLM names the objects in the correction, the detector gives their boxes, and the LLM proposes a new candidate point from the current target, the boxes, the instruction and the earlier turns §3.4 fig. 5. A correction may name no object, such as "a bit to the right" §3.4.
5. Confirmation and control. After confirmation the LLM is queried once more with an end-effector prompt that picks the left or right hand and the task type (support contact or reaching) §3.5 fig. 3. The image point is turned into a 3D point in the camera frame using the point cloud, then into the robot's world frame with forward kinematics §3.5 fig. 6. A spline Cartesian trajectory from the current hand pose leads to it. SEIKO, a sequential-QP retargeting plus a multi-contact admittance controller, computes joint positions that respect joint, velocity and balance limits and regulates the contact forces §3.5. An unreachable target makes the robot approach as closely as it can without losing balance §3.5.

Key formula, inline in the text (no numbered equation exists in the paper):

[i_{\max}, j_{\max}] = \arg\max_{i,j} \, H_{ij} §3.3

Here H is the segmentation heatmap, upscaled to the image size §3.3.

# Interaction loop: one module per instruction  [§3] [fig. 2]
# Input:  instructions (text, in order), image (H, W, 3)
# Output: confirmed point (x, y), end-effector, task type

def contact_session(instructions, image):
    # The first instruction always starts with a prediction  [§3] [fig. 2]
    point = predict_point(instructions[0], image)
    history = [instructions[0]]
    for instruction in instructions[1:]:
        # Classify the instruction into one of three modules  [§3.2]
        category = module_selector(instruction)
        if category == "confirmation":
            break
        if category == "correction":
            point = correct_point(instruction, point, image, history)
        else:
            point = predict_point(instruction, image)
        history.append(instruction)
    # Hand and task type from one final LLM query  [§3.5] [fig. 3]
    end_effector, task_type = select_end_effector(history)
    return point, end_effector, task_type

# Prediction module: absolute or relative position  [§3.3] [fig. 4]
# Input:  instruction (text), image (H, W, 3)
# Output: pixel point (x, y)
# x is the column, y the row  [code: words2contact/saygment.py#L19-L24]

def predict_point(instruction, image):
    # Chain of thought, object names and position type from the LLM  [§3.3]
    analysis = prompt_analyzer(instruction)
    if analysis.position == "absolute":
        # Heatmap of the named object; the point is its maximum  [§3.3]
        heatmap = segment_by_text(image, analysis.objects)
        row, column = argmax_2d(heatmap)
        return (column, row)
    # Boxes for the named objects from an open-set detector  [§3.3]
    boxes = detect_by_text(image, analysis.objects)
    # LLM writes arithmetic over the box numbers, not the answer  [§3.3]
    expressions = write_expressions(instruction, boxes)
    # Parse and evaluate both expressions outside the LLM  [§3.3]
    return (evaluate(expressions.x), evaluate(expressions.y))

# Correction module  [§3.4] [fig. 5]
# Input:  correction (text), current point (x, y), image, earlier turns
# Output: new pixel point (x, y)

def correct_point(correction, point, image, history):
    # Object names in the correction; the list may be empty  [§3.4]
    objects = find_objects(correction)
    # Boxes for those names; none if the list is empty  [§3.4]
    boxes = detect_by_text(image, objects)
    # New target from the current point, boxes and earlier turns  [§3.4]
    expressions = write_correction(correction, point, boxes, history)
    return (evaluate(expressions.x), evaluate(expressions.y))

Hyperparameters

The paper gives almost no numeric settings for the models. The "Where" column cites the paper or the code (commit eab11b9). Values marked "not stated" are absent from both.

Results

Author-reported. Setup: 78 image, prompt and mask tuples at 1280 × 720 §4.1. The LLMs are Calme-7b-Instruct, mixtao-7bx2-moe and GPT-3.5-turbo; the detectors are GroundingDINO and Florence-2; the segmenters are CLIPSeg and CLIP Surgery; the baseline is random point sampling §4.1. Each cell is the median with the 25th and 75th percentiles in brackets. The paper does not state how many samples each cell used.

The paper's main summary is that splitting the task into sub-tasks lets smaller models match larger ones §4.1. The text says the best combination selects appropriate contacts in about 70% of absolute cases and 50% of relative cases §4.1.

Pilot study (11 volunteers, no prior use). Phase 1 gave minimal instructions and did not mention the correction step. Phase 2 explained the system and gave two prediction and two correction examples. Each volunteer then located 10 targets on 10 images §4.2. The distance is between the centres of the target circle and the placed point. The paper reports that the median distance at the fifth prompt was 21 px, comparable to the 18 px target radius, after the explanation §4.2 fig. 8. Phase 1 medians are only shown in the figure fig. 8.

Real robot (Talos, four reaching settings). The prompts and corrections per setting are listed in §4.3 and captioned in fig. 9. The robot performed all four tasks successfully; the text gives no number of trials §4.3.

Paper vs code

Compared at commit eab11b9. The repository contains the LLM and VLM part only. The README points to a separate controller repository, which was not checked README.md.

• Interaction loop and dispatch. The paper describes the module loop and the absolute/relative dispatch §3 fig. 2. In the code they are methods of Words2Contact words2contacts.py [code: words2contact/words2contacts.py#L436-L477], but main.py calls only predict, which never reaches the segmentation branch and never loops main.py.
• Classifier few-shot count. The paper uses five shots for every module §3.2. The classifier file has six examples [code: words2contact/prompts/prompts.json#L5-L30].
• Logit bias. The paper biases the three label words §3.2. The code adds +5 to four token ids and only on the GPT branch [code: words2contact/words2contacts.py#L200-L204].
• Conversation history. The paper says the correction prompt includes the history §3.4. The code records it but never sends it [code: words2contact/words2contacts.py#L405] [code: words2contact/words2contacts.py#L386-L393].
• Models. The paper names Calme-7b-Instruct, mixtao-7bx2-moe and GPT-3.5 §4.1. The entry point defaults to Calme-7B with ChatML main.py. The class defaults to a 10.7B Tess model with Orca-Vicuna, and to CLIP Surgery as segmenter words2contacts.py. mixtao is not referenced in the code. The best paper combination uses CLIPSeg tab. 1.
• Object description format. The paper's example reads "Cup is at [100,150] with width=120 and height=90" §3.3. The code writes the box centre and size as "(x, y), with dimensions: WxH" geom_utils.py.
• Math evaluation. The paper says a Python parser computes the expressions §3.3. The code has its own tokenizer and recursive-descent evaluator math_pars.py [code: words2contact/math_pars.py#L144-L156].
• Controller. The paper's controller and point-cloud steps are in §3.5. They are not in this repository README.md.

No further discrepancies were checked for the control module, which is not in the repository.

Pitfalls

• The loop is not in the repository. Module selection, correction, confirmation and the absolute branch exist only as methods, so an implementation must build the loop and the dispatch from §3 and fig. 2, not from main.py main.py words2contacts.py.
• The shipped path is relative-only and always detects. predict runs the detector and then the LLM arithmetic, even for "on top of the book", which the paper sends to segmentation §3.3 words2contacts.py.
• Broken relative dispatch (read from code, not run). predict_rel calls self.predict(prompt, img, False, objects) on a method that takes three positional arguments, which raises a TypeError [code: words2contact/words2contacts.py#L482-L483] [code: words2contact/words2contacts.py#L278]. It is reachable only through prompt_analyzer, which nothing calls yet.
• Broken local correction (read from code, not run). On the local-model path the code reads correct_x and correct_y [code: words2contact/words2contacts.py#L430-L433], but the grammar it uses forces the keys use_eef, task_type, target_x and target_y [code: words2contact/grammar/eef_grammar.gbnf#L2-L3]. The lookup would raise a KeyError outside any try block. The GPT branch is the working reference: it clips the result and falls back to the old target [code: words2contact/words2contacts.py#L406-L419].
• Silent (0, 0) on any failure. If the JSON or the arithmetic fails, predict returns the point (0, 0) with a generic reason [code: words2contact/words2contacts.py#L355-L358]. The robot would aim at the top-left corner. An implementation needs an explicit failure state.
• Conversation history is not sent. The paper says it is included §3.4, but the code only appends it to self.history [code: words2contact/words2contacts.py#L405].
• Image flip and axis convention. main.py flips the image vertically before inference main.py. The correction prompt treats up as positive y prompts.json, so coordinates are in the flipped frame with y pointing up. The paper does not state this frame, and the controller must use the same one.
• Clipping is off by one. Coordinates are clipped to [0, width] and [0, height] words2contacts.py. Clip to width - 1 and height - 1 in a re-implementation.
• Box convention is ambiguous. The BoundingBox docstring says the anchor is the bottom-left corner geom_utils.py. The LLM sees the centre and size geom_utils.py. Detector boxes are stored as x, y, width and height from corner coordinates [code: words2contact/yello.py#L166-L172]. The paper's "[100,150]" example does not say whether it is a centre or a corner §3.3. Pick one and test it.
• Malformed few-shot examples. One prediction example wraps its answer in an extra "assistant" key prompts.json. Two object-detection examples leave their lists unquoted [code: words2contact/prompts/prompts.json#L49] [code: words2contact/prompts/prompts.json#L53]. The model sees these as demonstrations.
• Local arithmetic grammar may block parentheses (read from grammar, not run). The nested term must begin with a quote [code: words2contact/grammar/rel_pos_grammar.gbnf#L11-L12], yet the prediction examples use parentheses prompts.json. The grammar also has no sign or decimal in numbers [code: words2contact/grammar/rel_pos_grammar.gbnf#L13]. Check with llama.cpp before relying on it.
• Local module selector uses the wrong examples. The local branch passes the prediction examples as the few-shot set [code: words2contact/words2contacts.py#L215-L216]. The GPT branch uses the classifier examples words2contacts.py.
• Detector deduplication does nothing. seen_classes is created and tested but never filled yello.py, so every box above threshold is kept, not one per class.
• Multi-object prompts are underspecified. CLIPSeg returns after the first object saygment.py. Florence-2 rejects more than one object saygment.py, and the Florence-2 detector uses only objects[0] yello.py. The paper does not say how multi-object prompts are handled §3.3.
• CLIPSeg detector boxes are corrupted (read from code, not run). resize_segmentation multiplies the heatmap by the width ratio before it is converted to 8-bit yello.py [code: words2contact/yello.py#L286]. Values above 1 wrap around. The default detector, GroundingDINO, is not affected.
• Truncation. / is float division and int() truncates toward zero [code: words2contact/math_pars.py#L122] [code: words2contact/words2contacts.py#L346-L347]. A computed 399.9 becomes 399.
• Hidden costs and fixed devices. VLMs run on cuda:0 by default [code: words2contact/saygment.py#L32] yello.py. The constructor loads both detector and segmenter even if only one is used [code: words2contact/words2contacts.py#L147-L149]. The shipped path makes two LLM calls per instruction plus a detector pass [code: words2contact/words2contacts.py#L283-L289] [code: words2contact/words2contacts.py#L335-L338]. The paper reports no latency.
• Success is not defined. The paper says the masks mark acceptable contact areas §4.1 fig. 7. That a hit means a point inside the mask is an inference, not stated.
• The "best" claim is ambiguous. The text calls GPT-3.5 + GroundingDINO + CLIPSeg the best combination §4.1, and the table bolds it on the overall column tab. 1. GPT-3.5 + Florence-2 + CLIPSeg has the higher absolute median (0.74 against 0.71) tab. 1. The sample size per cell and the statistic behind the quartiles are not stated.
• Pilot design. Phase 2 always follows phase 1, so practice and the explanation are confounded. The paper calls the improvement significant but reports no test §4.2.
• Robot evidence is thin. Four scenes, one operator (the text says "the operator"), no trial counts and no timing §4.3.
• Coordinate frame and gaps for implementers. The control module (point cloud to world frame, spline parameters, SEIKO gains, control rate) is described only in words §3.5. These values are not stated, and the repository does not contain them README.md.

Lineage

• Builds on: SEIKO, the multi-contact whole-body controller of Rouxel, Ivaldi and Mouret (RA-L 2024), used here for execution §2 §3.5. No arXiv id is stated in the paper, and there is no card for it.
• Uses: Florence-2 §4.1, see Florence-2. GroundingDINO, CLIPSeg and CLIP Surgery are used for detection and segmentation §3.3 §4.1; no cards exist for them yet.
• Compared against: random point sampling and the other LLM and VLM combinations tab. 1 §4.1.
• Related approaches named by the paper, without cards: Inner Monologue, Code-as-Policies, LILAC and DROC §2.
• Superseded by: not known from this wiki's content.