Social-LLaVA: Enhancing Robot Navigation through Human-Language Reasoning in Social Spaces

In plain words

Robots that move through crowds have to act the way people do: slow down for a person crossing their path, wait in line, or go around a conversation instead of through it. Most older methods either follow hand-written rules or copy what a person did in a recorded demonstration. This paper asks whether a robot could instead describe what it sees in ordinary language, reason about it in words, and then say what to do.

To test that idea, the authors built a question-and-answer dataset from recordings of a robot moving among people. Each example pairs a camera image with questions about what is visible, what people are likely to do, what the robot should do next, and why. They then tuned an existing vision-language model on these examples. Judges rated its answers higher than those of two general-purpose models that were only prompted. The work is a proof of concept: the answers are text, and turning them into real robot motion is left open.

TL;DR

• The paper contributes SNEI, a human-annotated visual question-answering dataset for social robot navigation, and Social-LLaVA, a LLaVA-v1.5-7B model fine-tuned on it §III-B §IV-A.
• Fifteen human judges preferred Social-LLaVA's answers over prompted GPT-4V and Gemini 1.5 Pro on all five task types tab. 1 (author-reported).
• No code or dataset release is stated; the robot demo maps language to three hard-coded motions §IV-C.
• Use it as a dataset reference and a sanity check for language-based social reasoning, not as evidence of a deployed navigation stack.

Problem

Hand-crafted social navigation rules and learned models trained on demonstrations often fail to capture the context that people use to judge what is socially acceptable §I. Off-the-shelf VLMs struggle with spatial relations between people and with predicting their movement §III-A. Existing VQA datasets for driving or general images do not cover crowded, unstructured public spaces §I §III-A. The paper assumes that human-annotated, language-based reasoning about a scene can supply the missing link between perception and a socially compliant action §I §III-A.

Method

Dataset (SNEI). The authors manually select about 2K scenarios from the SCAND robot-navigation dataset in which the robot directly interacts with people §I §III-B. For each scenario, annotators fill in two kinds of labels. Categorical labels use fixed options for context (environment type, crowd density, indoor or outdoor, terrain), robot (goal, movement direction, speed level, action intent), obstacles (type, proximity, position) and agents (type, proximity, position, current action, facing direction) §III-B. Free-form answers cover five aspects: perception of the robot's view, prediction of nearby agents' movement, chain-of-thought reasoning that leads to a high-level instruction, the final action in words, and an explanation §III-B. The result is over 40K human-annotated VQA pairs §I §III-A.

Model (Social-LLaVA). The authors take LLaVA-v1.5-7B, which combines a vision encoder, a language model and a vision-language connector, and fine-tune it on SNEI with Low-rank Adaptation (LoRA) §IV-A. The paper states that it makes no algorithmic novelty claim; the model is a proof-of-concept use of the dataset §IV-A.

Evaluation. There is no automatic benchmark. The authors sample 50 VQA items §I, feed them to Social-LLaVA, GPT-4V and Gemini 1.5 Pro, and have human raters score each answer from 1 to 5 §IV-B. The baselines receive a long task description and three worked examples for prompt tuning; they are not fine-tuned §IV-B. Scores are averaged per task type §IV-B tab. 1.

Robot demo. The model's language output is mapped to three low-level motions (go straight, turn left, turn right) by a hard-coded rule, and the authors show one real-world run §IV-C fig. 2.

# SNEI record and Social-LLaVA fine-tuning  [§III-B] [§IV-A]
# Input:  SNEI records (image, question, answer); LLaVA-v1.5-7B weights
# Output: LoRA-adapted vision-language model

def make_record(image, question, answer):
    # One VQA item: an image and free-form text  [§III-B]
    # Question wording and answer format are not stated.
    return {"image": image, "question": question, "answer": answer}

def fine_tune(base_model, records, epochs):
    # Attach LoRA adapters to the frozen base  [§IV-A]
    # Rank, alpha, target layers and learning rate are not stated.
    model = add_lora_adapters(base_model)
    # Train for 15 epochs with batch size 4  [§IV-A]
    for epoch in range(epochs):
        for batch in batches(records, batch_size=4):
            # Loss function is not stated in the paper.
            loss = loss_fn(model, batch)
            # Optimizer and schedule are not stated in the paper.
            update(model, loss)
    return model

Hyperparameters

Results

Author-reported. Setup: SNEI-based fine-tuning of LLaVA-v1.5-7B against GPT-4V and Gemini 1.5 Pro, both only prompted; 50 VQA items; scores from fifteen human judges on a 1-5 scale §IV-B tab. 1. The table reports averages per task type. The paper does not state the sample size behind each cell.

The paper calls the gap "significantly higher" §IV-B, but reports no statistical test, variance or confidence interval. The qualitative example shows a narrow-passage case where both baselines suggest moving forward and GPT-4V hallucinates across all five aspects §IV-B fig. 3.

Paper vs code

No official code. The abstract links a project page (https://cs.gmu.edu/~xiao/Research/SNEI/). It lists no code repository, dataset download or license. GitHub searches for "social-llava", "snei", "snei navigation" and "social robot navigation vlm dataset" returned no repository from the authors or their labs. The paper states no dataset release. Nothing was cloned.

Pitfalls

• Dataset availability is unstated. The paper does not say SNEI is released, nor under which license. Do not assume the annotations are available. The arXiv license covers the paper text, not the dataset.
• Source data licence not checked. SNEI is built on SCAND §III-B. The paper does not state SCAND's license, and this card did not verify it.
• Judge protocol is vague. The 1 to 5 scale has no published rubric, the raters' background is not stated, and it is not stated whether raters knew which model produced an answer §IV-B.
• Baselines are prompted, the proposed model is fine-tuned. The comparison is asymmetric: the baselines got three hand-picked examples and a long prompt, while Social-LLaVA was trained on about 2K scenarios §III-B §IV-B. The paper itself says the baselines might improve if they could be fine-tuned, but that could not be tested §IV-B.
• Small evaluation. The quantitative claim rests on 50 VQA items and fifteen judges, with no split or seed reporting §IV-B.
• Annotation ambiguity. The authors note that there may be several, or no agreed, socially compliant actions for a scene, and that the annotation effort faces this §V-B. No inter-annotator agreement is reported.
• Single frames only. Video and temporal context were left out §V-C. Predictions about moving humans come from one image.
• Robot action is not learned. The real-robot run maps language to three motions by rule §IV-C. Grounding language into motion is named as open §V-E.
• Naming. The paper writes both "GPT4-V" (table) and "GPT-4V" (text). This card uses GPT-4V.
• Figure captions. The qualitative figure marks hallucinated phrases in red and accurate phrases in blue fig. 3. The judgment is the authors'.

Lineage

• Builds on: LLaVA §II-B (visual instruction tuning, base of LLaVA-v1.5-7B §IV-A); LoRA §IV-A.
• Data source: SCAND robot-navigation dataset §I §III-B.
• Inspired by: DriveLM's graph VQA for driving §III-A §V-C.
• Compared against: GPT-4V and Gemini 1.5 Pro, prompted only §IV-B.
• Related: social navigation rules and learned demonstration methods §II-A.