LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents

In plain words

LLaVA-Plus is a chatbot that can look at a picture and then call specialised vision programs while it answers. A typical vision chatbot answers only from what it learned during training. This one can, in the middle of a conversation, ask a detector to find objects, a segmenter to outline them, a captioner or text reader to read the image, or an image generator to make a new picture. It then writes its reply using those results. The picture stays in view for the whole conversation, not just the first message.

The model learns this from examples that show when to call a tool, which tool to use, what to pass to it, and how to reply with the result. Each example has three parts: the model's reasoning, the tool call (a list that is empty when no tool is needed), and the reply the user sees.

TL;DR

• LLaVA, a large multimodal model (LMM), is trained with visual instruction tuning on dialogues that include tool calls, so the model itself chooses and fills in tools from a skill repository §2, §2.1.
• Each assistant turn has three fields: thoughts, actions (a JSON list of API_name and API_params) and value §2.1.
• The skill repository covers detection and grounding, segmentation, tagging, captioning, OCR, retrieval, image generation and editing, visual-prompt segmentation, and composed skills. Most training data is labelled by GPT-4 §2.2, tab. 2.
• Author-reported, 7B: LLaVA-Plus (All Tools) scores 69.5 against 57.1 for LLaVA on LLaVA-Bench In-the-Wild tab. 4. The 13B model scores 35.0 against 32.5 for LLaVA on MM-Vet tab. 6.
• The paper gives no training hyperparameters. The official repository does, and this card uses them only as code-stated values [Hyperparameters].

Problem

• Prompt-based tool chaining (VisProg, ViperGPT, Visual ChatGPT, MM-REACT) cannot reliably choose and combine tools from a large toolset, and it is not robust enough for real multimodal tasks §1.
• End-to-end LMMs are hard to extend with new skills such as detection, segmentation and generation, because their outputs are mostly text §1, §2.
• Tool-tuned LLMs (GPT4Tools, Gorilla) only receive the image when a tool is called, so the planner loses the image during the rest of the session §1, §3.
• The method assumes that tool outputs can be written as text the LMM reads, and that a base LMM such as LLaVA can be extended with extra data without losing its chat ability §2.1, §2.3.

Method

LLaVA-Plus keeps the LLaVA data format and adds a second kind of dialogue. In the skill-oriented dialogue, the model writes a tool call, receives the tool output as a new human turn, and then writes the final answer. The paper calls the first kind user-oriented and the second skill-oriented §2.1.

1. Unified data format. A sample is a human turn (image and question) followed by an assistant answer §2.1 eq. 1:

   \texttt{Human}: \mathbf{I}_q \; \langle\backslash n\rangle \; \mathbf{X}_q \; \texttt{<STOP>} \; \texttt{Assistant}: \mathbf{X}_{answer} \; \texttt{<STOP>} eq. 1

2. Skill-oriented dialogue. The full sequence has four turns §2.1 eq. 2. The assistant emits a skill-use turn. The human side returns the skill result. The assistant then emits the answer turn. Only the assistant spans get loss §2.1.

   \texttt{Human}: \mathbf{I}_q \; \langle\backslash n\rangle \; \mathbf{X}_q \; \texttt{<STOP>} \; \texttt{Assistant}: \mathbf{X}_{skill\_use} \; \texttt{<STOP>} \; \texttt{Human}: \mathbf{X}_{skill\_result} \; \texttt{<STOP>} \; \texttt{Assistant}: \mathbf{X}_{answer} \; \texttt{<STOP>} eq. 2

3. Three fields per assistant turn §2.1:

   • thoughts: reasoning that decides whether a tool is needed and which one.
   • actions: a list of function calls. Each item has API_name and API_params. An empty list means no tool is invoked.
   • value: the natural-language text. In the answer turn it is the reply shown to the user. In a skill-use turn it describes the tool call.

4. Skill result. The tool outputs are written as text first, then the original question is repeated §2.2 (iii).

5. Data generation §2.2. GPT-4 writes questions that need a tool. A random question is rewritten for diversity §2.2 (i). The thoughts and skill-use value come from preset responses with rewriting. The actions field is set directly from the tool template §2.2 (ii). The final answer comes from a language-only GPT-4 that sees the earlier questions, tool outputs and image context (ground-truth captions, object coordinates and categories) §2.2 (iv).

   • For tools that need an argument, such as a category to detect, the placeholder is replaced by categories present in the image. Separately, GPT-4 writes full multi-turn dialogues from two hand-written seed samples §2.2.

6. Skill set tab. 2. Individual skills are detection (Grounding DINO), semantic segmentation (OpenSeeD), instance segmentation (Grounding DINO + SAM), caption + grounding (BLIP-2 + Grounding DINO), tagging + grounding (RAM + Grounding DINO), caption (BLIP-2), tagging (RAM), OCR (EasyOCR), retrieval (CLIP retrieval), image generation (Stable Diffusion), image editing (InstructPix2Pix), and three visual-prompt segmenters (SAM, Semantic-SAM, SEEM). Composed skills are a mix of detection, segmentation, tagging and caption, interactive segmentation with inpainting, semantic segmentation with generation, and generation or editing with a social-media post tab. 2.

7. Extended skills §2.2.2 app. B:

   • External knowledge: InfoSeek questions and images are used. CLIP retrieval finds top-K items in LAION. ChatGPT keeps only the questions whose answers the retrieved text supports.
   • Generation: JourneyDB prompts teach Stable Diffusion. Instruct-Pix2Pix edits are used directly.
   • Visual prompts: a random point is written as text and appended to the question. Semantic-SAM supplies multi-granularity masks.
   • Composition: four scenario types are listed in §2.2.2.

8. Training mix §2.3. The paper merges the tool-use data with LLaVA-158K. In the LLaVA samples the actions list is empty, and the thoughts are generated by ChatGPT app. A.

   • All Tools: outputs of all understanding tools except segmentation are run on the input image. Their results are written as symbolic text and added to the image context in both training and evaluation §2.3.
   • Fly: only tool results relevant to the instruction are added. Quantitative numbers use 81K understanding samples. The demo uses the full data §2.3.

9. Serving §2.3. FastChat runs the web server, model workers and controller. The 7B model and all tools fit on one 80 GB GPU. The user sees only the answer's value. Skill-oriented turns are hidden unless debugging §2.1.

Serialization and the serving loop, as implemented in the official code:

# Serialize one assistant turn into the three tagged fields  [§2.1]
# Tags and JSON layout: official code  [code: llava/mm_utils.py#L134-L146]
def serialize_turn(thoughts, actions, value):
    # One tagged line per field, with the action list as JSON
    lines = []
    lines.append('"thoughts🤔" ' + thoughts)
    lines.append('"actions🚀" ' + json_dumps(actions))
    lines.append('"value👉" ' + value)
    return "\n".join(lines) + "\n"

# Build one skill-oriented training sample  [§2.1] [eq. 2]
# Human turns, incl. the skill result, get no loss
# [code: llava/train/train.py#L419-L499]
def build_sample(image, question, skill_use, skill_result, answer):
    # Turn 1: the user's question with the image  [eq. 2]
    turns = [("human", image, question)]
    # Turn 2: the model's tool call  [§2.1]
    turns.append(("gpt", serialize_turn(*skill_use)))
    # Turn 3: tool output, then the original question  [§2.2 (iii)]
    turns.append(("human", skill_result))
    # Turn 4: the final answer, with an empty action list  [§2.1]
    turns.append(("gpt", serialize_turn(*answer)))
    return turns

# Serving loop for one user turn  [§2.1]
# Parsing and default tool arguments
# [code: llava/serve/gradio_web_server_llava_plus.py#L498-L533]
def serve_turn(model, history, image):
    # Generate the raw assistant text  [§2.1]
    output = model.generate(history)
    # Parse the three fields; actions is JSON  [code: ...#L499-L508]
    thoughts, actions, value = parse_fields(output)
    if actions is None or len(actions) == 0:
        return value
    # The code accepts exactly one tool call per turn  [code: ...#L518-L520]
    tool = actions[0]
    # Image comes from chat state, not the model  [code: ...#L522-L525]
    params = {"image": image, "box_threshold": 0.3, "text_threshold": 0.25}
    # Model's arguments override the defaults  [code: ...#L528-L533]
    params.update(tool["API_params"])
    result = run_tool(tool["API_name"], params)
    # Feed the result back and generate the answer  [§2.1] [eq. 2]
    history = history + [skill_result_turn(result)]
    return model.generate(history).value

Hyperparameters

The paper body does not state training hyperparameters. Values below come from the official 7B training script, which the card records as code-stated, not paper-stated.

Results

Author-reported. The paper evaluates on LLaVA-Bench (COCO: 30 images and 90 questions; In-the-Wild: 24 images and 60 questions) §4.1, SEED-Bench (19K multiple-choice questions) §4.1, a new 44-sample LLaVA-Bench (Tools) §4.1, MM-Vet (200 images, 218 questions) §4.2, and the VisIT-Bench Elo leaderboard as of 27 Sept 2023 §4.2, tab. 7. The caption and false-positive comparisons are in the appendix app. C.

LLaVA-Bench tab. 4. Relative scores, All column.

SEED-Bench tab. 4. Average accuracy: LLaVA 44.45, LLaVA (Tools in Test) 51.54, LLaVA-Plus (All Tools) 55.01, LLaVA-Plus (Fly) 53.25.

LLaVA-Bench (Tools) tab. 5. Scores by capability.

MM-Vet tab. 6. Total score. Columns: Rec, OCR, Knowledge, Generation, Spatial, Math.

VisIT-Bench Elo tab. 7. LLaVA-Plus (13B): Elo 1203, 678 matches, win rate 35.07% against the reference. LLaVA (13B): 1095, 5420 matches, 18.53%. Human reference: 1382. The paper reports this as the leaderboard state on 27 Sept 2023.

Caption (COCO, BLIP-2 comparison) app. C. LLaVA-Plus-7B: CLIP score 0.815, CIDEr 31.0. BLIP-2: CLIP score 0.788, CIDEr 123.5. LLaVA-7B: CLIP score 0.785, CIDEr 0.0.

Grounding DINO false positives on the first 100 COCO val images app. C. Grounding DINO: 90 false-positive instances in 41 images. LLaVA-Plus-7B: 20 instances in 12 images.

Paper vs code

Checked against the official repository, LLaVA-VL/LLaVA-Plus-Codebase, at commit a9f9d6f, linked from the paper's project page. The repository is Apache-2.0 LICENSE.

• Training hyperparameters. The paper gives none. The code gives 3 epochs, learning rate 2e-5, cosine schedule and related values training_llava_plus_v1.3_7b.sh. See [Hyperparameters].
• Training data. The paper says LLaVA-158K plus tool data, and 81K understanding samples for quantitative runs §2.3. The code trains on llava-150k-tool-aug.json plus llava-plus-v1-117k-tool-merge.json training_llava_plus_v1.3_7b.sh. The 117k count comes from the file name and was not checked against the file contents. The script therefore does not reproduce the 81K quantitative setting.
• Output format. The paper names three fields. The code writes them with literal tags, "thoughts🤔", "actions🚀" and "value👉", one line each, with the action list as JSON mm_utils.py. An implementation must emit these tags to match the parser gradio_web_server_llava_plus.py.
• Single quotes. The parser retries JSON with single quotes replaced by double quotes gradio_web_server_llava_plus.py. The paper does not mention this.
• Tool calls per turn. The paper describes composed skills §2.2.2 tab. 2. The serving code asserts exactly one tool per turn gradio_web_server_llava_plus.py. Composition therefore happens across turns at serving time.
• Default tool arguments. The code adds box threshold 0.3 and text threshold 0.25 to every tool call the model does not set gradio_web_server_llava_plus.py. The paper does not mention these.
• Loss masking. The paper says only the assistant spans get loss §2.1. The v1 preprocessing masks the human turns, which includes the skill result train.py. This is consistent with the paper.
• Base model and vision encoder. The paper body does not name the base LLM or the vision encoder. The code uses Vicuna-7B v1.3 and CLIP ViT-L/14 with feature layer -2 training_llava_plus_v1.3_7b.sh. The paper's stage-1 checkpoint text is commented out in the source, so it is not published.

Pitfalls

• "All Tools" is preprocessing, not planning. Tool outputs for the image are inserted as symbolic text in both training and evaluation §2.3. That variant scores highest on the benchmarks, but it does not test tool selection. The tool-selection evidence is the Fly variant and the no-thoughts ablation tab. 4.
• Mixed sources in the MM-Vet table. The authors re-ran LLaVA-7B (24.1) but took the LLaVA-13B row from the MM-Vet paper (32.5) tab. 6. The 13B gap therefore compares different runs. The 7B comparison is like-for-like.
• Thoughts matter, per the authors' ablation. Removing thoughts drops the Fly variant from 59.1 to 48.7 on In-the-Wild tab. 4.
• Benchmark is the authors' own. LLaVA-Bench (Tools) has 44 samples, is not a public benchmark in the paper, and the judging method is not stated §4.1.
• Data and tool licences are separate. The card's licence (CC BY 4.0) covers the paper. The training data comes from COCO, HierText, InfoSeek, JourneyDB and the Instruct-Pix2Pix dataset, and the labels come from GPT-4 and ChatGPT §2.2, tab. 2. The tools include Grounding DINO, SAM, Semantic-SAM, SEEM, OpenSeeD, BLIP-2, RAM, EasyOCR, Stable Diffusion, InstructPix2Pix, ControlNet and CLIP retrieval. The base model is Vicuna in the code. Check each licence before commercial use. This card does not check them.
• Generation prompts are examples only. The paper shows example prompts, and the preset thoughts and value responses are not listed in full §2.2, app. A.
• Hallucination and tool conflicts remain. The authors list both as open problems §5. Grounding DINO's false positives drop from 90 to 20 instances on 100 COCO images, which is an author-run test app. C.
• Cost. Serving the 7B model and all tools needs one 80 GB GPU §2.3. A tool call adds a second generation pass and the tool's own latency. The paper does not measure latency, so this is inferred from the two-turn design §2.1.
• Leaderboard snapshot. The VisIT-Bench Elo values are from 27 Sept 2023 tab. 7 and will have changed.

Lineage

• Builds on LLaVA (visual instruction tuning, cited in the paper as liu2023visual). No card in this wiki yet.
• Compared against: tool chaining (VisProg, ViperGPT, Visual ChatGPT, MM-REACT) and tool-tuned models (GPT4Tools, Gorilla) §1, §3. No cards in this wiki.
• Superseded by: not known from the paper or the repository.