Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

In plain words

Language models are good at reading a question and deciding what to do, but they cannot see a picture, look up today's news, or do exact arithmetic on a table by themselves. Chameleon adds a menu of helper tools (a captioner for images, a text reader, a web search, table-cutting helpers, Python code, and simple rules) and asks the language model to write a short plan that names which helpers to use and in what order.

The plan is a list of tool names. Each tool passes its result to the next one through a small shared memory, and the last tool gives the answer. Nothing is trained: the language model learns the job from a written description of each tool and a few example plans in the prompt.

The authors test this on two benchmarks: science questions that come with images, and math word problems over tables. The main point is that the plan changes from question to question instead of following one fixed pipeline.

TL;DR

• Chameleon is a plug-and-play framework: an LLM planner writes a sequence of tools, and the tools run in that order with cached context §3.
• Thirteen module types are defined, mixing LLM prompts, a captioner, OCR, web search, Python program tools and rule-based table helpers §4.1.
• No training is involved; the planner and the LLM-based modules use few-shot prompts §3 app. A.2.
• Headline author-reported results with GPT-4: 86.54% on ScienceQA tab. 3 and 98.78% on TabMWP tab. 4.
• Code exists and reproduces the pipeline, but several components are precomputed or differ from the paper; see Paper vs code.

Problem

Large language models cannot reliably access current information, use external tools, or do precise mathematical and logical work §1. Earlier tool-augmented systems either use only a few tools, rely on domain-specific tools, or need task-specific prompts or supervision to decide which tool to call §1 §2. The paper asks how an LLM can compose heterogeneous tools for new queries without training §1.

Method

Chameleon has two parts: a module inventory of tools and an LLM-based planner §3. Given a query x0, the planner P receives a task instruction, the descriptions of the modules in the inventory with their constraints, and a few demonstrations, and returns a plan of T module names §3 eq. 1. Modules are bound to the plan by string matching §3.

Each module M^t reads the current input x^(t-1) and cache c^(t-1), and returns output y^t §3 eq. 2. The cache holds things such as image descriptions, retrieved knowledge or generated programs. A module-specific update_input function changes the query (for example, a row lookup replaces the table), and an update_cache function stores new information for later modules §3 [app. A.2]. The output of the last module is the response §3 eq. 5.

p \leftarrow \mathcal{P}(x_0; \mathcal{I}, \mathcal{M}, \mathcal{G}, \mathcal{D}) eq. 1

y^t \leftarrow M^t(x^{t-1}; c^{t-1}) eq. 2

x^t \leftarrow \texttt{update\_input}(x^{t-1}, y^t) eq. 3

c^t \leftarrow \texttt{update\_cache}(c^{t-1}, y^t) eq. 4

r = y^T \leftarrow M^T(x^{T-1}; c^{T-1}) eq. 5

Module inventory

The paper describes these thirteen modules §4.1: Knowledge Retrieval (an LLM generates background knowledge), Bing Search (web snippets), Query Generator (an LLM writes the search query), Image Captioner, Text Detector (OCR with coordinates), Row Lookup, Column Lookup (both simplify a table to the relevant rows or columns), Table Verbalizer (turns a small table into text), Program Generator, Program Verifier, Program Executor (Python), Solution Generator (chain-of-thought answer) and Answer Generator (extracts the answer from text). The inventory figure is tab. 2 (rendered as an image in the paper) and the per-task subsets are tab. 6.

For ScienceQA the planner may choose from Query Generator, Bing Search, Image Captioner, Text Detector, Knowledge Retrieval, Solution Generator and Answer Generator tab. 8. For TabMWP it may choose from Row Lookup, Column Lookup, Table Verbalizer, Knowledge Retrieval, Program Generator, Program Verifier, Program Executor, Solution Generator and Answer Generator tab. 9 §4.3.

Plan validity and defaults

ScienceQA: a plan is invalid unless its last two modules are Solution Generator and then Answer Generator. Invalid plans default to that two-step sequence, which the paper calls the chain-of-thought baseline §4.2.

# ScienceQA plan check  [§4.2]
def is_valid_plan_scienceqa(modules):
    # The last two modules must be Solution then Answer generator  [§4.2]
    return modules[-2:] == ["solution_generator", "answer_generator"]

TabMWP: a valid plan must include Answer Generator, and Program Generator must come before Program Verifier and Program Executor. Non-compliant plans default to Program Generator, Program Verifier, Program Executor, Answer Generator, the program-of-thought baseline with verification added §4.3.

When the planner places Program Verifier directly after Program Generator, the paper's program merges them into one step: the program is regenerated until it passes the verification check model.py.

# Program generator with verification, merged  [§4.3]
# [code: run_tabmwp/model.py#L300-L327]
def generate_verified_program(prompt, patience):
    # Program is None until a generation passes the check
    program = None
    for attempt in range(patience):
        # Temperature 0 first, then 0.1 on retries
        # [code: run_tabmwp/model.py#L313-L317]
        temperature = 0.0 if attempt == 0 else 0.1
        program = llm(prompt, temperature)
        if program_is_valid(program):
            break
    return program

Run loop

# Chameleon run loop  [§3] [eq. 1]-[eq. 5]
def answer(query, inventory, planner_prompt, default_plan):
    # Planner LLM writes a module-name sequence  [eq. 1]
    plan_text = planner_llm(planner_prompt, query, inventory)
    # Parse the plan; use the task default if it is invalid  [§4.2] [§4.3]
    modules = parse_plan(plan_text)
    if not is_valid_plan(modules):
        modules = default_plan
    # Cache holds captions, OCR text, knowledge, programs  [§3]
    cache = {}
    # The input starts as the raw query  [§3]
    module_input = query
    for module in modules:
        # Run the module on the current input and cache  [eq. 2]
        output = module(module_input, cache)
        # Update the input and cache for the next module  [eq. 3] [eq. 4]
        module_input = update_input(module, module_input, output)
        cache = update_cache(module, cache, output)
    # The last module's output is the response  [eq. 5]
    return output

Module details, by task

ScienceQA §4.2 app. A.2: Query Generator output is a search query of at most 64 tokens. Bing Search returns the top three responses. Image Captioner uses a captioning model; Text Detector uses EasyOCR. Knowledge Retrieval and Solution Generator use 3 and 2 demonstrations respectively. The Answer Generator pulls the answer phrase from the solution and picks the most similar option.

TabMWP §4.3 app. A.2: Row Lookup and Column Lookup are only enabled for tables above a size threshold (more than three rows and 18 cells; two or more columns and 18 or more cells, as written in the appendix). Knowledge Retrieval uses 5 demonstrations, Row Lookup 7, Column Lookup 6, Table Verbalizer 7 and Solution Generator 16. Row Lookup and Column Lookup output at most 256 tokens; Table Verbalizer, Knowledge Retrieval and Solution Generator output at most 512. The Program Generator output is capped at 256 tokens.

Hyperparameters

Unless stated otherwise, all LLM calls use temperature 0 app. A.2. "Not stated" means the paper and its appendix do not give the value.

Results

Author-reported. Setup: ScienceQA test set and TabMWP test set, with accuracy in percent; GPT-4 and ChatGPT as the LLM; no fine-tuning of the planner tab. 3 tab. 4. The paper also reports a human-performance row and fine-tuned baselines.

The text reports gains of 11.37 points over the best published few-shot ScienceQA result and 17.0 points over Codex PoT-SC on TabMWP §5.1.

Ablation (ChatGPT, 500 test examples §5.2): disabling Knowledge Retrieval costs 7.8 points on ScienceQA and 2.2 on TabMWP. On ScienceQA, disabling Bing Search costs 7.4, Text Detector 8.4 and Image Captioner 6.0. On TabMWP, disabling Program Generator costs 7.4 and Table Verbalizer 0.2 tab. 5.

Planner behaviour: GPT-4 generated 11 distinct programs on ScienceQA and 19 on TabMWP, against 14 and 28 for ChatGPT, with average program lengths of about 3.4 and 4.1 modules tab. 7 §5.2.

Paper vs code

Official code: the repository lupantech/chameleon-llm at the commit above, Apache-2.0 LICENSE. The paper names its project page (chameleon-llm.github.io) on the title page p. 1; the repository sits under the GitHub account of the first author, Pan Lu README.md. Code and results are not identical to the paper:

• Bing results. The paper says the search module returns the top three responses app. A.2. The code default is one result run.py, and the published run commands do not change it README.md.
• Row/column thresholds. The paper's appendix uses 18 cells app. A.2. The code defaults are 12 cells run.py run.py, and Row Lookup is skipped only when rows are at most 3 or cells are at most 12 model.py. The README's TabMWP commands pass 18 README.md. Column Lookup is skipped when columns are at most 2 or cells at most 12, so it needs more than two columns and more than 12 cells model.py.
• Program verifier. The paper describes a verifier that checks syntax and logic §4.1. In code, the standalone verifier only records a flag model.py. Its check is that the text contains ans = and that execution gives a non-empty result model.py. The retry loop that enforces validity is in the merged step model.py.
• Plan constraints on TabMWP. The paper says ordering constraints force the default program §4.3. The code's ordering checks compare the list of module names with ["program_verifier"], which is a list inside a list of strings, so these checks never fire model.py. Only the presence of Answer Generator is enforced model.py.
• Answer precision. The paper says the Answer Generator normalises numerical answers to two decimal places app. A.2. The answer generator does not round model.py. The comparison rounds both values to the precision of the reference string run.py utilities.py utilities.py.
• Captioner and OCR. The paper describes a live captioner and EasyOCR app. A.2. The ScienceQA code loads precomputed captions and OCR output from JSON files model.py; the live captioner is a TODO that returns an empty string when no cached caption exists model.py. The README says live calls are not yet available README.md. OCR is produced by a separate script generate_ocr.py.
• Captions in the prompt. Chameleon sets use_caption to false, so the question text never includes a caption. A caption reaches later prompts only as cached response text, and only when the Image Captioner module runs model.py model.py.
• Retries. The paper states temperature 0 app. A.2. The code raises the temperature to 0.1 on every retry of Solution Generator and Program Generator model.py model.py.
• Planner parsing. The paper says invalid plans default to the fallback sequence §4.2 §4.3. Code parses the plan with eval on the model's text model.py and runs each name as solver.<name> run.py. The ScienceQA check only looks at the last two modules, so an unknown module name earlier in the plan is not caught.
• Demonstration counts. Spot checks match the appendix: Knowledge Retrieval has 3 ScienceQA demonstrations and 5 TabMWP demonstrations; Row Lookup 7, Column Lookup 6 and Table Verbalizer 7 in the TabMWP prompt files. The remaining counts were not checked.
• Engines in the README. The ScienceQA README's ChatGPT CoT command passes --sg_engine gpt-4 README.md. The TabMWP README uses gpt-3.5-turbo for the same setting.

Pitfalls

• Model snapshots and dates. The paper names gpt-4 and gpt-3.5-turbo but does not give snapshot dates app. A.2. Results depend on the API version at run time, and the code's dependencies pin openai==0.27.0, which uses the legacy ChatCompletion call requirements.txt utilities.py.
• Search results drift. The Bing module returns live web snippets. The paper does not say when the search was run. The ScienceQA run without a Bing key loses a little accuracy README.md.
• Retry cost. Each retry is an extra LLM call. Solution Generator retries up to 5 times on ScienceQA and 20 on TabMWP, and the merged generator/verifier up to 20 times run.py run.py run.py. The paper does not state these limits.
• Scoring is lenient. A prediction is counted correct when both values match after rounding to the precision of the reference string utilities.py utilities.py. A reference with one decimal therefore accepts any prediction that rounds to it.
• Eval of model output. Plans are parsed with eval. Use a strict parser with a whitelist of module names before reusing this design.
• No re-planning. The planner writes the whole program up front, with no re-planning as modules run. The paper assumes the module list and descriptions fit in the context window app. B.
• Module sets are fixed per task. The planner can only choose from the subset listed for each task tab. 8 tab. 9. Adding a tool means editing the prompt, the demonstrations and the update functions §3 app. A.2.
• Precomputed inputs. The ScienceQA captions and OCR are fixed files in the repository. Changing the captioner means regenerating them.
• Ablation numbers come from a subset. They use 500 test examples with ChatGPT §5.2, not the full test set or GPT-4.
• Fine-tuned baselines. On ScienceQA, the fine-tuned MM-COT (Large) model scores higher overall than Chameleon with GPT-4 tab. 3. The paper's claim is about few-shot methods.
• Human rows. Human scores are taken as printed in tab. 3 and tab. 4; this card does not verify them.

Lineage

• Builds on: Chain-of-thought prompting for the Solution Generator and the chain-of-thought baseline §2 §4.2. The program-of-thought baseline (cited as chen2022program) is the TabMWP program branch's reference point §4.3; the paper gives no arXiv id for it.
• Compared against: Toolformer, ViperGPT, Visual ChatGPT, VisProg, HuggingGPT and AutoGPT, which the paper contrasts on tool count, supervision and domain specificity §1 tab. 1.
• Superseded by: not stated.