Gorilla: Large Language Model Connected with Massive APIs

In plain words

Language models are good at writing code but often get the details of an external tool wrong. Asked to call a machine-learning library or a web API, they tend to invent function names, model repositories or arguments that do not exist. Gorilla is a LLaMA-7B model trained further on pairs of plain-English requests and the correct API call, built from public model hubs. It can also be trained with the relevant API documentation pasted into its prompt, so that it learns to read the documentation rather than memorise it. That makes it more useful when the documentation changes after training. The authors say it writes fewer invented calls than GPT-4 on their benchmark, which uses only those hub APIs.

TL;DR

Gorilla fine-tunes LLaMA-7B on about 1,645 hub APIs, each paired with ten synthetic user requests, so it outputs one API call per request. It is evaluated by parsing the output and matching it against the reference APIs. Zero-shot, it beats GPT-4 on all three hubs and hallucinates less. Results are author-reported on the authors' own benchmark. The retriever-in-prompt variant depends on a good retriever and is the weak point of the design.

Problem

Tool use needs a model to pick one API from a large, overlapping and changing set, which cannot all fit in one prompt §1. The authors argue that prior tool-use work used a small, well-documented API set that fits in the prompt §1. Their benchmark is built to be harder: many similar ML APIs with nuanced differences §1 §3.1.

Without help, zero-shot prompting of GPT-4 and GPT-3.5 produces hallucinated calls, such as loading a model from an arbitrary repository name [§4, paragraph "Hallucination with LLM"]. The paper's definition: a hallucination is a call that matches no API in the database, while an error is a call to a real but wrong API §3.3. A retriever can supply current documentation at answer time, but the paper finds that an imperfect retriever at test time can lower accuracy §4.

Method

Overview. Gorilla is a LLaMA-7B model, instruction-tuned on synthetic (instruction, API) pairs §3.2. The pipeline has three parts: build the APIBench dataset from model hubs §3.1; fine-tune, with or without a retrieved document in the prompt §3.2; and evaluate by AST sub-tree matching §3.3.

Dataset collection §3.1. The authors collected model cards from HuggingFace (the top 20 models per task category), TorchHub and TensorFlow Hub. Each card was turned into a JSON record that covers the task domain, the framework, what the API does, the call and its arguments, the software requirements, example code, reported performance and a short description §3.1 app. Dataset Details. Each record gets ten synthetic user requests. GPT-4 writes the requests, given three in-context examples out of six per hub and the API documentation. The requests must not name the API. The six hand-written examples per hub (18 in total) are the only manually written data §3.1.

Fine-tuning §3.2. Each (instruction, API) pair becomes a one-round user-agent chat. Gorilla is then trained by standard instruction fine-tuning of the base LLaMA-7B §3.2. For the retriever variant, the user turn also includes the reference documentation, introduced by the phrase "Use this API documentation for reference:" §3.2 §4. The paper trains both a model with and without the retriever §3.2.

Inference §3.2 §4. In zero-shot mode the request goes in unchanged. In retrieval mode a retriever (BM25 or a GPT-based index) returns the single most relevant API document, which is appended to the request. No other prompt tuning is used §3.2. Oracle retrieval, which returns the correct document, is an upper bound and is not a deployable retriever §4.

# Answer one request, zero-shot or with a retriever  [§3.2] [§4]
# Input:  request (str), api_docs (list of dict), retriever (or None)
# Output: one API call (str), no further prompt tuning

def answer(request, api_docs, retriever=None):
    prompt = request
    if retriever is not None:
        # Retrieve the single most relevant document (top-1)  [§4]
        doc = retriever.top_1(request, api_docs)
        # Append the reference to the user prompt, nothing else  [§3.2]
        prompt = (
            prompt
            + "\nUse this API documentation for reference: "
            + json_text(doc)
        )
    return model.generate(prompt)

# Build one training example per API  [§3.1] [§3.2]
# Input:  api_doc (dict), instructions (list of str), with_retriever (bool)
# Output: list of user-agent chat examples

def build_examples(api_doc, instructions, with_retriever):
    examples = []
    for instruction in instructions:
        # User turn: the request, plus the reference doc if retrieval-aware  [§3.2]
        user_turn = instruction
        if with_retriever:
            user_turn = (
                user_turn
                + "\nUse this API documentation for reference: "
                + json_text(api_doc)
            )
        # Agent turn: the API call the model must produce  [§3.2] [§4.1]
        examples.append({"user": user_turn, "agent": api_doc["api_call"]})
    return examples

Evaluation by AST sub-tree matching §3.3 fig. 4. Test-time output is parsed into a syntax tree. The paper checks whether the generated call contains a sub-tree that matches a reference API in the database. Only the arguments the database marks as required are compared. Optional arguments such as pretrained=True are ignored §3.3 fig. 4.

• Torch Hub: the API torch.hub.load, with arguments repo_or_dir and model app. Gorilla Details.
• TensorFlow Hub: hub.KerasLayer and hub.load, with argument handle app. Gorilla Details.
• HuggingFace: pretrained_model_name_or_path, except for pipeline, which picks a model when task is given app. Gorilla Details.

# Decide which reference API a generated call matches  [§3.3] [fig. 4]
# Input:  call (str), reference_apis (list of dict with required args)
# Output: index of the matched reference, or -1 for hallucination

def match_reference(call, reference_apis):
    # Parse the generated call into a syntax tree  [§3.3]
    call_tree = parse_python(call)
    # Collect every sub-tree of the call  [§3.3]
    call_subtrees = all_subtrees(call_tree)
    for index, reference in enumerate(reference_apis):
        # Same API name, and every required argument appears in the call  [§4]
        if has_match(call_subtrees, reference):
            return index
    # No reference matches: the call is a hallucination  [§3.3]
    return -1

# Count a matched call as functionally correct when the domains agree  [§4]
# Input:  index (int), expected_domain (str), reference_apis (list of dict)
# Output: bool

def is_correct(index, expected_domain, reference_apis):
    # The matched reference's domain must equal the question's domain  [§4]
    return reference_apis[index]["domain"] == expected_domain

Figure references: fig. 1 shows example calls from GPT-4, Claude and Gorilla for one prompt. fig. 2 plots accuracy against hallucination for the four settings. fig. 3 is the training and inference system. fig. 5 is the accuracy grid with the GPT retriever. fig. 6 shows the test-time documentation change examples. No figure is reproduced on this page.

Hyperparameters

Training hyperparameters are from the paper only. The repository at code_commit has no training script (see Paper vs code).

Results

Author-reported, on the authors' APIBench test split §4. Accuracy is the share of requests whose call matches a reference API and has the expected domain. The baselines are zero-shot GPT-4 (gpt-4-0314), GPT-3.5-turbo (gpt-3.5-turbo-0301), Claude (claude-v1) and LLaMA-7B §4. No seeds, error bars or variance are reported.

Overall accuracy and hallucination rate, from tab. 1. The rows show only the zero-shot and retrieval baselines discussed in the text, not the full table:

Effect of training with the retriever, from tab. 2 (overall accuracy). "Trained without" is the model fine-tuned without documentation. "Trained with oracle" is the model fine-tuned with the correct document:

Constraint-aware calls, Torch Hub subset with a stated accuracy, "Accuracy const" row, from tab. 3 §4.3:

Test-time documentation changes are shown only as qualitative examples fig. 6 §4.2. The paper gives no accuracy numbers for that claim.

Paper vs code

Code commit 6ea57973c7a6097fd7c5915698c54c17c5b1b6c8 (HEAD of the ShishirPatil/gorilla clone, shallow). The repository has moved on since the paper: it now also holds later projects. Only the files read below were checked. The official repository is linked from the authors' project page, which the abstract names [paper abstract], and its owner is an author's account.

• Retrieval prompt wording. The paper's phrase is "Use this API documentation for reference:" §3.2 §4. The code appends "Here are some reference docs:" and then one line per document, "API i:" get_llm_responses_retriever.py.
• Extra instructions in the prompt. The code's prompt adds a format template and domain lists that the paper does not describe get_llm_responses_retriever.py. This is a difference in the evaluation prompt, not in the training data.
• GPT retriever embedding model. The paper says "GPT-Index" refers to text-davinci-003 §4. The code embeds documents with text-embedding-ada-002 gpt.py.
• Top-k. The paper uses top-1 §4. The script default is 1 get_llm_responses_retriever.py, matching the paper. The GPT retriever class defaults to 5, but the script overrides it gpt.py.
• Argument matching is substring-based. The paper matches required arguments at the AST level §3.3. The code checks each argument's text for a substring of the candidate's text ast_eval_th.py, and it selects arguments whose text contains "repo_or_dir" or "model" ast_eval_th.py.
• Fallback in the matcher. If no candidate sub-tree has the reference's API name, the loop leaves the last candidate in place rather than failing ast_eval_th.py.
• Training code. The paper gives training hyperparameters tab. 4. The commit has no training script or config, so these cannot be checked against code.
• Claude. The code calls claude-v1 with at most 2048 tokens to sample get_llm_responses_retriever.py, matching §4.
• Hallucination and error. The code labels a call a hallucination when no reference matches ast_eval_th.py, and a correct call when the matched reference's domain equals the expected domain ast_eval_th.py. This matches §3.3 and §4.

Searched for the code: the abstract links the project page; the project page links github.com/ShishirPatil/gorilla; a GitHub search under the owner found the same repository, licensed Apache-2.0 [LICENSE file at the clone root].

Pitfalls

• Training target is ambiguous. Section 3.2 says the training data is (instruction, API) pairs. Section 4.1 says the retrieval-trained model uses "the example output generated by GPT-4" as its target. The card cannot tell which target was used for the main model.
• The retrieval variant hurts without retrieval. A model trained with the oracle document scores 0 overall zero-shot in every hub tab. 2. TorchHub hallucination for that row is 100 tab. 2. A deployment without a retriever therefore needs the zero-shot model, not the retrieval-trained one.
• Test-time retrievers can lose a lot. For the model trained without documentation, BM25 at test time drops overall accuracy on every hub (37.63, 11.28, 34.30 against zero-shot 59.13, 71.68, 83.79) tab. 2. GPT-Index is slightly higher than zero-shot on Torch Hub (60.21 vs 59.13) and lower on HuggingFace and TensorFlow Hub tab. 2.
• Not all rows compare like with like. For HuggingFace, the paper checks only the domain for models other than Gorilla §4.1. The HuggingFace column therefore mixes two metrics and is not a like-for-like comparison.
• Dataset counts do not add up. The text gives 925 HuggingFace models as the top 20 per domain, but 36 domains times 20 is 720 §3.1. It gives 626 TensorFlow Hub models after filtering and 95 Torch Hub, for 1,646 in total; it states 1,645 §3.1. The appendix gives 696 TensorFlow Hub APIs app. Dataset Details. The limitations section says "over 11,000 instruction-API pairs" §6, while 1,645 × 10 is 16,450 §3.1.
• Some text figures do not match the tables.
  • "83% better than LLaMA" §4: LLaMA scores 0 on Torch Hub, and the gap in tab. 1 is 59 to 84 points, not a percentage.
  • "23.46% better in HuggingFace" §4: tab. 2 gives 91.26 − 45.58 = 45.68 points against the oracle column and 91.26 − 71.68 = 19.58 against zero-shot. Neither gives 23.46.
  • The "29.20%" and "52.27%" drops match the TensorFlow Hub rows of tab. 2 (94.16 − 64.96 and 94.16 − 41.89), but the text does not name the hub.
  • "Matches GPT-3.5 with retrieval" §4.3: in the "Accuracy const" row of tab. 3, Gorilla is 30.28 and 26.76 against 33.80 and 33.09.
• The 20.43-point claim is in points. "20.43% better than GPT-4" is the difference in tab. 1 (59.13 − 38.70), not a relative gain.
• Unclear checkpoints in tab. 1. The text does not say whether the BM25 and GPT-Index rows of Gorilla use the model trained with or without the retriever. tab. 2 suggests the model trained without.
• Oracle rows are an upper bound. Oracle retrieval returns the correct document, so those rows cannot be reached in practice §4.
• Argument matching is loose. Required arguments are matched as substrings, so any call whose text contains the expected string passes ast_eval_th.py. The paper does not discuss this. I did not measure how often it changes a score.
• Costs not stated. No inference latency, training time, or cost per request is given. The GPT-generated instructions cost is not stated.

Lineage

• Builds on: LLaMA-7B, the base model §3.2; self-instruct, for generating the synthetic instructions §3.1; Toolformer and related tool-use work §2.
• Compared against: GPT-4 (gpt-4-0314), GPT-3.5-turbo (gpt-3.5-turbo-0301), Claude (claude-v1), LLaMA-7B; with BM25 and a GPT-based index as retrievers §4.
• Superseded by: not known from content already in this wiki.