Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

In plain words

Lawyers and ordinary people are starting to ask chatbots legal questions, such as whether a court case exists, which court decided it, or what it held. This paper asks a plain question: do these chatbots actually know the law, or do they make things up?

The authors took about five thousand random US federal court cases from each level of the court system and asked four popular chatbots fourteen kinds of questions about them, from easy ones (does this case exist?) to hard ones (what was its central holding?). For the easy questions, the authors could check the answer against court metadata. For the hard ones, they asked the chatbot the same question twice and checked whether its two answers contradicted each other.

The chatbots often gave confident, wrong answers. Mistakes were more common for lower courts, less famous cases and newer or very old cases. When a question rested on a false legal premise, the chatbots often went along with it. The chatbots' own confidence was also a poor guide to when they were wrong. The paper's conclusion is that these tools should not be used for legal work without careful human checking.

TL;DR

The paper defines a legal hallucination as a response that conflicts with the actual state of the law, and it measures how often four LLMs do this on 14 tasks over US Supreme Court, Courts of Appeals and District Court cases §1 §2.2 §3. Reference-based tasks (9 of 14) compare a greedy answer at temperature 0 with court metadata §4.2 eq. 1. Reference-free tasks (5 of 14) sample two answers at temperature 1 and count a GPT-4 contradiction label as a hallucination; this gives only a lower bound §4.3 eq. 2. Author-reported headline: GPT-4 hallucinates in at least 58% of the time and Llama 2 in up to 88% on direct questions about random federal cases §1. Pooled expected calibration error: PaLM 2 0.057, GPT-3.5 0.099, GPT-4 0.190, Llama 2 0.421 §5.3. Relevant for evaluation design and for anyone building a legal QA system on a general LLM. The code is official (reglab, authors' lab) but has no license file.

Problem

Legal research depends on exact knowledge of authorities: which court decided a case, whether it is still good law, and what it held. The authors argue that earlier work measured legal reasoning on exam-style benchmarks but not whether a model stores accurate legal knowledge, and that anecdotal reports of fabricated citations were not systematic §1. They also note that open-domain hallucination (a response that is not factual about the world) is the kind that matters most for legal advice §2.2. Their argument is that such errors could harm pro se litigants most, since those users benefit most from cheap LLM help but have the least means to check it §1 §6.

Method

The paper is a measurement study. Its protocol has four parts.

1. Typology. Hallucinations are split into closed-domain (inconsistent with the prompt), open-domain against the training corpus, and open-domain against the facts of the world. Only the last type is measured §2.2.
2. Data. For each court level, 5,000 cases are drawn by stratified random sampling: year strata for the Supreme Court, circuit-year strata for the Courts of Appeals and state-year strata for the District Courts. Metadata come from the Caselaw Access Project, the Supreme Court Database, the Appeals Courts Database, the Library of Congress and Shepard's §4.1.
3. Tasks. Fourteen tasks in three complexity levels: low (existence, court, citation, author), moderate (disposition, quotation, authority, overruling year) and high (doctrinal agreement, factual background, procedural posture, subsequent history, core legal question, central holding) §3.1 §3.2 §3.3. Disposition is not asked for District Courts §3.2; doctrinal agreement is limited to Supreme Court pairs §3.3. The first nine tasks are reference-based and the last five reference-free §4.2 §4.3.
4. Robustness and calibration. Two false-premise variants (a dissent that never happened; an overruling of a case that was never overruled) §5.2, and an expected calibration error per task using 10 temperature-1 samples compared with the greedy answer app. D.1 tab. 6.

Reference-based hallucination rate:

\pi = \hat{\pi} = \frac{1}{N} \sum \mathbbm{1}[\text{response} \neq \text{response}^\prime] eq. 1 §4.2

Reference-free lower bound, where response^(1) and response^(2) are two independent temperature-1 answers:

\pi \geq \hat{\pi} = \frac{1}{N} \sum \mathbbm{1}[\text{response}^{(1)} \neq \text{response}^{(2)}] eq. 2 §4.3

Reference-based implementation:

# Reference-based hallucination rate  [§4.2] [eq. 1]
# Input: prompts, ground-truth answers, one LLM
# Output: share of responses that are hallucinations

def reference_based_rate(prompts, truths, llm):
    # Greedy answer: temperature 0  [§4.2] [code: api.py#L146]
    responses = [llm.greedy(prompt) for prompt in prompts]
    # Abstentions are counted as non-hallucinations  [§4.2]
    hallucinated = [
        response != truth and not is_abstention(response)
        for response, truth in zip(responses, truths)
    ]
    return mean(hallucinated)

Reference-free implementation:

# Reference-free lower bound  [§4.3] [eq. 2]
# Input: prompts, one LLM, a GPT-4 contradiction judge
# Output: lower bound on the hallucination rate

def reference_free_bound(prompts, llm, judge):
    contradicted = []
    for prompt in prompts:
        # Two independent answers at temperature 1  [§4.3] [code: models.py#L128]
        first = llm.sample(prompt, temperature=1)
        second = llm.sample(prompt, temperature=1)
        # Any contradiction proves at least one answer is wrong  [§4.3]
        contradicted.append(judge.contradicts(first, second))
    return mean(contradicted)

Expected calibration error for one task:

# Expected calibration error, one task  [app. D.1]
# Input: per query, greedy answer, 10 temperature-1 samples, ground truth
# Output: scalar ECE over 10 equal-size bins

def expected_calibration_error(queries, bins=10):
    # Confidence: share of samples equal to the greedy answer  [app. D.1]
    confidences = [
        mean(sample == query.greedy for sample in query.samples)
        for query in queries
    ]
    # Outcome: greedy answer is not a hallucination  [app. D.1] [§5.3]
    correct = [query.greedy == query.truth for query in queries]
    # Plug-in estimate with a bias correction, 10 equal-size bins  [app. D.1]
    return binned_gap(confidences, correct, bins)

Zero-shot and three-shot prompts are used for every query §4.4. Few-shot results are the focus of the discussion §5.1. The zero-shot and few-shot templates are in the appendix app. B; the paper gives this template for the existence task asks whether the named case, with its citation and year, is real, and asks for a bare yes or no app. B.

Hyperparameters

Results

Author-reported. Setup: four LLMs, three court levels, 5,000 sampled cases per level (100 for the high-complexity tables), zero-shot and three-shot prompts. Values are hallucination rates (or ECE in Table 6). Standard errors from the paper are not reproduced; see the source tables. A dash means the paper did not run the task for that court level.

Table 2. Low-complexity tasks, hallucination rate tab. 2:

SC = Supreme Court (n=5,000), CoA = Courts of Appeals (n=5,000), D = District Courts (n=5,000).

Table 3. Moderate-complexity tasks, hallucination rate tab. 3:

Table 4. High-complexity tasks, hallucination rate, n=100 per cell tab. 4. Doctrinal agreement is reference-based; all other rows are lower bounds from the reference-free method. Only doctrinal agreement has few-shot rows.

Doctrinal agreement footnote: SC 1796–2005, n=5,000 tab. 4. Other high-complexity rows: n=100 per cell.

Table 5. Contra-factual premises, zero-shot only, n=1,000 per level tab. 5:

Table 6. Expected calibration error by task, n=100 per cell tab. 6. Each confidence is estimated from 10 temperature-1 samples (table note).

Headline and section results, author-reported:

• Overall: GPT-4 is lowest at 58% and Llama 2 highest at 88% on direct reference-based questions §1 fig. 1.
• Reference-free tasks: factual background and procedural posture hallucinate at least 49% of the time; core legal question at least 59%; central holding at least 63% §5.1.1. The paper states these are lower bounds.
• Doctrinal agreement hovers near 0.5 for the binary label, with no improvement over chance and sometimes worse §5.1.1 tab. 4.
• Court level: hallucination is lowest for the Supreme Court and highest for district courts §5.1.2 fig. 3.
• Circuit: best in the Ninth, Second, Third and First Circuits; worst in the central circuits; the paper reports the D.C. Circuit as worst, against expectation §5.1.3 fig. 4. The full-period version is in app. E.2.
• Prominence: hallucination falls with citation-network PageRank, with a sharp change near the 90th percentile for GPT-4, GPT-3.5 and PaLM 2 but not Llama 2 §5.1.4 fig. 5.
• Year: hallucination is highest for the oldest and newest Supreme Court cases and lowest for the Warren Court (1953–1969) §5.1.5 fig. 6.
• Inductive bias: a few justices are over-represented in the Author task; Llama 2 favours Justice Story and PaLM 2 favours Justice McLean §5.1.6 fig. 7.
• Contra-factual premises: GPT-3.5, PaLM 2 and GPT-4 often accept the false premise, PaLM 2 almost always; Llama 2 usually rejects it, sometimes by claiming the case or justice does not exist §5.2 tab. 5.
• Calibration: pooled ECE of PaLM 2 0.057, GPT-3.5 0.099, GPT-4 0.190 and Llama 2 0.421 §5.3. GPT-4 is the least hallucination-prone but less calibrated than PaLM 2 or GPT-3.5. Calibration errors are positive in every case, meaning the models are overconfident §5.3.
• Post-hoc Platt scaling lowers pooled ECE to 0.033 (GPT-4), 0.051 (GPT-3.5), 0.049 (PaLM 2) and 0.035 (Llama 2), but the scaled curves stay poor in the 0.2–0.4 range app. D.2.
• Existence and fake cases: GPT-3.5 and PaLM 2 assert existence for fake cases too; GPT-4 and Llama 2 mostly deny existence app. E.1.
• Abstentions: rare for most tasks; common in zero-shot quotation app. F.

Paper vs code

Official code: https://github.com/reglab/legal_hallucinations at commit 620734a1ccb84efc6827635ac76a2fb6504b2f64 (owner reglab, the Stanford lab of author Daniel E. Ho). The repository has no LICENSE file, so no code is quoted verbatim. Data and replication materials are the Harvard Dataverse release and a Hugging Face dataset named in the paper's title footnote.

Differences found by reading the paper against the code:

• Reference-free sampling temperature. The paper says temperature 1 §4.3 fn. 2. The code uses 0.999 for reference-free tasks tasks_zero_resource.py. Reference-based tasks use 1 tasks_scotus.py.
• ECE sample count for PaLM 2. The paper gives no K in the text and says 10 in the table note tab. 6. The code uses 8 samples for PaLM 2 models.py models.py and 10 for the other models models.py.
• Contradiction judge shot count. The paper does not state how many examples its judge prompt uses §4.3 app. C. The code uses 3 tasks_zero_resource.py; the function's default is 5 correctness_checks.py.
• Contradiction verdict. The code reads the judge's verdict from the last 40 characters of its reply, matching "the answer is yes" or "the answer is no" correctness_checks.py. The paper says only that the judge labels pairs as contradictory or not §4.3.
• Llama 2 backend. The paper names Llama-2-13b-chat-hf §4.4. The code can call a local Manifest server or Together AI; the active backend is chosen in an if-branch tasks_zero_resource.py. The paper does not say which was used.
• Model snapshots. The code names gpt-4-1106-preview, gpt-3.5-turbo-0613 and text-bison-001 api.py api.py api.py, matching the paper.
• Citation task token limit. Citation tasks use max_tokens 30 tasks_scotus.py tasks_scotus.py. The paper states no generation limit.
• Binning. The paper says ECE uses 10 equal-size bins app. D.1. The code adds random noise before quantile binning plot.py, which breaks ties among equal confidences.
• Case subsample. The code draws 100 cases with random.sample after seeding with 47 tasks_zero_resource.py settings.py. The paper says 100 per level without a seed.
• Data location. settings.py points to an absolute path on the authors' computer settings.py. The repository's data/samples holds only a README README.md. Re-running needs the case sources (Caselaw Access Project, Supreme Court Database, Shepard's, and others) obtained separately §4.1 app. A.

Pitfalls

• Reference-free numbers are lower bounds. A contradiction proves a hallucination, but two consistent answers can still be wrong §4.3. Every reference-free figure in the paper is a floor, and the paper says so in the table notes tab. 4. Do not report them as rates.
• Doctrinal agreement is reference-based. It sits in the high-complexity table but uses Shepard's labels, so it is an exact rate, not a bound §3.3 §4.2 tab. 4 note.
• High-complexity tasks have no few-shot rows, except doctrinal agreement. The text says the paper focuses on three-shot results §5.1 app. E.4. For the other high-complexity tasks, zero-shot is the only condition reported.
• Abstentions count as correct. An abstention is not a hallucination §4.2. Zero-shot quotation is where most abstentions happen, so its zero-shot rates understate hallucination app. F app. E.4.
• The judge decides the reference-free result. The paper validates GPT-4 against two human coders on 100 queries, with κ of 0.87 and 0.77 against the coders and 0.79 between coders app. C. The validation sample is small, and the judge's own temperature and few-shot count are not in the text.
• Existence is inflated by a yes-bias. Every prompted case is real, so "yes" is correct §3.1. The fake-case test shows that GPT-3.5 and PaLM 2 also say yes to fake cases, which the real-case result hides app. E.1.
• Confidence is estimated, not observed. For GPT-4, GPT-3.5 and PaLM 2 the paper cannot read token probabilities, so it estimates confidence from 10 temperature-1 samples against the greedy answer app. D.1. Use the ECE as a relative measure.
• Contra-factual tests are not symmetric. Refusing to answer a false premise counts as correct, and Llama 2's denials of existence were also scored as rejections §5.2. A low contra-factual rate can come from over-refusal.
• Snapshots age. The results depend on dated model snapshots §4.4. Newer models may behave differently; that is an inference, not a claim in the paper.
• Figure and table numbering. The text cites Table 2 for existence and Figure 4 for circuits app. E. Other numbers here follow source order.
• Cost. The full run is more than 800,000 queries, 200,000+ per LLM §4.4. Budget for repeated sampling and for the GPT-4 judge calls.

Lineage

• Builds on: the self-consistency contradiction check of Mündler et al. (2023) §4.3, which the paper adapts to contradiction rather than consistency.
• Compared against: the paper's own four LLMs. It cites Mündler et al. for a GPT-3.5 general-QA self-contradiction rate of 14.3% as context §5.1.1.
• Superseded by: not stated.