A Short Survey of Viewing Large Language Models in Legal Aspect

In plain words

Language models now draft, summarise and search legal material, but a fluent answer is not the same as an answer a lawyer can rely on. This survey argues that the thing to judge is the whole system around the model. A reliable legal answer has to point to a source that binds the relevant decision maker, that is still in force on the relevant date, that applies in the right jurisdiction, and that actually says what the answer claims. The authors call systems that meet this bar "authority-grounded".

The survey walks through the legal tasks, the techniques people use (adapting models to legal text, looking up documents at answer time, knowledge graphs, agents), how each is tested, and the rules and responsibilities around them. It counts the technical studies it reviewed to show where that evidence comes from. The framework is a proposed way to organise and compare this work, not a tested standard.

TL;DR

• A critical narrative survey of large language models in legal AI, not a systematic review, and it proposes no new model or training method.
• Core idea: a legal output is only relied on if its source is valid, applicable and supports the claim; relevance of retrieved text alone is not enough §1.1 §3.2.
• It organises the literature on three non-exclusive axes: legal task, system mechanism and assurance evidence §2.3 fig. 2.
• Quantitative snapshot: 327 technical studies from 2022 to 2025, screened at title-and-abstract level by one reviewer §2.1.
• Use it as a vocabulary and reporting checklist. It supplies no implementation, no validated thresholds and no benchmark results of its own.

Problem

Studies of legal LLMs usually measure one property, such as task accuracy, retrieval relevance or citation support, and the word "reliability" is applied to all of them §1.1. A legal conclusion also depends on the hierarchy and binding force of the source, its jurisdiction, its temporal validity and its procedural posture §1.1 §3.2. The survey argues that retrieving relevant material does not by itself establish that the material is authoritative §3.2. Authority can also be lost by amendment or overruling, so a source that was valid when indexed may not be valid when used §3.2 §5.3.

The survey names specific failure modes reported in the cited studies: citation hallucination in commercial legal research tools, failure to abstain in case-based argument generation, and bias in case-judgment summaries §8 intro §8.3. It says these do not show that every legal LLM has every failure §8 intro.

Method

The survey has no training or inference method. Its method is a framework and a review procedure.

Three axes. Legal tasks (judgment prediction, statutory reasoning, case retrieval, legal QA, summarisation, drafting, compliance and extraction, agents) form one layer §3.1 tab. 3. System mechanisms form a second layer: prompting, domain pretraining, instruction tuning, retrieval-augmented generation (RAG), knowledge graphs, tools and multi-agent workflows, neuro-symbolic constraints §2.3. Assurance and evaluation form the third: grounding, jurisdictional robustness, temporal validity, fairness and safety, professional usability §2.3 tab. 6. The facets overlap and a single system can sit in several cells §2.3 fig. 2.

Authority grounding. A claim counts as grounded when the system can record and test whether a candidate source applies under stated conditions, and can show, in a way others can inspect, that the source backs the claim. A source that is outdated, outside the jurisdiction or does not support the claim fails this test §3.2. The survey lists six attributes a system should record: source type; binding, persuasive or informative status; hierarchy and jurisdiction; valid time and later treatment; procedural posture and the difference between holding and dicta; and visible conflicts between sources §3.2. It notes that the legal effect of each must be specified for the jurisdiction under study §3.2.

Integration contract. The survey proposes four parts for a system to meet this definition: checks on source applicability; proposition-level links from claims to sources; explicit rules for re-retrieval, escalation or abstention; and an assurance record covering evaluation, review and updating §1.2. It labels these as design requirements, not a certified standard §1.2.

Verification stages. The verification described in §3.2 has three stages. The sketch below is a reading of that section. The paper states no algorithm, and the decision rules and their order are not stated.

# Authority-grounded check of one generated proposition  [§3.2] [§1.2]
# Input:  proposition (text); candidates (sources with version metadata)
# Output: "supported", "re-retrieve" or "escalate"
# Not stated: how applies() and supports() decide, and the order of outcomes.

def check_proposition(proposition, candidates, jurisdiction, date):
    # Stage 1, retrieval eligibility: admissible, identifiable versions  [§3.2]
    admissible = [c for c in candidates if c.version is not None]
    if not admissible:
        return "re-retrieve"
    # Stage 2, legal applicability: hierarchy, binding force, jurisdiction,
    # valid time and procedural context  [§3.2]
    applicable = [
        c for c in admissible
        if applies(c, jurisdiction, date)
    ]
    if not applicable:
        return "re-retrieve"
    # Stage 3, proposition support: the cited text supports the claim  [§3.2]
    supporting = [
        c for c in applicable
        if supports(c.text, proposition)
    ]
    if not supporting:
        return "escalate"
    # Conflicting authority stays visible rather than being merged  [§3.2]
    if has_conflict(supporting):
        return "escalate"
    return "supported"

Mechanisms. The survey compares mechanisms by benefit, typical legal use and main risk tab. 5. Prompting is cheap but unstable. Domain pretraining gives legal vocabulary but stale knowledge and licensing issues. Instruction tuning aligns output format but can overfit synthetic data. RAG connects outputs to candidate evidence but can miss sources or cite the wrong authority. Knowledge graphs make citation structure explicit but are costly to build and maintain. Agents decompose long workflows but compound errors and blur accountability. Neuro-symbolic constraints add explicit rules but are brittle to formalise tab. 5. No mechanism alone establishes authority grounding tab. 5.

RAG workflow. The survey splits a retrieval-based system into five stages: retrieval, reranking, generation, citation verification and human review §6.1. Errors can propagate between stages. High-recall retrieval can admit low-status or outdated material, and a correct citation can still misapply the facts §6.1. It recommends recording stage inputs and outputs and using ablations or component swaps to find where a change in performance comes from §6.1.

Evidence-map procedure. The authors searched arXiv and OpenAlex for 2022 to 2025 records, deduplicated them, applied record-level filters and screened 381 candidates §2.1. The steps and counts are in the Results section.

Evaluation package. For a legal system, the survey suggests publishing, next to the accuracy score, its retrieval quality, how well citations support the claims, the jurisdiction and date it covers, when it abstains, privacy or fairness tests where they apply, and the cost of human review §7.3. It calls this a reporting agenda, not an existing consensus §7.3.

Governance layers. Five layers, each needing different evidence: model and corpus; retrieval sources; professional workflows; institutional compliance; post-deployment monitoring §8.3. The survey does not accept any single item, such as a benchmark score, a retrieval design or a user warning, as a complete control §8.3.

Hyperparameters

The survey has no model hyperparameters. The table lists the review parameters that determine its quantitative snapshot. A reader who reproduces the search needs them.

Results

The survey reports no new experiments. Its counts describe its own evidence map and are the authors' figures. Benchmark scale numbers come from the cited benchmark papers as summarised in the survey, not from re-runs.

Evidence map (authors' counts, 2022 to 2025 window) §2.1:

Scale of cited benchmarks (as the survey reports them):

The survey says results across these benchmarks must not be ranked against each other, because tasks, metrics, sources and protocols differ §4.4 §7.1.

Paper vs code

The paper names one repository, a literature index (https://github.com/Jeryi-Sun/LLM-and-Law), in its Data and Materials Availability statement. It is not implementation code. The clone at 2482679 contains only a README, which describes the repository as a tracker of papers and resources on LLMs in law README.md. It has no LICENSE file; the README shows an MIT badge README.md. The README does not name the authors, so the link rests on the paper's own statement.

The repository holds no training or inference code, so no hyperparameter or algorithm step can be checked against code. The pseudocode in Method is a reading of §3.2, not an algorithm from the paper.

Pitfalls

• Source mismatch. The arXiv metadata used by this wiki (raw/arxiv/2303.09136.json, version v2, updated 2026-09-10) gives a title about a short survey of LLMs in legal aspects, one author, and a different abstract. The fetched LaTeX carries a longer title centred on authority grounding and system governance, and lists four authors. This card follows the LaTeX for content and names all four authors, so the title and author list need a check against the arXiv listing before the card is reviewed.
• Framework, not a finding. The authors say many requirements are proposals, not validated standards, and that the framework does not show that it improves legal outcomes §1.2 §9.2 §10. Read "should" in the source as a recommendation.
• Counts are not prevalence. One reviewer screened title and abstract only, with no second coder. The survey says its counts support navigation, not prevalence estimates §2.1 §10.
• Search blind spots. The arXiv query needs legal-domain words in the title, and the OpenAlex query stops at 200 ranked records. Work without these words in its title can be missed §2.1.
• Time window. The quantitative snapshot excludes 2026 records, while the narrative cites some 2026 work §2.1. Do not read the 327 count as a complete 2026 picture.
• Jurisdiction and binding force have no single rule. The six attributes are a checklist, and their legal effect must be set per jurisdiction §3.2.
• Benchmarks are not comparable. LexTime tests ordering of events, not whether the valid version of a rule is retrieved §7.1. Benchmark "reliability" labels mean different things in different papers §7.1.
• Regulatory references are context. Whether the Chinese interim measures apply depends on who provides the service to whom. Citing an instrument does not establish compliance §8.2.1.
• Repository. The linked repository is a literature index with no LICENSE file, so code_license is left unstated. Do not treat it as a source of code or data.
• Figures. The framework map fig. 1 and the facet diagram fig. 2 are described in words only; no image is included.

Lineage

• Builds on: legal NLP surveys and resources. The survey positions itself against them in tab. 1; none has a card in this wiki, so nothing from them is used here.
• Compared against: the prior reviews in tab. 1.
• Superseded by: not stated.