A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validation

In plain words

Large language models answer questions fluently, but they can be wrong, leak data, be tricked into bad behaviour, and cost a lot of energy to train. This survey asks a practical question: could the checking methods that engineers already use for ordinary software, and for older neural networks, be adapted to make these models safer?

The authors sort the known problems into three groups: flaws the model has on its own, attacks by people, and bugs that come from how the model is built into products. They then walk through four kinds of checks: testing the model for failures, trying to prove properties of it, watching it while it runs, and setting rules for how it is used. The paper is a map of existing work and open gaps. It does not propose a new method or run new experiments.

TL;DR

• A survey of LLM safety and trustworthiness from a verification and validation (V&V) angle, with 370+ references (the bibliography in the source lists 374 entries).
• Vulnerabilities are grouped as inherent issues (performance, sustainability, other trustworthiness), attacks (privacy, robustness, backdoor, poisoning and disinformation) and unintended bugs (exposure of user data, bias).
• Four V&V families are discussed: falsification and evaluation, verification, runtime monitoring, and regulation and ethical use.
• The authors' main conclusion is that formal verification of LLMs is largely absent from the literature, and that runtime monitoring of LLM outputs is also barely studied.
• Use it as an entry point and a bibliography, not as an implementable method.

Problem

The paper asks whether V&V techniques, which check an implementation against its specification and purpose §1, can be extended across the whole lifecycle of LLMs §1. The authors argue that current work mostly identifies vulnerabilities and has limited systematic methods to evaluate or verify safety properties §1.

Two assumptions shape the scope. First, white-box techniques that take the learned parameters as input are judged impractical at LLM scale, so the authors focus on black-box methods §1. Second, LLMs are non-deterministic in the sense that identical inputs can give different outputs across runs, which the authors say most existing V&V methods do not handle §1. The survey explicitly excludes image synthesis issues such as fake-image detection §2.1.

Method

This is a literature review, so the "method" is the structure of the survey and the framework it proposes for placing V&V techniques on the LLM lifecycle.

Scope and categories §2.1. LLMs are split into text-based conversational AI and text-based image synthesis, and the survey focuses on the former §2.1. Models are grouped as encoder-only, decoder-only and encoder-decoder §2.1.

Lifecycle §2.2 fig. 1 fig. 3. Offline construction has three steps: pre-training, adaptation tuning (instruction tuning and alignment tuning), and utilisation improvement (for example in-context learning and chain-of-thought) §2.2. After training an evaluation step checks basic performance, safety impact in an application, and public benchmarks; a failure sends the process back to a training step §2.2. Deployment places the model in a product, usually behind a guardrail §2.2.

Key techniques §2.3. RLHF is described as learning a reward model from human comparisons and then optimising the policy with an RL algorithm such as PPO §2.3.1. Guardrails are a refusal layer in front of end users; the authors call them a thin layer that prompt injection can bypass §2.3.2.

Vulnerability taxonomy §3 fig. 4. Inherent issues cannot be solved by the model itself but can be reduced with more data and better training §3. Attacks are deliberate and tied to lifecycle stages §3. Unintended bugs come from developers and integration §3.

General verification framework §4 fig. 5. Falsification and evaluation run in the evaluation stage, in parallel with explanation techniques. Verification, which is costly, is only needed once a model passes those checks §4. Ethical principles and regulation apply throughout §4.

Testing formalism §5.4. The survey writes an LLM as a function M from a data space D to D, and a human response as a function H on the same space. Automated testing needs an oracle O, a coverage metric C and a test set P. Verification then reduces to checking whether the share of test cases in P that pass O exceeds a set threshold §5.4. Statistical evaluation can give a probabilistic bound: the failure rate is below a threshold l with probability greater than 1 minus epsilon §5.4. The authors note that coverage metrics and test-generation algorithms are the hard parts for LLMs, because the models must be treated as black boxes and the input space is natural language §5.4.

Runtime monitoring formalism §7.1 eq. 2. An ID confidence score S(x) is compared with a threshold gamma. An input is flagged OoD when S(x) is below gamma eq. 2. Three families of score are listed: input density estimation, feature-space distance, and output confidence such as maximum softmax probability §7.1.

# Out-of-distribution flag from a confidence score  [eq. 2] [§7.1]
# Input:  x (one input), threshold gamma (set in advance)
# Output: "ID" or "OoD"

def monitor(x, score_fn, gamma):
    # Confidence S(x) from one of three families: density,
    # feature distance, or output confidence  [§7.1]
    s = score_fn(x)
    if s >= gamma:
        return "ID"
    return "OoD"

Carbon estimate §3.1.2 eq. 1. The survey gives an estimate of CO2-equivalent emissions from GPU hours, GPU power draw and a power usage effectiveness factor (PUE) eq. 1.

# Training emissions estimate  [eq. 1] [§3.1.2]
# Input:  gpu_hours (float), gpu_power_kw (float), pue (float)
# Output: tonnes CO2-equivalent (float)

def co2_equivalent_tonnes(gpu_hours, gpu_power_kw, pue):
    # Constant 0.385 is taken from the paper's equation  [eq. 1]
    # Units of gpu_power_kw are not stated in the paper  [eq. 1]
    return 0.385 * gpu_hours * gpu_power_kw * pue

Randomised smoothing §6.1.3. The smoothed classifier returns the class c that is most likely under small random noise added to the input x, written as argmax over c of the probability that f(x + epsilon) equals c, with epsilon drawn from a fixed distribution such as a Gaussian §6.1.3. The survey describes WordDP and RanMASK as NLP applications §6.1.3.

Hyperparameters

Not applicable: the paper reports no experiments and sets no training or test hyperparameters. The only numeric constants stated in the text are below.

Results

Author-reported, from the survey's own tables and text. These are literature compilations, not new measurements.

Hardware and energy for LLM training tab. 2. The survey compiles parameter counts, dataset sizes, hardware and energy from the literature. Selected rows:

Other rows in tab. 2 (Chinchilla, ERNIE 3.0, PanGu-alpha, YaLM, AlexaTM, Galactica, Cerebras-GPT, PanGu-Sigma) are cited there with their own figures. Energy is "not stated" where the table gives "-". The paper's text cites GPT-4's training energy from two external sources §3.1.2.

Carbon example §3.1.2 eq. 1. The paper's worked example gives 552 tonnes CO2-equivalent for GPT-3 training (1287 MWh times 0.385 times 1.114) §3.1.2.

Scale comparison §3.1.2. The survey puts GPT-4's training energy at roughly 0.05% of Bitcoin's 2021 consumption, which it estimates at 108 TWh §3.1.2.

Performance example tab. 1. One family-relations question was put to five chatbots on 24 August 2023. Four answered that Alice is Eve's granddaughter (Bing Chat adds that she is Adam's daughter); GPT-4 answered that she is Eve's daughter tab. 1. The text says most of these models could not give a correct answer §3.1.1. The table is an anecdote, not a benchmark.

Robustness examples cited §3.2.2. The survey reports that one study found ChatGPT surpasses other models on adversarial classification tasks (AdvGLUE and ANLI) while still far from perfect §3.2.2. Another found it weaker than commercial systems on biomedical and Reddit translation §3.2.2.

Paper vs code

No official code. The paper contains no code link: the only URLs are to ChatGPT, Bing and a footnoted third-party guide §2.2 §3.1.2. The abstract and the arXiv metadata carry no code link. A GitHub repository search (api.github.com/search/repositories) for the title, and for the title words with the authors' names, returned zero results.

Pitfalls

• No method to implement. Everything here is a summary of other work. An agent should not treat the formulas in Method as a validated pipeline; they are reproduced from the survey's text and cite the originals.
• Wrong table reference. The emissions text says GPU power draw is "provided in Table 1" §3.1.2, but Table 1 is the reasoning-error example. The hardware and energy data is in Table 2 tab. 2. The text reference is inconsistent with the paper's own numbering.
• PUE inconsistency. The text calls PUE "commonly set" to 1.1, but the GPT-3 worked example uses 1.114 §3.1.2. The GPU power unit in eq. 1 is not stated.
• Mixed-source numbers. The energy and parameter figures come from different sources with different methods, and the table mixes measured hardware counts with estimates (for example, "about 25000" A100s for GPT-4 and "about 640 MWh" for PaLM) tab. 2. Treat them as order-of-magnitude.
• Dated model behaviour. Several claims about ChatGPT refer to specific March and June 2023 versions §5.3 §5.1. Their behaviour may have changed.
• Advocacy in places. Sections 7 and 9 include opinions about what the field should do. They are argued positions, not findings. Section 7.4 states the authors' view that models able to judge their own inputs and outputs reliably are very hard to reach in the foreseeable future §7.4.
• Coverage is selective. The authors state the survey is not complete and focuses on V&V-relevant work §1. Applications of LLMs in specific domains are not covered §1.
• Output-failure monitoring is largely absent. The authors report that they found no monitoring work for reasoning or code errors, or for intended attacks such as backdoor or poisoning §7.4. Do not assume tools exist for these.
• Verification of LLMs is largely absent. The authors say there is little research on verifying LLMs and encourage black-box verification methods §6.2. Techniques in §6.1 (interval bound propagation, abstract interpretation, randomised smoothing) are for NLP models and small networks, not LLMs of the scale discussed.
• Figures are not reproduced. The paper's figures (lifecycle, vulnerability taxonomy, V&V framework, verification pipelines) are described in words only fig. 1–fig. 5; no image is included here.

Lineage

• Builds on: the vulnerability and verification literature the authors cite for convolutional networks §1, including surveys of safety of machine-learning systems §1.
• Compared against: other LLM survey papers. The authors name Zhou et al. and Zhao et al. surveys and a categorical archive of ChatGPT failures, and state that their survey focuses on V&V §1.
• Superseded by: not known from content already in this wiki.