---
kind: concept
slug: functional-safety
title: Functional safety (for AI systems)
summary: >-
  Checking that an AI system works as claimed, before judging its ethics or trade-offs. Six sources: a failure taxonomy from 283 incidents, a three-layer framework for generative-AI safety evaluation, a corner-case description and a fault-diagnosis framework for automated driving, a formal-verification position paper, and a risk taxonomy built from 74 AI risk frameworks. None sets a threshold.
tags: [safety, policy, accountability, evaluation, autonomous-driving, fault-diagnosis, formal-verification, taxonomy]
papers: ["2109.09607", "2206.09511", "2310.11986", "2408.12622", "2411.09643", "2411.14520"]
related: [adversarial-robustness, algorithmic-fairness, anomaly-detection, dataset-curation, fault-diagnosis, llm-ethics, llm-evaluation, benchmark-validity, red-teaming, temporal-logic-runtime-monitoring]
status: draft
updated_at: 2026-10-31
---

## Overview

The slug covers one idea: an AI system has to work as claimed before other
questions, such as fairness or trade-offs, are worth asking. Six papers on
record take this up from different angles.

The first treats functionality as a prior question. It argues that policy and
research often assume deployed systems work as advertised, and that this
assumption is left unstated [arxiv:2206.09511 §3]. It also says functionality
is hard to pin down. Fitness for intended use is a starting point, but some
intended uses cannot be met at all [arxiv:2206.09511 §4].

The second is a framework for evaluating generative AI for safety. It argues
that a test of the model alone does not show whether a capability causes harm,
because harm depends on who uses the system, for what, and where
[arxiv:2310.11986 §2] [arxiv:2310.11986 §2.1].

The third is about data. Automated-driving systems need rare or critical
situations in their training and validation data, and such situations are
scarce. It proposes describing them in a machine-readable form so that
coverage can be checked [arxiv:2109.09607 §1] [arxiv:2109.09607 §3 RQ4].

The fourth is a fault-diagnosis framework for an autonomous shuttle bus. Small
checkers watch single subsystems and report one of five states. Dependency
rules then mark downstream parts as ignored, so the root fault stands out
[arxiv:2411.09643 §4.3] [arxiv:2411.09643 §5.1]. It is evaluated only on six
qualitative fault runs [arxiv:2411.09643 §6].

The fifth is a position paper on formal verification of driving software. It
names two open problems: reasoning about black-box probabilistic components,
and mapping software onto many-core hardware with redundancy and voting
[arxiv:2411.14520 §2]. It suggests assume-guarantee verification with runtime
checks as directions, and builds nothing [arxiv:2411.14520 §3.1]
[arxiv:2411.14520 §5].

The sixth is a systematic review of AI risk frameworks. It screens 74 included
documents and extracts 1,725 risks from them, and sorts those risks by cause
and by harm domain [arxiv:2408.12622 p. 4] [arxiv:2408.12622 tab. 1]
[arxiv:2408.12622 tab. 2]. Its counts show how often a risk appears in the
source documents. The authors say they do not estimate likelihood or severity
[arxiv:2408.12622 p. 13].

None of the six defines functional safety in the engineering sense. The
first borrows that vocabulary and proposes existing safety-engineering
standards as templates for AI [arxiv:2206.09511 §5.2.2]. The second is scoped
to harms of generative systems [arxiv:2310.11986 §1]. The third is an early
overview with no implementation [arxiv:2109.09607 §1]. The fourth uses the
dependability terms of the field [arxiv:2411.09643 §3]. The fifth is a
position paper with no method [arxiv:2411.14520 §5]. The sixth uses "AI
system safety" only as the name of one of its seven harm domains
[arxiv:2408.12622 tab. 2]. Nothing here is a
validated procedure with measured performance.

## Variants

None of the approaches specifies settings. They answer different questions:
the first asks which kinds of failure to look for, the second asks how to
organise an evaluation, the third asks how to describe rare situations so that
data coverage can be checked, the fourth asks how to locate a fault in a
running system, the fifth asks which verification problems remain open, and
the sixth asks which kinds of risk have been reported and how to label them.

### Failure taxonomy and response levers

The paper sorts failures from 283 cases in a crowdsourced incident database,
covering 2012 to 2021, into three groups and ten categories
[arxiv:2206.09511 §4.1] [arxiv:2206.09511 tab. 1]. The groups are impossible
tasks, engineering failures (design, implementation, missing safety features),
deployment failures (robustness, adversarial attacks, unanticipated
interactions) and communication failures (overstated or misrepresented
capabilities) [arxiv:2206.09511 §4.2.1] [arxiv:2206.09511 §4.2.2]
[arxiv:2206.09511 §4.2.3] [arxiv:2206.09511 §4.2.4].

On the response side, the paper maps legal tools such as consumer protection,
products liability and warranties [arxiv:2206.09511 §5.1]. It also maps
organisational tools such as internal audits, documentation, product
certification and procurement requirements [arxiv:2206.09511 §5.2]. Card:
[The Fallacy of AI Functionality](/papers/2206.09511). The category boundaries
are the authors' judgement. The coder count and tagging rules are not stated
[arxiv:2206.09511 §4.1].

### Layered sociotechnical evaluation

The framework has three layers, each with a different target
[arxiv:2310.11986 §2]:

- **Capability.** The model and its parts: training data, components, outputs
  and filters. Methods are automated tests, human annotation and adversarial
  testing [arxiv:2310.11986 §2.1] [arxiv:2310.11986 §4.2].
- **Human interaction.** Real people using the system, including usability,
  overreliance and psychological harm. Requires human participants
  [arxiv:2310.11986 §2.2] [arxiv:2310.11986 §4.2].
- **Systemic impact.** Effects on society, the economy and the environment.
  Methods are staged releases, pilots, impact assessments, and forecasts or
  simulations [arxiv:2310.11986 §2.3] [arxiv:2310.11986 app. A.2.3].

The layers are not sequential, and evaluations at each layer can run at the
same time [arxiv:2310.11986 §2.4]. The paper also maps the evaluation landscape
to six harm areas [arxiv:2310.11986 §3.1] [arxiv:2310.11986 tab. 1]. That list
is a basis for mapping and not a new research artefact. Its survey covers
evaluations known to the authors up to 10 October 2023, and is not systematic
[arxiv:2310.11986 §3.2]. Card:
[Sociotechnical Safety Evaluation of Generative AI Systems](/papers/2310.11986).

### Describing corner cases for automated driving

Corner cases are situations that occur infrequently or are critical, and that
automated-driving models need in their training and validation data
[arxiv:2109.09607 §1]. Existing scenario description languages are, the paper
says, limited in describing them [arxiv:2109.09607 §2]. The paper proposes a
machine-interpretable corner-case description (CCD) that combines a high-level
category with a concrete scenario description [arxiv:2109.09607 §3 RQ1].

The uses it lists are offline dataset analysis and coverage, generating or
recording corner cases (including in simulators), and online sensor weighting
and an interface for prediction and planning [arxiv:2109.09607 §3 RQ3]
[arxiv:2109.09607 §3 RQ4] [arxiv:2109.09607 §3 RQ5]. It is an extended
abstract with six research questions and three conceptual examples, and it
reports no experiments [arxiv:2109.09607 §3] [arxiv:2109.09607 §4]. The schema,
the labelling process and the severity metric are left open
[arxiv:2109.09607 §3 RQ2] [arxiv:2109.09607 §5]. Card:
[Description of Corner Cases in Automated Driving: Goals and Challenges](/papers/2109.09607).

### Module-level fault diagnosis with dependency propagation

The framework is built from diagnosis modules, each of which checks one
component or data stream. The paper lists five common kinds: the component's
own quality estimate, message frequency or latency, a single value against a
threshold, a reachability probe, and a custom domain check
[arxiv:2411.09643 §4.4]. Every module and group reports one of five states:
OK, WARNING, ERROR, IGNORE or UNKNOWN [arxiv:2411.09643 §4.3].

Modules are collected into groups that stand for subsystems, and groups can be
nested [arxiv:2411.09643 §4.5] [arxiv:2411.09643 §5.1]. Dependency edges pass
an error from a failed group to the groups that rely on it, and the chain
leads back to the root cause [arxiv:2411.09643 §5.1]. A driving-state flag
switches diagnosis of planning and execution on only in autonomous mode
[arxiv:2411.09643 §5.2] [arxiv:2411.09643 §6]. Card:
[Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems](/papers/2411.09643).
The authors ran it on research shuttle buses in six fault scenarios. They
report no metrics and no official code [arxiv:2411.09643 §6]. Thresholds and
window sizes are not stated [arxiv:2411.09643 §4.4] [arxiv:2411.09643 §6].

### Formal verification and runtime checks for multi-supplier driving software

The paper's starting point is that a driving system is assembled from parts
from different suppliers, some of them black boxes. Certification then has to
rely on what each part promises, expressed as assumptions and guarantees
[arxiv:2411.14520 §1] [arxiv:2411.14520 §3.1]. The suggested directions are
assume-guarantee verification across heterogeneous components, runtime
verification with uncertainty for probabilistic outputs, and, for mapping
software onto cores, probabilistic model checking of resource schemes
[arxiv:2411.14520 §3.1] [arxiv:2411.14520 §3.2]. The paper reports no
experiments [arxiv:2411.14520 §5]. Card:
[Open Challenges in the Formal Verification of Autonomous Driving](/papers/2411.14520).

### Risk taxonomy from a systematic review

The review screens records with active learning and a stopping rule, then
extracts risks from 74 included documents, which are frameworks, reviews and
reports [arxiv:2408.12622 p. 4] [arxiv:2408.12622 p. 17]. Each risk is kept in
the wording of its source [arxiv:2408.12622 p. 18]. Two taxonomies then sort
them. The causal one tags each risk by entity (human, AI or other), intent
(intentional, unintentional or other) and timing (pre-deployment,
post-deployment or other) [arxiv:2408.12622 tab. 1] [arxiv:2408.12622 p. 6].
The domain one has seven domains and 24 subdomains. Domain 7, AI system
safety, failures and limitations, has six subdomains, among them dangerous
capabilities and lack of capability or robustness [arxiv:2408.12622 tab. 2]
[arxiv:2408.12622 p. 10].

The domains may overlap, but each causal category has one level per risk
[arxiv:2408.12622 p. 10] [arxiv:2408.12622 tab. 1]. The authors report that AI
systems cause 42% of coded risks and humans 38%, and that 62% of coded risks
are post-deployment [arxiv:2408.12622 p. 11]. These are shares of coded risks
in the source documents, not of frameworks, and not estimates of likelihood or
severity [arxiv:2408.12622 p. 13]. Coverage is uneven: a typical framework
covers 8 of the 24 subdomains on average [arxiv:2408.12622 p. 11]. The paper points to a
living database of the risks [arxiv:2408.12622 p. 1]. Card:
[The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence](/papers/2408.12622).

## When to use which

The sources give no measured comparison between these approaches. The table
sorts by the question being asked.

| Situation | Use | Why |
|---|---|---|
| Deciding whether a product should ship at all | Ask whether the task can be done and the data exists, before any fairness or ethics review | The paper treats functionality as a prior question [arxiv:2206.09511 §3] [arxiv:2206.09511 §6] |
| Reviewing an incident or a launch claim | The failure taxonomy, to name the failure type; the communication categories for marketing claims | Categories and examples are in [arxiv:2206.09511 §4.2]; overstated claims are in [arxiv:2206.09511 §4.2.4] |
| Evaluating a generative model before release | Add human-interaction and systemic checks, not only capability tests | A capability test alone does not show whether a capability causes harm in context [arxiv:2310.11986 §2] |
| Evaluating audio, image or video output | Plan extra work: repurpose existing evaluations, transcribe output to text, or use a model as judge | Few evaluations cover non-text output [arxiv:2310.11986 §3.3]; each route has known losses [arxiv:2310.11986 §4.3] |
| Using a model as an automated judge | Cross-check it against human judgements, and expect it to find unsafe outputs but not missing capability | Its costs and limits are set out in [arxiv:2310.11986 §4.3.3] |
| Checking whether a driving dataset covers rare, critical situations | Describe the cases in a structured form and count them per dataset; the paper proposes this but gives no schema | Offline dataset coverage is an intended use [arxiv:2109.09607 §3 RQ4]; the format is open [arxiv:2109.09607 §3 RQ2] |
| Generating or recording extra corner cases for training or testing | Use the description to drive scenario catalogues or simulators, and check the result yourself | The paper names these uses [arxiv:2109.09607 §3 RQ3]; no method or validation is given |
| Locating which subsystem of a running vehicle is the root of a fault, for an operator display | Module checks grouped by component, with dependency propagation and a driving-state gate | The display rules are described [arxiv:2411.09643 §5.1] [arxiv:2411.09643 §5.3]; you must choose every threshold yourself [arxiv:2411.09643 §4.4] |
| Verifying a system built from black-box parts supplied by several vendors | Document assumptions and guarantees per part, then add runtime checks; treat this as a direction, not a finished method | The paper names the approach but gives no contract language or verdict logic [arxiv:2411.14520 §3.1] |
| Mapping software onto redundant multi-core hardware | Probabilistic model checking of the resource scheme, as a suggestion only | Named as a direction without results [arxiv:2411.14520 §3.2] |
| Building a shared list of risks for a safety case, audit or policy document | Use the causal and domain labels as a shared vocabulary, and do not treat the list as complete | The labels are in [arxiv:2408.12622 tab. 1] and [arxiv:2408.12622 tab. 2]; the list covers only the 74 included documents [arxiv:2408.12622 p. 14] |
| Setting a pass/fail threshold | Not answered by any of the six; set it through an accountable process | No value is given; the paper says thresholds must come from a legitimate process [arxiv:2310.11986 §5.3.2] |
| Scoring how severe a corner case is | Not answered by these sources | Metric-based severity is proposed as future work without a formula [arxiv:2109.09607 §5] |
| Making evaluations independent | Keep assurance evaluations away from the developers | The paper warns that a measure used as a target stops measuring well [arxiv:2310.11986 §5.4] |
| Choosing a concrete safety standard to implement | Not answered by these sources | Standards are named as templates, without their requirements [arxiv:2206.09511 §5.2.2] |
| Measuring how often a system fails in use, or how fast a fault is detected | Not answered by these sources | The taxonomy counts cases, not rates [arxiv:2206.09511 §4.1]; the survey counts evaluations, not failures [arxiv:2310.11986 §3.3]; the fault-diagnosis paper reports no rates or delays [arxiv:2411.09643 §6]; the risk review counts appearances in documents, not rates [arxiv:2408.12622 p. 13] |

## Implementation sketch

Of the six sources, only two give a procedure to run on a system: the
fault-diagnosis rules and the layered evaluation sketched below. The sixth
gives pseudocode for its review method, which builds a taxonomy from a
literature and is not a release check [arxiv:2408.12622 p. 19]. The first has
no algorithm for building or testing a system.
Its only explicit procedure is the case-selection and tagging process behind
the taxonomy, which is a research method and not a release gate
[arxiv:2206.09511 §4.1]. The corner-case paper has no algorithm either. Its
conceptual record format is on its card [arxiv:2109.09607 §4]. The
formal-verification paper has no algorithm; its runtime-check outline is on its
card, and the authors' reading is marked as ours [arxiv:2411.14520 §3.1].

The fault-diagnosis paper gives its state-collapsing and display rules as
pseudocode on its card, with the partly undefined cases marked there
[arxiv:2411.09643 §4.5] [arxiv:2411.09643 §5.1]. Read its Pitfalls before
implementing it.

The second source has no algorithm box. The sketch below is our rendering of
its operational procedure [arxiv:2310.11986 §4.1], with the validity step from
[arxiv:2310.11986 §4.1.1]. The names are ours, and the sketch is a structure to
adapt, not a validated tool.

```pseudocode
# Layered evaluation of one harm area  [arxiv:2310.11986 §4.1]
# Input:  harm (one area of the taxonomy), threshold (set outside the eval)
# Output: per-layer measurements and a judgement

def evaluate_harm(harm, threshold):
    # Split the harm into narrower concepts that can be measured  [app. A.3]
    concepts = split_into_concepts(harm)
    measurements = {}
    # Layers are independent and may run in parallel  [§2.4]
    for layer in ["capability", "human_interaction", "systemic"]:
        # Map each concept to observable metrics at this layer  [§4.1]
        metrics = choose_metrics(concepts, layer)
        # Use a method suited to the layer  [§4.2]
        measurements[layer] = run_methods(metrics, layer)
    # Document limits and cross-validate before judging  [§4.1.1]
    validated = check_validity(measurements)
    # Judge against a threshold from an accountable process  [§5.3.2]
    return judge(validated, threshold)
```

## Contradictions

Where the sources give numbers, they measure different things, so no
numerical conflict is claimed across papers. The papers differ in framing, and
each side is cited here.

- The first places functionality before ethics and fairness work, and argues
  that those fields presume it [arxiv:2206.09511 §3].
- The second treats social and ethical harms as part of safety evaluation,
  with their own harm areas [arxiv:2310.11986 §3.1].
- The third treats rare situations as a data-coverage problem for automated
  driving [arxiv:2109.09607 §1].
- The fourth treats safety as a runtime state that a system reports about its
  own parts [arxiv:2411.09643 §4.3].
- The fifth treats a system as a composition of components with checkable
  contracts [arxiv:2411.14520 §3.1], whereas the second argues that a test of
  the model alone misses how the system is used [arxiv:2310.11986 §2].
- The sixth sorts risks by cause, intent, timing and harm domain, and does not
  check a running system against them [arxiv:2408.12622 tab. 1]
  [arxiv:2408.12622 tab. 2].

Within the sixth source, the human share of coded risks is 38% in the abstract
and results, and 37% in the practical implications text
[arxiv:2408.12622 p. 1] [arxiv:2408.12622 p. 11] [arxiv:2408.12622 p. 12]. Check
the supplementary tables before quoting one of the two values.

Within the fourth source, the aggregation rule disagrees with itself. Section
4.5 says a group is OK only if all its submodules are OK. The evaluation
aggregates with OR, so a group is ERROR if any module is ERROR
[arxiv:2411.09643 §4.5] [arxiv:2411.09643 §6]. Both are cited on its card.

The cards record no citation between two of these six papers. Their counts
also measure different things, so the case counts in the first, the evaluation
counts in the second and the risk shares in the sixth should not be compared.
The third, fourth and fifth report no counts.

## Papers

- [Description of Corner Cases in Automated Driving: Goals and Challenges](/papers/2109.09607): proposes a machine-interpretable corner-case description on top of scenario languages, with six research questions and three conceptual examples. Extended abstract, no experiments.
- [The Fallacy of AI Functionality](/papers/2206.09511): argues that functionality is a prior question to AI ethics, builds a failure taxonomy from 283 cases and maps legal and organisational levers. Position paper, no experiments.
- [Sociotechnical Safety Evaluation of Generative AI Systems](/papers/2310.11986): a three-layer framework for generative-AI safety evaluation, plus a survey of existing evaluations by harm area, output modality and layer. No experiments of its own; survey counts are author-reported.
- [Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems](/papers/2411.09643): modular diagnosis checks with five states, dependency-based aggregation and a driving-state gate, run on shuttle buses in six qualitative fault scenarios. No metrics, no official code.
- [Open Challenges in the Formal Verification of Autonomous Driving](/papers/2411.14520): position paper on two open challenges, composing black-box probabilistic components and mapping software onto redundant many-core hardware. No experiments.
- [The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence](/papers/2408.12622): a systematic review of 74 AI risk frameworks, yielding 1,725 risks sorted by a causal and a domain taxonomy. No model is trained or evaluated; the shares are author-reported.