Functional safety (for AI systems)

Overview

The slug covers one idea: an AI system has to work as claimed before other questions, such as fairness or trade-offs, are worth asking. Six papers on record take this up from different angles.

The first treats functionality as a prior question. It argues that policy and research often assume deployed systems work as advertised, and that this assumption is left unstated 2206.09511 §3. It also says functionality is hard to pin down. Fitness for intended use is a starting point, but some intended uses cannot be met at all 2206.09511 §4.

The second is a framework for evaluating generative AI for safety. It argues that a test of the model alone does not show whether a capability causes harm, because harm depends on who uses the system, for what, and where 2310.11986 §2 2310.11986 §2.1.

The third is about data. Automated-driving systems need rare or critical situations in their training and validation data, and such situations are scarce. It proposes describing them in a machine-readable form so that coverage can be checked 2109.09607 §1 2109.09607 §3 RQ4.

The fourth is a fault-diagnosis framework for an autonomous shuttle bus. Small checkers watch single subsystems and report one of five states. Dependency rules then mark downstream parts as ignored, so the root fault stands out 2411.09643 §4.3 2411.09643 §5.1. It is evaluated only on six qualitative fault runs 2411.09643 §6.

The fifth is a position paper on formal verification of driving software. It names two open problems: reasoning about black-box probabilistic components, and mapping software onto many-core hardware with redundancy and voting 2411.14520 §2. It suggests assume-guarantee verification with runtime checks as directions, and builds nothing 2411.14520 §3.1 2411.14520 §5.

The sixth is a systematic review of AI risk frameworks. It screens 74 included documents and extracts 1,725 risks from them, and sorts those risks by cause and by harm domain 2408.12622 p. 4 2408.12622 tab. 1 2408.12622 tab. 2. Its counts show how often a risk appears in the source documents. The authors say they do not estimate likelihood or severity 2408.12622 p. 13.

None of the six defines functional safety in the engineering sense. The first borrows that vocabulary and proposes existing safety-engineering standards as templates for AI 2206.09511 §5.2.2. The second is scoped to harms of generative systems 2310.11986 §1. The third is an early overview with no implementation 2109.09607 §1. The fourth uses the dependability terms of the field 2411.09643 §3. The fifth is a position paper with no method 2411.14520 §5. The sixth uses "AI system safety" only as the name of one of its seven harm domains 2408.12622 tab. 2. Nothing here is a validated procedure with measured performance.

Variants

None of the approaches specifies settings. They answer different questions: the first asks which kinds of failure to look for, the second asks how to organise an evaluation, the third asks how to describe rare situations so that data coverage can be checked, the fourth asks how to locate a fault in a running system, the fifth asks which verification problems remain open, and the sixth asks which kinds of risk have been reported and how to label them.

Failure taxonomy and response levers

The paper sorts failures from 283 cases in a crowdsourced incident database, covering 2012 to 2021, into three groups and ten categories 2206.09511 §4.1 2206.09511 tab. 1. The groups are impossible tasks, engineering failures (design, implementation, missing safety features), deployment failures (robustness, adversarial attacks, unanticipated interactions) and communication failures (overstated or misrepresented capabilities) 2206.09511 §4.2.1 2206.09511 §4.2.2 2206.09511 §4.2.3 2206.09511 §4.2.4.

On the response side, the paper maps legal tools such as consumer protection, products liability and warranties 2206.09511 §5.1. It also maps organisational tools such as internal audits, documentation, product certification and procurement requirements 2206.09511 §5.2. Card: The Fallacy of AI Functionality. The category boundaries are the authors' judgement. The coder count and tagging rules are not stated 2206.09511 §4.1.

Layered sociotechnical evaluation

The framework has three layers, each with a different target 2310.11986 §2:

• Capability. The model and its parts: training data, components, outputs and filters. Methods are automated tests, human annotation and adversarial testing 2310.11986 §2.1 2310.11986 §4.2.
• Human interaction. Real people using the system, including usability, overreliance and psychological harm. Requires human participants 2310.11986 §2.2 2310.11986 §4.2.
• Systemic impact. Effects on society, the economy and the environment. Methods are staged releases, pilots, impact assessments, and forecasts or simulations 2310.11986 §2.3 2310.11986 app. A.2.3.

The layers are not sequential, and evaluations at each layer can run at the same time 2310.11986 §2.4. The paper also maps the evaluation landscape to six harm areas 2310.11986 §3.1 2310.11986 tab. 1. That list is a basis for mapping and not a new research artefact. Its survey covers evaluations known to the authors up to 10 October 2023, and is not systematic 2310.11986 §3.2. Card: Sociotechnical Safety Evaluation of Generative AI Systems.

Describing corner cases for automated driving

Corner cases are situations that occur infrequently or are critical, and that automated-driving models need in their training and validation data 2109.09607 §1. Existing scenario description languages are, the paper says, limited in describing them 2109.09607 §2. The paper proposes a machine-interpretable corner-case description (CCD) that combines a high-level category with a concrete scenario description 2109.09607 §3 RQ1.

The uses it lists are offline dataset analysis and coverage, generating or recording corner cases (including in simulators), and online sensor weighting and an interface for prediction and planning 2109.09607 §3 RQ3 2109.09607 §3 RQ4 2109.09607 §3 RQ5. It is an extended abstract with six research questions and three conceptual examples, and it reports no experiments 2109.09607 §3 2109.09607 §4. The schema, the labelling process and the severity metric are left open 2109.09607 §3 RQ2 2109.09607 §5. Card: Description of Corner Cases in Automated Driving: Goals and Challenges.

Module-level fault diagnosis with dependency propagation

The framework is built from diagnosis modules, each of which checks one component or data stream. The paper lists five common kinds: the component's own quality estimate, message frequency or latency, a single value against a threshold, a reachability probe, and a custom domain check 2411.09643 §4.4. Every module and group reports one of five states: OK, WARNING, ERROR, IGNORE or UNKNOWN 2411.09643 §4.3.

Modules are collected into groups that stand for subsystems, and groups can be nested 2411.09643 §4.5 2411.09643 §5.1. Dependency edges pass an error from a failed group to the groups that rely on it, and the chain leads back to the root cause 2411.09643 §5.1. A driving-state flag switches diagnosis of planning and execution on only in autonomous mode 2411.09643 §5.2 2411.09643 §6. Card: Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems. The authors ran it on research shuttle buses in six fault scenarios. They report no metrics and no official code 2411.09643 §6. Thresholds and window sizes are not stated 2411.09643 §4.4 2411.09643 §6.

Formal verification and runtime checks for multi-supplier driving software

The paper's starting point is that a driving system is assembled from parts from different suppliers, some of them black boxes. Certification then has to rely on what each part promises, expressed as assumptions and guarantees 2411.14520 §1 2411.14520 §3.1. The suggested directions are assume-guarantee verification across heterogeneous components, runtime verification with uncertainty for probabilistic outputs, and, for mapping software onto cores, probabilistic model checking of resource schemes 2411.14520 §3.1 2411.14520 §3.2. The paper reports no experiments 2411.14520 §5. Card: Open Challenges in the Formal Verification of Autonomous Driving.

Risk taxonomy from a systematic review

The review screens records with active learning and a stopping rule, then extracts risks from 74 included documents, which are frameworks, reviews and reports 2408.12622 p. 4 2408.12622 p. 17. Each risk is kept in the wording of its source 2408.12622 p. 18. Two taxonomies then sort them. The causal one tags each risk by entity (human, AI or other), intent (intentional, unintentional or other) and timing (pre-deployment, post-deployment or other) 2408.12622 tab. 1 2408.12622 p. 6. The domain one has seven domains and 24 subdomains. Domain 7, AI system safety, failures and limitations, has six subdomains, among them dangerous capabilities and lack of capability or robustness 2408.12622 tab. 2 2408.12622 p. 10.

The domains may overlap, but each causal category has one level per risk 2408.12622 p. 10 2408.12622 tab. 1. The authors report that AI systems cause 42% of coded risks and humans 38%, and that 62% of coded risks are post-deployment 2408.12622 p. 11. These are shares of coded risks in the source documents, not of frameworks, and not estimates of likelihood or severity 2408.12622 p. 13. Coverage is uneven: a typical framework covers 8 of the 24 subdomains on average 2408.12622 p. 11. The paper points to a living database of the risks 2408.12622 p. 1. Card: The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence.

When to use which

The sources give no measured comparison between these approaches. The table sorts by the question being asked.

Implementation sketch

Of the six sources, only two give a procedure to run on a system: the fault-diagnosis rules and the layered evaluation sketched below. The sixth gives pseudocode for its review method, which builds a taxonomy from a literature and is not a release check 2408.12622 p. 19. The first has no algorithm for building or testing a system. Its only explicit procedure is the case-selection and tagging process behind the taxonomy, which is a research method and not a release gate 2206.09511 §4.1. The corner-case paper has no algorithm either. Its conceptual record format is on its card 2109.09607 §4. The formal-verification paper has no algorithm; its runtime-check outline is on its card, and the authors' reading is marked as ours 2411.14520 §3.1.

The fault-diagnosis paper gives its state-collapsing and display rules as pseudocode on its card, with the partly undefined cases marked there 2411.09643 §4.5 2411.09643 §5.1. Read its Pitfalls before implementing it.

The second source has no algorithm box. The sketch below is our rendering of its operational procedure 2310.11986 §4.1, with the validity step from 2310.11986 §4.1.1. The names are ours, and the sketch is a structure to adapt, not a validated tool.

# Layered evaluation of one harm area  [arxiv:2310.11986 §4.1]
# Input:  harm (one area of the taxonomy), threshold (set outside the eval)
# Output: per-layer measurements and a judgement

def evaluate_harm(harm, threshold):
    # Split the harm into narrower concepts that can be measured  [app. A.3]
    concepts = split_into_concepts(harm)
    measurements = {}
    # Layers are independent and may run in parallel  [§2.4]
    for layer in ["capability", "human_interaction", "systemic"]:
        # Map each concept to observable metrics at this layer  [§4.1]
        metrics = choose_metrics(concepts, layer)
        # Use a method suited to the layer  [§4.2]
        measurements[layer] = run_methods(metrics, layer)
    # Document limits and cross-validate before judging  [§4.1.1]
    validated = check_validity(measurements)
    # Judge against a threshold from an accountable process  [§5.3.2]
    return judge(validated, threshold)

Contradictions

Where the sources give numbers, they measure different things, so no numerical conflict is claimed across papers. The papers differ in framing, and each side is cited here.

• The first places functionality before ethics and fairness work, and argues that those fields presume it 2206.09511 §3.
• The second treats social and ethical harms as part of safety evaluation, with their own harm areas 2310.11986 §3.1.
• The third treats rare situations as a data-coverage problem for automated driving 2109.09607 §1.
• The fourth treats safety as a runtime state that a system reports about its own parts 2411.09643 §4.3.
• The fifth treats a system as a composition of components with checkable contracts 2411.14520 §3.1, whereas the second argues that a test of the model alone misses how the system is used 2310.11986 §2.
• The sixth sorts risks by cause, intent, timing and harm domain, and does not check a running system against them 2408.12622 tab. 1 2408.12622 tab. 2.

Within the sixth source, the human share of coded risks is 38% in the abstract and results, and 37% in the practical implications text 2408.12622 p. 1 2408.12622 p. 11 2408.12622 p. 12. Check the supplementary tables before quoting one of the two values.

Within the fourth source, the aggregation rule disagrees with itself. Section 4.5 says a group is OK only if all its submodules are OK. The evaluation aggregates with OR, so a group is ERROR if any module is ERROR 2411.09643 §4.5 2411.09643 §6. Both are cited on its card.

The cards record no citation between two of these six papers. Their counts also measure different things, so the case counts in the first, the evaluation counts in the second and the risk shares in the sixth should not be compared. The third, fourth and fifth report no counts.

Papers

• Description of Corner Cases in Automated Driving: Goals and Challenges: proposes a machine-interpretable corner-case description on top of scenario languages, with six research questions and three conceptual examples. Extended abstract, no experiments.
• The Fallacy of AI Functionality: argues that functionality is a prior question to AI ethics, builds a failure taxonomy from 283 cases and maps legal and organisational levers. Position paper, no experiments.
• Sociotechnical Safety Evaluation of Generative AI Systems: a three-layer framework for generative-AI safety evaluation, plus a survey of existing evaluations by harm area, output modality and layer. No experiments of its own; survey counts are author-reported.
• Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems: modular diagnosis checks with five states, dependency-based aggregation and a driving-state gate, run on shuttle buses in six qualitative fault scenarios. No metrics, no official code.
• Open Challenges in the Formal Verification of Autonomous Driving: position paper on two open challenges, composing black-box probabilistic components and mapping software onto redundant many-core hardware. No experiments.
• The AI risk repository: A meta-review, database, and taxonomy of risks from artificial intelligence: a systematic review of 74 AI risk frameworks, yielding 1,725 risks sorted by a causal and a domain taxonomy. No model is trained or evaluated; the shares are author-reported.