---
kind: concept
slug: fault-diagnosis
title: Fault diagnosis (for multi-component systems)
summary: >-
  Finding faulty parts of a system made of many interacting components, then separating the root cause from its knock-on effects. Two sources: a rule-based framework for autonomous driving (qualitative runs only) and an LLM-agent benchmark on live Kubernetes faults (author-reported 38-73% diagnosis success).
tags: [fault-diagnosis, safety, autonomous-driving, dependency-graph, fault-aggregation, root-cause-analysis, llm-agents, site-reliability]
papers: ["2411.09643", "2605.07161"]
related: [functional-safety, anomaly-detection, human-in-the-loop, llm-agent, llm-as-a-judge]
status: draft
updated_at: 2026-10-31
---

## Overview

A system built from many parts can fail in many places, and one failed part
can make others look broken. Fault diagnosis has two jobs: detect which parts
are faulty, and show where the fault starts instead of listing every symptom.
Two sources cover it, and they tackle the second job differently. The
rule-based design in [arxiv:2411.09643 §4] reports every check, and every group
of checks, into one small state vocabulary (OK, WARNING, ERROR, IGNORE,
UNKNOWN) [arxiv:2411.09643 §4.3]. Because the states share one vocabulary,
checks can be added or removed without changing the rest of the system. The
LLM-agent benchmark in [arxiv:2605.07161 §2] instead gives an agent monitoring
tools and asks it to name the root cause of a fault in a live cluster, then
fix it.

## Variants

The first four variants below come from the rule-based framework. The fifth
is a different kind of method. The two sources are not compared against each
other, and the rule-based paper does not compare against other diagnosis
systems.

### Module kinds (what a check looks at)

The paper lists five common kinds of diagnosis module [arxiv:2411.09643 §4.4]:
a component's own quality estimate, message frequency or latency, a single
value against a threshold, a reachability probe (watchdog), and a custom
domain check. Each kind reports into the shared state vocabulary. The paper
names the checks but gives no threshold, rate or timeout values, so an
implementer must choose them [arxiv:2411.09643 §6].

Card: [Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems](/papers/2411.09643).

### Group aggregation

Module states are collected into groups that represent subsystems, and groups
can contain groups [arxiv:2411.09643 §4.5] [arxiv:2411.09643 §5.1]. The
stated rule is that a group is OK only when all its submodules are OK, and
otherwise WARNING or ERROR [arxiv:2411.09643 §4.5]. The evaluation instead
reports the worst state among its modules [arxiv:2411.09643 §6]. See
Contradictions.

### Dependency-based propagation

Groups are linked by dependency edges. When a group is in ERROR, the groups
that depend on it are shown as IGNORE, so the operator sees one root cause
rather than many alarms [arxiv:2411.09643 §5.1] [arxiv:2411.09643 §6].
Traversing the graph from the faulty component is how the paper identifies the
root cause [arxiv:2411.09643 §5.1]. The paper does not say whether the rule
is transitive over several levels.

### Driving-state gating

A state flag suppresses diagnosis of components that are not needed in the
current mode. In the paper, Planning and Execution are not diagnosed outside
autonomous mode [arxiv:2411.09643 §5.2] [arxiv:2411.09643 fig. 5]. The flag
is driven by a mission component, and the paper's state machine has start-up,
idling, driving and shut-down states [arxiv:2411.09643 §5.2]. The transition
conditions are not defined.

### LLM agents on live systems (root-cause diagnosis)

An LLM agent gets metrics, logs, traces and a `kubectl` wrapper, and must say
what is wrong and then mitigate it [arxiv:2605.07161 §2.3] [§2.1]. The
benchmark deploys an application on Kubernetes, injects one or more faults,
and optionally adds background noise [arxiv:2605.07161 §2.4] [§3]. The
diagnosis and the fix are graded separately. A checklist judge scores the
written diagnosis against the known cause, and a check on live system state
scores the fix [arxiv:2605.07161 §2.5].

The checklist has nine yes/no questions in three dimensions: fault
localisation, fault characterisation and failure scope [arxiv:2605.07161 §2.5]
[app. A]. The paper's pass threshold is 7/9, but the released code uses 0.70,
and the two differ on some score patterns. The card's Paper vs code section
gives an example [arxiv:2605.07161 §2.5]. The judge agrees with one human
expert on 100 diagnoses at Cohen's κ = 0.90 for the default model, and at
0.70 and 0.76 for two others [arxiv:2605.07161 §2.5] [tab. 2].

Card: [SREGym](/papers/2605.07161), a live Kubernetes benchmark for SRE agents.

## When to use which

The rule-based rows rest on the paper's qualitative runs and design argument
only, because it reports no accuracy, latency or detection-rate figures. The
agent rows rest on author-reported benchmark numbers.

| Situation | Use | Why |
|---|---|---|
| Components form a data chain, and one fault floods many alarms | Dependency-based propagation | In the sensor-outage run, the dependency analysis collapses the alarms to one ERROR with IGNORE downstream [arxiv:2411.09643 §6] [arxiv:2411.09643 fig. 6b]. Without it, the fault spread to unlinked modules [arxiv:2411.09643 fig. 6a]. |
| Some parts only matter in certain modes | Driving-state gating | A planner fault shows as IGNORE while the vehicle is parked and as ERROR once it is active [arxiv:2411.09643 §6] [arxiv:2411.09643 fig. 6d] [arxiv:2411.09643 fig. 6f]. |
| Operators must read the state without expert knowledge | Grouped display with five states | The paper's interface shows eight groups and greys out ignored ones [arxiv:2411.09643 §5.3] [arxiv:2411.09643 fig. 4]. Its user testing is not reported. |
| A fault must trigger a physical reaction | Vital-ERROR rule | The paper sets an immediate deceleration of 1 m/s² for a vital ERROR [arxiv:2411.09643 §5.4]. The paper gives this value for its own shuttle bus. |
| Thresholds or detection rates must be justified | Not from the rule-based source | The paper gives no threshold values, detection rates or false-alarm rates [arxiv:2411.09643 §6]. Set them from your own data. |
| Components cannot report timing or reachability | Not covered | The module kinds assume timing and probe data are available [arxiv:2411.09643 §4.4]. |
| Signals are logs, metrics and traces, and a person would be the on-call diagnoser | LLM-agent diagnosis, with a checklist judge | Agents were given the tools a human on-call engineer uses [arxiv:2605.07161 §2.3]. Diagnosis succeeded in 38.1% to 72.6% of runs across four agent-model pairs, and noise lowered it for every pair [arxiv:2605.07161 tab. 3] [§3.1]. |
| Fault is a disk-level, metastable or compound failure | Not reliable from this source | On the latent disk-error problem, no run of two agents scored above 0.22 on the diagnosis checklist. On the metastable problem, no agent identified both interacting components [arxiv:2605.07161 §3.2]. |
| Mitigation is chosen from the agent's diagnosis | Check the diagnosis first | After a correct diagnosis, mitigation succeeded 68% to 89% of the time; after an incorrect one, 22% to 62% [arxiv:2605.07161 §3.3] [tab. 5]. |
| Judging agent diagnoses with an LLM | Judge validated on your own expert labels | Judge agreement with the expert ranged from κ = 0.70 to 0.90, and the validation used one expert and 100 samples [arxiv:2605.07161 §2.5] [tab. 2]. Swapping the judge can move absolute scores. |

## Implementation sketch

The sketch combines the group rule, propagation and gating. It is our
restatement of the rule-based paper's rules, and the marked points are where
the paper leaves the behaviour open. The LLM-agent variant has its own
checklist judge sketch in its card, so it is not repeated here.

```pseudocode
# Group state from module states  [arxiv:2411.09643 §4.5] [§6]
# Input:  module_states: list of OK, WARNING, ERROR, IGNORE, UNKNOWN
# Output: one state for the group

def group_state(module_states):
    # Any ERROR makes the group ERROR  [§6]
    if ERROR in module_states:
        return ERROR
    # Any WARNING makes the group WARNING  [§6]
    if WARNING in module_states:
        return WARNING
    # OK only if all modules are OK  [§4.5]
    if all(s == OK for s in module_states):
        return OK
    # IGNORE and UNKNOWN mixed with OK: not defined  [§4.5] [§6]
    raise NotImplementedError
```

```pseudocode
# Operator view: gating, then propagation  [§5.1] [§5.2] [§6]
# Input:  group_states: dict group -> state
#         parents: dict group -> groups it depends on
#         active: bool, set by the mission component
# Output: dict group -> state shown to the operator

def display_states(group_states, parents, active):
    shown = dict(group_states)
    # Gate groups that are not needed in the current mode  [§5.2]
    if not active:
        for group in ["Planning", "Execution"]:
            shown[group] = IGNORE
    # A dependent of a group in ERROR is shown as IGNORE  [§6] [§5.1]
    # Whether this is transitive is not stated  [§6]
    for group in group_states:
        if any(group_states[p] == ERROR for p in parents[group]):
            shown[group] = IGNORE
    return shown
```

Points the rule-based paper leaves open: transitive propagation, the response
to a vital WARNING [arxiv:2411.09643 §5.4], the transition conditions of the
driving-state machine [arxiv:2411.09643 §5.2], and the thresholds and sliding
windows for each module kind [arxiv:2411.09643 §6]. The LLM-agent paper leaves
its agent prompts and cluster hardware unstated [arxiv:2605.07161 §2.2].

## Contradictions

- **Group rule versus evaluation (rule-based source).** The paper's general
  section states that a group is OK only when every submodule is OK
  [arxiv:2411.09643 §4.5]. The evaluation aggregates with OR over module
  outputs, so a group is ERROR if any module is ERROR and WARNING if any module
  is WARNING [arxiv:2411.09643 §6]. Both readings agree for OK, WARNING and
  ERROR. Neither says how IGNORE or UNKNOWN submodules combine. A literal reading
  of §4.5 would make an IGNORE submodule non-OK, although §4.3 defines IGNORE as
  "irrelevant for the current diagnosis" [arxiv:2411.09643 §4.3].
- **Not comparable (both sources).** The two papers do not disagree on any
  number, because they measure different things. The rule-based paper reports
  no numbers [arxiv:2411.09643 §6]. The agent benchmark reports success rates
  on injected faults in Kubernetes [arxiv:2605.07161 tab. 3]. Do not read one
  paper's results as a measure of the other's approach.

## Papers

- [Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems](/papers/2411.09643): a modular diagnosis concept for autonomous shuttle buses, with five diagnostic states, group aggregation, dependency-based propagation and driving-state gating. Qualitative only.
- [SREGym](/papers/2605.07161): a live Kubernetes benchmark of 90 fault problems for LLM SRE agents, with a checklist judge for diagnosis and state checks for mitigation. Author-reported results; MIT-licensed code.