Fault diagnosis (for multi-component systems)

Overview

A system built from many parts can fail in many places, and one failed part can make others look broken. Fault diagnosis has two jobs: detect which parts are faulty, and show where the fault starts instead of listing every symptom. Two sources cover it, and they tackle the second job differently. The rule-based design in 2411.09643 §4 reports every check, and every group of checks, into one small state vocabulary (OK, WARNING, ERROR, IGNORE, UNKNOWN) 2411.09643 §4.3. Because the states share one vocabulary, checks can be added or removed without changing the rest of the system. The LLM-agent benchmark in 2605.07161 §2 instead gives an agent monitoring tools and asks it to name the root cause of a fault in a live cluster, then fix it.

Variants

The first four variants below come from the rule-based framework. The fifth is a different kind of method. The two sources are not compared against each other, and the rule-based paper does not compare against other diagnosis systems.

Module kinds (what a check looks at)

The paper lists five common kinds of diagnosis module 2411.09643 §4.4: a component's own quality estimate, message frequency or latency, a single value against a threshold, a reachability probe (watchdog), and a custom domain check. Each kind reports into the shared state vocabulary. The paper names the checks but gives no threshold, rate or timeout values, so an implementer must choose them 2411.09643 §6.

Card: Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems.

Group aggregation

Module states are collected into groups that represent subsystems, and groups can contain groups 2411.09643 §4.5 2411.09643 §5.1. The stated rule is that a group is OK only when all its submodules are OK, and otherwise WARNING or ERROR 2411.09643 §4.5. The evaluation instead reports the worst state among its modules 2411.09643 §6. See Contradictions.

Dependency-based propagation

Groups are linked by dependency edges. When a group is in ERROR, the groups that depend on it are shown as IGNORE, so the operator sees one root cause rather than many alarms 2411.09643 §5.1 2411.09643 §6. Traversing the graph from the faulty component is how the paper identifies the root cause 2411.09643 §5.1. The paper does not say whether the rule is transitive over several levels.

Driving-state gating

A state flag suppresses diagnosis of components that are not needed in the current mode. In the paper, Planning and Execution are not diagnosed outside autonomous mode 2411.09643 §5.2 2411.09643 fig. 5. The flag is driven by a mission component, and the paper's state machine has start-up, idling, driving and shut-down states 2411.09643 §5.2. The transition conditions are not defined.

LLM agents on live systems (root-cause diagnosis)

An LLM agent gets metrics, logs, traces and a kubectl wrapper, and must say what is wrong and then mitigate it 2605.07161 §2.3 §2.1. The benchmark deploys an application on Kubernetes, injects one or more faults, and optionally adds background noise 2605.07161 §2.4 §3. The diagnosis and the fix are graded separately. A checklist judge scores the written diagnosis against the known cause, and a check on live system state scores the fix 2605.07161 §2.5.

The checklist has nine yes/no questions in three dimensions: fault localisation, fault characterisation and failure scope 2605.07161 §2.5 app. A. The paper's pass threshold is 7/9, but the released code uses 0.70, and the two differ on some score patterns. The card's Paper vs code section gives an example 2605.07161 §2.5. The judge agrees with one human expert on 100 diagnoses at Cohen's κ = 0.90 for the default model, and at 0.70 and 0.76 for two others 2605.07161 §2.5 tab. 2.

Card: SREGym, a live Kubernetes benchmark for SRE agents.

When to use which

The rule-based rows rest on the paper's qualitative runs and design argument only, because it reports no accuracy, latency or detection-rate figures. The agent rows rest on author-reported benchmark numbers.

Implementation sketch

The sketch combines the group rule, propagation and gating. It is our restatement of the rule-based paper's rules, and the marked points are where the paper leaves the behaviour open. The LLM-agent variant has its own checklist judge sketch in its card, so it is not repeated here.

# Group state from module states  [arxiv:2411.09643 §4.5] [§6]
# Input:  module_states: list of OK, WARNING, ERROR, IGNORE, UNKNOWN
# Output: one state for the group

def group_state(module_states):
    # Any ERROR makes the group ERROR  [§6]
    if ERROR in module_states:
        return ERROR
    # Any WARNING makes the group WARNING  [§6]
    if WARNING in module_states:
        return WARNING
    # OK only if all modules are OK  [§4.5]
    if all(s == OK for s in module_states):
        return OK
    # IGNORE and UNKNOWN mixed with OK: not defined  [§4.5] [§6]
    raise NotImplementedError

# Operator view: gating, then propagation  [§5.1] [§5.2] [§6]
# Input:  group_states: dict group -> state
#         parents: dict group -> groups it depends on
#         active: bool, set by the mission component
# Output: dict group -> state shown to the operator

def display_states(group_states, parents, active):
    shown = dict(group_states)
    # Gate groups that are not needed in the current mode  [§5.2]
    if not active:
        for group in ["Planning", "Execution"]:
            shown[group] = IGNORE
    # A dependent of a group in ERROR is shown as IGNORE  [§6] [§5.1]
    # Whether this is transitive is not stated  [§6]
    for group in group_states:
        if any(group_states[p] == ERROR for p in parents[group]):
            shown[group] = IGNORE
    return shown

Points the rule-based paper leaves open: transitive propagation, the response to a vital WARNING 2411.09643 §5.4, the transition conditions of the driving-state machine 2411.09643 §5.2, and the thresholds and sliding windows for each module kind 2411.09643 §6. The LLM-agent paper leaves its agent prompts and cluster hardware unstated 2605.07161 §2.2.

Contradictions

• Group rule versus evaluation (rule-based source). The paper's general section states that a group is OK only when every submodule is OK 2411.09643 §4.5. The evaluation aggregates with OR over module outputs, so a group is ERROR if any module is ERROR and WARNING if any module is WARNING 2411.09643 §6. Both readings agree for OK, WARNING and ERROR. Neither says how IGNORE or UNKNOWN submodules combine. A literal reading of §4.5 would make an IGNORE submodule non-OK, although §4.3 defines IGNORE as "irrelevant for the current diagnosis" 2411.09643 §4.3.
• Not comparable (both sources). The two papers do not disagree on any number, because they measure different things. The rule-based paper reports no numbers 2411.09643 §6. The agent benchmark reports success rates on injected faults in Kubernetes 2605.07161 tab. 3. Do not read one paper's results as a measure of the other's approach.

Papers

• Modular Fault Diagnosis Framework for Complex Autonomous Driving Systems: a modular diagnosis concept for autonomous shuttle buses, with five diagnostic states, group aggregation, dependency-based propagation and driving-state gating. Qualitative only.
• SREGym: a live Kubernetes benchmark of 90 fault problems for LLM SRE agents, with a checklist judge for diagnosis and state checks for mitigation. Author-reported results; MIT-licensed code.