SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

In plain words

When a production service breaks, someone has to work out what went wrong and then fix it. AI agents are increasingly asked to do that job. Most existing tests for them are too tidy: they break one thing in an otherwise quiet system, or they only ask for a written answer about a frozen snapshot of logs.

SREGym runs a small but real cloud system on Kubernetes and breaks it on purpose. The breakage can be a wrong setting, a failing disk, an operator mistake or an overload, and it can happen alongside harmless background glitches. The agent gets the tools a human on-call engineer would use: metrics, logs, traces and a command line for the cluster. It has to say what is wrong and then fix it. Two separate checks grade the work. A written diagnosis is compared with the known cause through a checklist of yes/no questions, and the fix is checked against the health of the live system rather than against whether an alert has gone quiet.

The authors ran several coding agents and an SRE agent on the problems. They report that agents handle simple application faults well, do noticeably worse on low-level, compound and slowly building failures, and get worse at diagnosis when background noise is present. The code is open source under the MIT license.

TL;DR

• SREGym runs applications on Kubernetes, injects faults and background noise, and grades diagnosis and mitigation separately §2.
• It ships 90 problems, built from 47 fault primitives over 139 deployable services, with 3,623 viable fault-target pairs before noise or multi-fault composition §2.4 app. C.
• The diagnosis judge answers 9 yes/no questions in 3 dimensions and passes a diagnosis when the weighted score reaches 7/9 in the paper; the released code uses 0.70 §2.5 app. A rca_checklists.yaml.
• Author-reported, 3 runs per problem: diagnosis 38.1-72.6%, mitigation 40.4-78.5%, end-to-end 26.7-60.7% across four agent-model pairs tab. 3.
• All numbers are author-reported. Read the Paper vs code section before reusing the judge.

Problem

Existing SRE benchmarks fall short in several ways §1:

• Static question-answering sets test domain knowledge only.
• Root-cause analysis benchmarks run on static data and do not check whether the failure can be resolved.
• AIOpsLab and ITBench use live systems, but the authors consider their tasks oversimplified: mostly application-layer faults, mostly one fault in an otherwise clean environment, and bespoke designs that are hard to extend §1.
• Exact-match diagnosis labels are brittle, because agents often give a correct answer that the label does not match §2.5.

The paper assumes that realistic evaluation needs a live system, not a snapshot, because diagnosis and mitigation depend on how the system reacts to actions app. B. Its noise model is a simplification of production noise app. H.

Method

SREGym is a Python framework that deploys a problem into a Kubernetes cluster, runs an agent against it, and grades two submissions. A problem is a tuple of environment, agent interface, faults and noises, and oracles §2.1.

Environment. Applications are deployed with Helm and run in containers. The catalogue includes DeathStarBench, Train Ticket, the OpenTelemetry Astronomy Shop, and two applications the authors wrote themselves, one that simulates satellite orbits and one for air travel reservations. They are paired with data systems such as TiDB, MongoDB, Kafka and MySQL, which are managed by Kubernetes operators §2.2.

Agent interface. The framework makes no assumption about agent architecture. It exposes MCP servers for Prometheus metrics, Loki logs, Jaeger traces and a Kubernetes wrapper around kubectl, plus a submission endpoint. Raw observability and Kubernetes APIs are also reachable for agents that bring their own tools §2.3.

Submissions. The agent makes two submissions. The first is a natural-language diagnosis, which triggers the diagnosis oracle. The second signals that mitigation is finished, which triggers the mitigation oracle §2.1.

Faults. Table 1 lists the injectors. They cover killing processes and pods, stressing hardware, failing system calls through eBPF, corrupting disk sectors, misconfiguring services, Kubernetes objects and operators, buggy application code, client overload, and noise such as restarts of unrelated pods, latency or packet drops, and node-level stress tab. 1 §2.4. A problem can compose several faults and noises. Noises are transient, self-recovering disturbances that are not root causes, and the agent must tell them apart from the real fault §2.4.

Failure modes. The paper highlights three modes that its problem set covers. Metastable failures are self-sustaining collapses that persist after the trigger is removed. Concurrent failures are independent faults that happen together. Correlated failures share a common cause or dependency §2.4 fig. 4.

Diagnosis oracle. The oracle gives an evaluator the ground-truth root cause, the agent's diagnosis and a checklist of N = 9 questions in K = 3 dimensions: fault localisation, fault characterisation and failure scope. Each question gets a yes or no answer with evidence and a confidence. The dimension score is the fraction of yes answers. The aggregate is the equal-weighted sum of the three dimension scores. The verdict is a pass when the aggregate reaches the threshold §2.5 app. A.

# Diagnosis oracle: checklist LLM judge  [§2.5] [app. A]
# Input:  root_cause (text), diagnosis (text), checklist (9 questions)
# Output: verdict (bool), composite (float in [0, 1])

def judge_diagnosis(root_cause, diagnosis, checklist):
    # One yes/no judgment per question  [§2.5]
    answers = {}
    for question in checklist.questions:
        answers[question.id] = ask_evaluator(
            root_cause, diagnosis, question)  # 1 = yes, 0 = no
    # Per-dimension score: fraction of yes answers  [§2.5] (s_k)
    scores = []
    for dimension in checklist.dimensions:
        yes_count = sum(answers[q] for q in dimension.question_ids)
        scores.append(yes_count / len(dimension.question_ids))
    # Aggregate: weighted sum of dimension scores  [§2.5] (S, w_k = 1/3)
    composite = sum(w * s for w, s in zip(checklist.weights, scores))
    # Pass when the aggregate reaches the threshold  [§2.5] (tau = 7/9)
    verdict = composite >= checklist.threshold
    return verdict, composite

Mitigation oracle. Each problem defines its own check of system state. The check asks whether the injected fault is gone and whether the target system is healthy again. It uses client-side signals, such as the user request success rate, and system-side signals from processes and the cluster. The paper argues that state-based checks are harder to game than alert-based checks §2.5 app. B.

Noise schedule. The framework, not the problem, injects noise. During the noise condition, two noise patterns run every five minutes, each lasting two minutes §3.

# Problem run  [§2.1] [§2.3] [§3]
# Input:  problem (environment, faults, oracles), agent, noise_enabled
# Output: diagnosis verdict, mitigation verdict, end-to-end verdict

def run_problem(problem, agent, noise_enabled):
    # Deploy applications, data systems and operators  [§2.2]
    cluster = deploy(problem.environment)
    # Inject the target faults; they stay active during the run  [§2.4]
    inject(cluster, problem.faults)
    if noise_enabled:
        # Noise comes from the framework, not from the problem  [§3]
        start_noise_schedule(cluster)
    # Agent works through MCP tools and kubectl, then submits  [§2.3]
    # Run time is capped at 1800 s  [tab. 3]
    submissions = agent.solve(cluster, timeout_seconds=1800)
    # Grade the diagnosis with the LLM checklist  [§2.5]
    diagnosis_verdict, _ = judge_diagnosis(problem.root_cause,
                                           submissions.diagnosis)
    # Grade the fix from live system state  [§2.5]
    mitigation_verdict = problem.mitigation_oracle(cluster)
    # End-to-end success needs both verdicts in the same run  [§3]
    end_to_end = diagnosis_verdict and mitigation_verdict
    return diagnosis_verdict, mitigation_verdict, end_to_end

Judge validation. A domain expert labelled a stratified sample of 100 agent diagnoses. Claude Sonnet-4.6, the default judge, agrees with the expert at Cohen's κ = 0.90. GPT-5.4 and Kimi K2.5 reach κ = 0.70 and 0.76 against the same labels, and they reach κ = 0.94 against each other §2.5 tab. 2.

Hyperparameters

Results

Author-reported. Setup: 90 problems, 3 runs per problem, four agent-model pairs run with noise off and on, diagnosis judged by Claude Sonnet-4.6 §3.

Table 3 (overall results). TTD and TTM are capped at the 1800 s agent time cap. Diag, Mitig and E2E are success rates in percent tab. 3.

Table 4 (end-to-end success by problem group, from the paper's partition). Ported problems come from AIOpsLab and ITBench (n = 34). Similar Failures share a fault family with those problems but use different applications or components (n = 43). New Failures are fault families unique to SREGym (n = 13) tab. 4.

Table 5 (conditional mitigation probability given diagnosis outcome) tab. 5:

Findings reported in the text:

• Claude Code has the highest end-to-end rate without noise (60.7%). Stratus with Sonnet-4.6 has the highest mitigation rate (78.5%), which the authors attribute to its undo-and-retry mechanism §3.1.
• Claude Code uses 1.81x the tokens of Stratus with the same model, and Codex uses at least 2.44x §3.1 tab. 3.
• Diagnosis drops for every agent-model pair when noise is on. Mitigation is less affected §3.1.
• Agents tend to treat the first plausible anomaly as the target, which can be a noise event §3.1.
• On the latent sector error problem, no run of Stratus or Claude Code scored above 0.22 on the diagnosis checklist, and fault characterisation scored 0 in every run. Agents blamed application error handling and proposed no disk-level checks §3.2.
• On metastable problems, no agent identified both interacting components. Codex found the resource constraint once but dismissed the application trigger §3.2.
• Given a correct diagnosis, mitigation succeeds 68% to 89% of the time. Given an incorrect diagnosis, it succeeds 22% to 62% of the time §3.3 tab. 5.
• Most tool calls are read-only kubectl calls (60% to 72%). kubectl get and kubectl logs make up about 87% of reads §3.4.
• Ported problems are easy for the strongest agent: Stratus with Sonnet-4.6 reaches 83.3% mitigation on them tab. 4.

Paper vs code

Code: github.com/SREGym/SREGym at commit c4dbbbf72d4123bc5c5a0df677bf50f0bb56dd81, MIT license LICENSE.txt. The paper links this repository from its introduction §1.

• Pass threshold. The paper sets tau = 7/9, about 0.778 §2.5. The code's checklist file sets threshold 0.70 rca_checklists.yaml, and the Python fallback is also 0.70 judge.py. The two thresholds differ in outcome for some score patterns. For example, scores of (1, 1, 1/3) on the three dimensions give 0.773. That passes in the code and fails under the paper's 7/9. The paper does not say which threshold produced Table 3.
• Weights. The paper uses 1/3 for all dimensions §2.5. The code uses 0.33, 0.33 and 0.34 rca_checklists.yaml. This is minor but changes the score in the third decimal place.
• Checklist wording. The code's D2-Q1 adds "probe misconfiguration" to the list of mechanisms rca_checklists.yaml. The paper's appendix table omits it app. A.
• Metastable trigger. The paper describes the trigger as a transient CPU stress §2.4. The problem code also injects 100 ms of network latency alongside the CPU stress, at 60 s into the run capacity_decrease_rpc_retry_storm.py. The 50 ms timeout and retry count of 30 match the paper capacity_decrease_rpc_retry_storm.py.
• Noise. The paper says two patterns every five minutes, each lasting two minutes §3. The code runs up to two experiments per cycle, 120 s each, with a 300 s cooldown between cycles manager.py. The start-versus-end timing of the cooldown is not verified.
• Agent time cap. The paper uses 1800 s tab. 3. The code's per-problem recommended timeout for the coordination application is 2700 s application.py. The conductor's agent-timeout default is None conductor.py. I did not find where the 1800 s cap is set in code.
• Judge settings. The paper does not state the judge's temperature or output budget. The code uses temperature 0.0 and up to 16384 output tokens by default judge.py judge.py.
• Not checked. I did not count the problems in the repository, read the per-problem mitigation oracles, or run the benchmark.

Pitfalls

• Judge threshold. The paper's 7/9 and the code's 0.70 differ on some score patterns, as shown above. Pick one and say which one you used.
• Judge dependence. Only Sonnet-4.6 agrees strongly with the human expert (κ = 0.90). GPT-5.4 (κ = 0.70) and Kimi K2.5 (κ = 0.76) agree less. Swapping the judge can move absolute diagnosis scores. The validation uses one expert and 100 samples §2.5 tab. 2.
• Small samples. Each cell is 3 runs per problem. The text reports no confidence intervals. Differences of a few points, such as Stratus with Kimi diagnosis moving from 40.4% to 38.1% with noise, may be noise tab. 3.
• Group sizes. The New Failures group has 13 problems, so it has 39 runs per configuration. Each step in its percentages is about 2.6 points tab. 4.
• A non-monotone cell. The paper says diagnosis drops for every pair under noise §3.1. This holds in Table 3, but not in every group. Claude Code's end-to-end rate on New Failures rises from 28.2% to 48.7% with noise tab. 4. The text does not discuss this.
• The 40-point figure. The abstract says end-to-end results differ by up to 40 points. The overall spread in Table 3 is 34.0 points (26.7% to 60.7%). The Similar Failures column of Table 4 spans 30.2% to 70.5%, which is 40.3 points. The paper does not say which comparison it meant.
• Noise reproduction. Noise is random, and the paper gives no seed. To rebuild a noise schedule, use the code's constants, not the prose manager.py.
• Mitigation oracles are per problem. The paper describes them in general terms only. Re-implementing one means reading its code.
• Node-level faults. Some injectors act below Kubernetes, for example eBPF syscall failure, disk sector corruption and hardware stress tab. 1. Running them needs node-level access. The paper does not state what access is required.
• Cost and time. One run uses between 413K and 1.98M tokens (Table 3). Runs can reach the 1800 s cap, which the authors say happens to Stratus under noise app. F. Cluster hardware is not stated, so budgeting is hard to estimate.
• Scale. The largest application has 40 microservices, and the authors say results may not carry over to much larger systems app. H.

Lineage

• Builds on: AIOpsLab (MLSys'25) and ITBench (ICML'25) are the nearest live predecessors. The paper reuses their problems as its Ported group and rejects their brittle labels and their hardcoded workflow §1 §2.3 §2.5 tab. 4. Neither has an arXiv id in the paper's bibliography, so none is linked here.
• Compared against: Stratus (NeurIPS'25), Claude Code and Codex §3.
• Superseded by: not stated.