Paper · 2026-10-07 · cs.CR, cs.AI, cs.MA
From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage
AIDA is a multi-agent SOC alert-triage system. A Strategist gathers SIEM evidence and must commit to a thesis, a separate Adversary challenges it, and a Judge decides or asks for another round. On 1,247 replayed alerts, the author-reported F1 is 0.958 with GPT-4.1, versus 0.371 to 0.744 for five baselines; false negatives fall from 40.4% to 3.1%.
LLM agentMulti-agent systemsIntrusion detectionSelf-refinement (draft, critique, rewrite loop)alert-triagesecurity-operationsmulti-agentllm-agentsbenchmark
AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.
In plain words
Security teams receive many alerts from their detection tools, and in the authors' sample most are benign. Closing a real attack by mistake is the costly error, because it lets the attack go unnoticed. The paper studies language-model agents that search the logs before deciding. It finds that common ways of doing this often miss attacks. The usual failure is that a search comes back empty and the model closes the alert anyway. Asking the model to re-read its own conclusion did not reliably fix this either.
The proposed system, AIDA, splits the work across roles. One agent searches the logs and must state a clear proposed verdict with its evidence before anyone challenges it. A second agent sees the verdict and the evidence, but not the first agent's private reasoning, and looks for gaps. A third agent, the judge, makes the call. It needs real evidence before it closes an alert, and it can ask for another round of searching. Every step is written to a shared log that is never overwritten, so the record of how a decision was reached is kept.
TL;DR
- On ALERT-BENCH, which replays 1,247 alerts from one AIT-LDSv2 scenario through a Wazuh SIEM, five LLM reasoning approaches reach F1 0.371 to 0.744, and each misses at least 40.4% of attack alerts §IV-B.
- Attack alerts are most often dismissed when the investigation's searches return no records, and same-context self-review breaks more correct decisions than it repairs §IV-C.
- AIDA separates the roles: a Strategist commits a thesis, a separate Adversary challenges it from the evidence alone, and a Judge decides or requests another round, using an append-only Investigation Ledger §V.
- Author-reported with GPT-4.1: F1 0.958 (precision 0.948, recall 0.969), false-negative rate 3.1%, 18.4% of alerts escalated, about 33,942 tokens and $0.109 per alert §VI-A.
- The main comparison uses a single scenario, no run-to-run variance is reported, and no official code was found §III-B §IV-A §VII.
Problem
Tier-1 triage must close most alerts while missing few attacks. Dismissing a real attack lets it go unnoticed, and escalating benign activity uses analyst time §I. The authors' sample is dominated by benign alerts: 225 attack and 1,022 benign alerts §III-B.
Earlier automation follows a fixed procedure. Log anomaly detection, contextual event correlation and provenance-based analysis build their decision from a predefined representation and do not decide what further evidence to collect when the picture is unclear §I §II-A.
The paper asks how the reasoning structure of a tool-using LLM agent affects triage quality, and what it costs §I §IV-A. Its evaluation assumes that each investigation starts from one detector alert, with no preselected evidence, and that the ground-truth label is hidden from the system §III.
Method
Evaluation environment: ALERT-BENCH §III
- Replay the AIT-LDSv2 host and network logs into a Wazuh SIEM in timestamp order. Suricata alerts are ingested together with host telemetry. Original timestamps and host identities are kept §III-A.
- Label each alert with the event-matching procedure published with AIT-ADS. Wazuh alerts are matched to annotated events, Suricata alerts by flow, and multi-record audit alerts by their own event matching. If an event has several attack labels, a fixed priority order picks one phase §III-A.
- Alerts that match an attack annotation are attack-related. All other retained alerts are benign. Alerts from unrelated activity on the SIEM host are excluded, using source and host fields rather than the label §III-A.
- Each system receives one alert and an MCP interface with four operations and one host-inventory lookup: alert retrieval by rule, host and time window; entity search for an IP address or user name; alert counts by field; and pattern analysis for spikes and rare combinations. The alert timestamp anchors the time windows. Labels are not exposed §III-C.
- Each investigation ends with an escalate or dismiss decision and a justification. Scoring treats attack alerts as the positive class §III-C §III-D.
Scoring metrics are precision, recall, F1, accuracy and confusion counts. The false-negative rate is FN / (TP + FN). Investigation Load is (TP + FP) / total, the share of alerts sent to analysts. Negative predictive value is TN / (TN + FN) §III-D.
The evaluation sample is a stratified, seeded sample of 1,247 alerts from the russellmitchell scenario, which the authors chose because its detector output is benign-dominated and it has the smallest published alert volume of the eight AIT-ADS scenarios §III-B. Attack alerts split into 126 directory enumeration, 94 WordPress scanning, 3 service scanning and 2 sudo privilege escalation. Most benign alerts are successful Dovecot logins (433), with Suricata TLS handshakes (273) and generic IDS events (275) next tab. I §III-B.
Five baseline approaches §IV-A app. A
All five baselines get the same alerts, model (GPT-4.1), tool definitions and interface.
- Direct-Prompt: one pass of tool use, then a decision.
- ReAct: a thought, action and observation loop.
- Self-Consistency: three ReAct trajectories with majority vote.
- Self-Refine: an investigation, then self-critique and revision in the same context.
- CoVe: an investigation, then generated verification questions and a revision.
AIDA: Dialectical Commitment Protocol §V
The design follows three findings from the baseline study: commit to a checkable claim before review, review it from a context that does not carry the reasoning, and demand more evidence before closing an alert than before escalating it §V.A.
Each round has three phases: investigation, challenge and synthesis. The shared Investigation Ledger records the alert, retrieved evidence, theses, challenges, continuation directives and the verdict in order. Entries are added but never overwritten §V.B fig. 5.
# Dialectical Commitment Protocol (AIDA) [alg. 1]
# Input: alert, max rounds T, per-round tool budget B
# Output: verdict with evidence-grounded justification
def triage(alert, max_rounds, budget):
# Start the append-only ledger with the alert [§V.B] [alg. 1]
ledger = [alert]
for round_index in range(1, max_rounds + 1):
# Investigation: the strategist requests evidence until it
# commits to a thesis or the budget is spent [§V.B] [alg. 1]
thesis = None
calls_used = 0
while thesis is None and calls_used < budget:
step = strategist(ledger)
if is_thesis(step):
thesis = step
else:
# The executor runs the request and appends a typed
# record: success with rows, empty, or failed [§V.C.2]
ledger.append(executor(step))
calls_used += 1
# Forced Commitment Gate: commit from the evidence already
# collected if the budget ended first [§V.C.1] [alg. 1]
if thesis is None:
thesis = strategist(ledger, commit_only=True)
ledger.append(thesis)
# Challenge: the adversary sees the thesis and the evidence,
# but not the strategist's private reasoning [§V.C.3]
challenge = adversary(thesis, evidence_view(ledger))
ledger.append(challenge)
# Synthesis: the judge decides, or asks for another round
# [§V.C.4] [alg. 1]
decision = judge(
thesis, challenge, ledger, round_index, max_rounds
)
ledger.append(decision)
if is_verdict(decision):
return decision
# A continuation resets the tool budget and keeps the ledger
# [§V.C]
# Not reached: the judge must return a verdict in the last round
# [alg. 1] [§V.C.4]Roles and their inputs §V.C:
- Strategist (LLM). On every turn it outputs a retrieval request or a thesis. A thesis names True Positive or False Positive, cites ledger entries and gives a confidence in [0, 1]. It may keep retrieving until it commits or the per-round budget B is spent. After that, its next turn must be a thesis. It sees the newest retrieval in full, compact summaries of older evidence, and the previous thesis, challenge and continuation directive when present. An empty result is recorded as an unresolved gap unless it establishes a benign explanation §V.C.1.
- Executor (no LLM). It is the only component that calls the SIEM. It validates arguments, runs the operation, and retries transient failures. Each result becomes a typed ledger entry: a successful retrieval with the matching records compacted to needed fields, an empty retrieval, or a failed call. It does not interpret the alert §V.C.2.
- Adversary (LLM). It sees the alert, the thesis and its confidence, and the retrieved evidence, but not the Strategist's reasoning. It first checks each factual claim in the thesis against the ledger. Any claim the ledger does not back is written into the challenge rather than counted as evidence. It can file two kinds of challenge: one points at ledger entries that clash with the classification; the other lists evidence still absent that might flip the outcome. It reports a confidence in [0, 1], which the judge receives as the challenge strength. It does not retrieve evidence itself §V.C.3.
- Judge (LLM). It sees the thesis and confidence, the challenge and strength, the relevant evidence, and the round index with the round limit. It either closes the alert with a classification and a justification grounded in the evidence, or issues a continuation directive naming the evidence needed next. Escalation needs concrete evidence of malicious activity. Closing an alert needs positive proof of benign activity, or an investigation thorough enough to support closure. The confidence and strength values are not decision thresholds. In the last round the judge must decide §V.C.4 alg. 1.
Reference configuration: per-round budget B = 5, maximum T = 3 rounds. The budget resets after each continuation, and the ledger persists §V.C.
Development set: 170 russellmitchell alerts, disjoint from the evaluation sample, used to calibrate role instructions and check output schemas. The authors froze the prompts and configuration before running the evaluation §V.C.
Walkthrough §V.D: a benign Dovecot login alert with seven successful logins for one account in three minutes. The Strategist commits to True Positive with confidence 0.95. The Adversary notes that the ledger has no failed login and no external source, and records a challenge with strength 0.85. The Judge decides False Positive with confidence 0.90 in round 1.
Agent system prompts are reproduced in app. B. This card paraphrases them; see Pitfalls for where they differ from the protocol text.
Hyperparameters
| Name | Value | Where | Notes |
|---|---|---|---|
| Evaluation sample | 1,247 alerts: 225 attack, 1,022 benign | §III-B tab. I | stratified, seeded; seed value not stated |
| Ablation and model subset | 300 alerts, 54 attack | §VI-C §VI-D | drawn from the evaluation sample |
| Dev set | 170 russellmitchell alerts | §V.C | disjoint from evaluation |
| Reference model | GPT-4.1 (gpt-4.1-2025-04-14) | §IV-A §VI | one model for the end-to-end comparison |
| AIDA per-round tool budget B | 5 | §V.C | resets after continuation |
| AIDA maximum rounds T | 3 | §V.C alg. 1 | judge must decide in round T |
| AIDA temperature | not stated | — | the temperature 0.2 is for baselines only |
| Executor retry count | not stated | §V.C.2 | "retries transient failures" only |
| Baseline tool caps | Direct-Prompt 3; ReAct 5; Self-Refine 7; CoVe 10; Self-Consistency 3 trajectories of 5 calls | §IV-A | average use stays below the cap |
| Baseline temperature | 0.2; Self-Consistency 0.7 | §IV-A | |
| Confidence scales | Strategist confidence and Adversary strength in [0, 1] | §V.C.1 §V.C.3 | not thresholds in the protocol; see Pitfalls |
| Statistics | Wilson 95% CIs; McNemar exact test; stratified paired bootstrap; fixed seed | §IV-A §VI-A | |
| Hardware | Ubuntu 20.04; 2 × AMD EPYC 7313; 1.0 TiB RAM; 1 × NVIDIA A100 80 GB | §IV-A | hosts Wazuh, MCP service and harness |
| Open-weight models | Llama 3.1 8B and 70B; GPT-OSS 120B; Gemma4 31B; Qwen3.5 9B | tab. III | serving setup and quantisation not stated |
Results
Author-reported; this card has not re-run any experiment. The headline comparison is on the 1,247-alert sample with GPT-4.1 §IV-A §VI-A.
| Setting | Metric | Baseline | AIDA | Source |
|---|---|---|---|---|
| Full sample, GPT-4.1 | F1 | 0.744 (Direct-Prompt, best baseline) | 0.958 | §VI-A §I |
| Full sample | Precision / recall | 0.596 recall (Direct-Prompt) | 0.948 / 0.969 | §VI-A |
| Full sample | False-negative rate | 40.4% (best baseline); up to 70.2% for the worst | 3.1% | §I §VI-A |
| Full sample | Investigation Load | 8.5% to 14.0% | 18.4% | §I §VI-A |
| Full sample | Tokens / cost / latency per alert | 5,075 to 33,438 tokens; $0.016 to $0.107 | 33,942 tokens; $0.109; 12.7 s | §VI-A |
| Full sample | Tool calls per alert | caps of 3 to 10 per investigation | 4.3 average | §IV-A §VI-A |
| Full sample | F1 95% CI; vs Direct-Prompt F1 gain | — | [0.939, 0.976]; +0.214, CI [0.165, 0.269] | §VI-A |
| 300-alert subset | F1 (P / R) | — | 0.952 (0.980 / 0.926) | tab. II |
| Cross-scenario (wardbeck, 337 alerts) | F1 (P / R) | — | 0.927 (0.905 / 0.950); model not stated | §VI-D |
Baseline details reported in the text §IV-B §IV-C:
- Direct-Prompt: F1 0.744, recall 0.596, 5,075 tokens per alert, the fewest tokens in the study.
- Self-Consistency: recall 0.458, 8.5% Investigation Load, 54.2% false negatives, 28,002 tokens.
- CoVe: F1 0.371 (lowest), 33,438 tokens.
- Self-Refine: F1 0.665 before review and 0.550 after review.
- CoVe: F1 0.669 before review and 0.371 after review.
- The F1 range 0.371 to 0.744 covers the five approaches. The text does not name the approach at 0.625, and the other per-approach values are only in fig. 2 §IV-B.
Same-context review changed 95 Self-Refine decisions (16 repaired, 79 broken) and 157 CoVe decisions (25 repaired, 132 broken) §IV-C.
Ablations on the 300-alert subset, GPT-4.1 tab. II:
| Configuration | Precision | Recall | F1 | Tokens per alert |
|---|---|---|---|---|
| Only-Strategist | 0.909 | 0.926 | 0.917 | 6,737 |
| No-Judge (Strategist + Adversary) | 0.850 | 0.944 | 0.895 | 14,589 |
| No-Adversary (Strategist + Judge) | 0.929 | 0.963 | 0.946 | 15,383 |
| Full AIDA | 0.980 | 0.926 | 0.952 | 36,988 |
The paired 95% CI for the F1 gap between Full AIDA and No-Judge is [0.008, 0.131]. The gaps to No-Adversary (+0.006) and Only-Strategist (+0.035) have intervals that include zero §VI-C.
Five open-weight models, full protocol, 300-alert subset tab. III:
| Model | Precision | Recall | F1 |
|---|---|---|---|
| Llama 3.1 8B | 0.104 | 0.093 | 0.098 |
| Llama 3.1 70B | 0.578 | 0.759 | 0.656 |
| GPT-OSS 120B | 0.868 | 0.611 | 0.717 |
| Gemma4 31B | 0.635 | 1.000 | 0.777 |
| Qwen3.5 9B | 0.980 | 0.926 | 0.952 |
Trace-level results §VI-B:
- Attack investigations with at least one empty result: 97 to 208 of 225 for each baseline, but only 10 for AIDA. Six of the ten were escalated.
- AIDA overturned 31 of 1,247 committed theses, all of them escalations of benign alerts, and none of its correct theses. Challenge strength was at most 0.2 on the 1,216 upheld investigations and at least 0.8 on the 31 overturned ones.
- With GPT-4.1, every investigation ended in round 1.
- Before dismissal AIDA issued 4.79 tool calls and before escalation 2.27 (ratio 2.11, p = 1.5e-66).
Paper vs code
No official code. The AIDA and ALERT-BENCH code was not found: the LaTeX source links only third-party tools and resources (Wazuh, Suricata, MCP, the AIT-ADS alert data set and its labeling code), the arXiv abstract page lists no code or project page, and GitHub repository searches for "ALERT-BENCH", "AIDA alert", "dialectical commitment protocol", "investigation ledger alert" and "alert-triage SIEM wazuh agent" returned no repository owned by an author or their lab. Some search results were truncated, and third-party projects with similar names were not used. Discrepancies between the protocol text and the prompts in app. B are listed under Pitfalls.
Pitfalls
- One scenario, no variance. The main comparison uses one of eight AIT-LDSv2 scenarios (russellmitchell). The cross-scenario check uses one more scenario (wardbeck). No run-to-run variance is reported for any configuration §III-B §VI-D. The authors have not used independently collected SOC telemetry and list it as a limit §VII.
- Different sets for different numbers. The headline F1 of 0.958 is on the 1,247-alert sample. The ablation and model results (0.952, token counts 36,988) are on the 300-alert subset. The token count on the full sample is 33,942. Do not mix them in a table.
- Budgets are not matched. Baselines are capped at 3 to 10 tool calls per investigation. AIDA's per-round budget is 5 and it runs up to 3 rounds, so it may issue up to 15 calls. Its measured average is 4.3 calls per alert. The cost comparison therefore compares different tool budgets, not equal ones §IV-A §V.C §VI-A.
- Multi-round behaviour is barely exercised in the headline run. With GPT-4.1, every investigation concluded in round 1, so the headline result does not test the continuation loop. Continuation was studied only with other models §VI-B §VI-D.
- Prompt text versus protocol text.
- Dismissal rule. The protocol text accepts positive proof of benign activity, or an investigation thorough enough to support closure, as grounds for dismissal §V.A §V.C.4. The Judge prompt allows dismissal on benign explanation or on a payload that shows no attacker-tooling, reconnaissance or anomalous indicator, and it says that absent corroborating threat events are not enough by themselves app. B. The second rule is in the prompt but not in the text.
- The Strategist prompt tells the model to prefer True Positive when the payload carries attacker-tool or exfiltration signatures even with sparse corroboration app. B. This rule does not appear in the protocol section.
- Confidence. The text says confidence and strength are not decision thresholds §V.C.4. The Judge prompt gives numeric guidance: a challenge below 0.4 means the reviewer largely agrees, and confidence of 0.8 or more is reserved for directly supported verdicts app. B. Whether the Judge treats these as thresholds in practice is not measured.
- Challenge types. The text names contradiction and completeness challenges §V.C.3. The Adversary prompt describes the distinction but does not name the two types app. B.
- Dismissal rules are LLM-enforced. The asymmetric evidence requirement is applied by the Judge model, guided by its prompt. The paper describes no programmatic check on the evidence before closure §V.C.4. Behaviour may change with the model; the open-weight comparison shows large differences tab. III.
- Compaction is underspecified. The Strategist sees "compact summaries of earlier evidence". The paper does not say who writes these summaries, how they are produced, or how much they lose §V.C.1.
- Executor retries are unspecified. The number of retries and the backoff are not stated §V.C.2.
- AIDA's temperature is not stated. The stated temperatures (0.2, and 0.7 for Self-Consistency) apply only to the baselines §IV-A.
- Labels are a convention. An alert with no matching annotation is treated as benign, so an attack alert that the matching procedure fails to link would be labelled benign and, if escalated, counted as a false positive. Three of the 225 attack alerts have an identical benign log line with a different label, two of them among the 26 missed by all baselines §III-A §IV-D.
- Read FNR and Investigation Load together. A system that dismisses everything has zero load and misses every attack §III-D.
- Model dependence. Only Qwen3.5 9B matched AIDA on the 300-alert subset. Llama 3.1 8B reached F1 0.098. The authors note that effectiveness depends on the model running the protocol §VI-D. The subset is drawn from the evaluation sample, not the dev set, and the paper reports no selection step for the five models.
- Ablation uncertainty. Only the Judge's removal gives a CI that excludes zero. The paper does not isolate a significant effect of the Adversary alone §VI-C.
- Model versions. The GPT-4.1 snapshot is named. The versions of the open-weight models and their serving stack are not stated.
- Cost figures. The $0.109 per alert is described as approximate, and the pricing source is not stated §VI-A.
- Mapping to real SOCs. Alerts come from replayed telemetry in a lab with a Wazuh rule set. Organisation context such as asset roles and change records is not in the evidence interface, and the authors name it as a limit §VII.
Lineage
- Compared against: Direct-Prompt, ReAct, Self-Consistency, Self-Refine and CoVe, each implemented on the same interface §IV-A app. A.
- Draws on evidence-based debate and on tool-based correction from external feedback, which the paper cites for the separate challenge and the evidence-driven revision §V.A. The negative effect of same-context review is the authors' own measurement §IV-C.
- Related benchmarks named by the paper: CyberSOCEval, ExCyTIn-Bench, SIABench and Cybench §II-C §I.
- Superseded by: not stated in the paper.
Source and license
Adapted from “From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage” by Saimon Amanuel Tsegai, Alex Kantchelian, Danfeng Yao, Peng Gao (arXiv:2610.10608), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors.