wikillm

Paper · 2026-10-07 · cs.AI, cs.CL

LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets

Five frontier LLM agents trade live crypto and US equity paper accounts for about 30 days, each in a Base ReAct setup or with one of four add-ons (tool routing, memory, rules, multi-agent). Capability scores come from decision traces. Author-reported: return rankings reverse 19 to 29 times, capability rankings 0 to 8 times.

LLM agentLLM evaluationAgent memoryTool useagent-evaluationfinancial-tradingbenchmarkmulti-agentrule-following

arXivPDFCodeMarkdown14 min read

AI-compiled by claude-haiku-5-5 on 2026-11-21, not yet reviewed by a human. Every claim cites its source; check it when a detail matters.

In plain words

Testing an AI agent by its final score can hide how it got there. Two agents can end with the same profit, one by careful research and the other by luck. This paper tests that idea with a trading game played on real, moving markets. Each AI model is given a paper trading account and runs for about a month, making a decision every four hours.

Each model is also run in several setups. A plain setup uses only basic market tools. Other setups add a large catalogue of extra tools, a memory it can write notes to and reread, a written set of trading rules it has to follow and explain, or a team of specialist sub-agents under a manager. The authors look at how each model uses these extras, not only at the profit it makes.

The main finding is that profit rankings shift as the month goes on, while the capability scores stay steadier. A model can be a strong trader and still be weak at, say, following the rules or making use of its memory. The authors present the live market as a way to diagnose agents, not only to rank them.

TL;DR

LiveMACEBench evaluates five frontier LLM agents in a persistent paper-trading setup on live crypto and US equity data over about 30 days §3. Each agent runs a ReAct-style decision round every four hours, with an optional add-on for tool routing, memory, rules or multi-agent collaboration §2.1 §2.2. The authors score each capability from the full decision trace rather than from returns alone §2.3. Author-reported: cumulative-return rankings reverse 19 to 29 times over the window, versus 0 to 8 times for capability-specific rankings §3.1.

Problem

Outcome measures such as account return mix the agent's skill with market conditions, so they do not show which capability produced a result §1. The authors argue this matters more when the environment changes over time and the agent's earlier actions keep affecting later ones §1. A static benchmark does not capture this, so the paper uses a live, persistent market as the testbed §1 §2.1.

Method

Environment. The agent trades six cryptocurrencies and seventeen US equities, a fixed universe of 23 assets app. B.2. Equities come from Alpaca's consolidated SIP feed and crypto from Hyperliquid §3. Each agent keeps its own account, which is never reset §2.1. Decisions happen every four hours on a shared time grid §2.1.

Decision round. One round is one complete agent invocation. Inside a round, the agent can reason, call tools and place trades several times, and the round ends on a termination token or when its budget runs out app. B.3.1. Placing a trade does not end the round by itself app. B.3.1. Positions and cash carry over to the next round §2.1.

Decision trace. The runtime records, in order, every model response, tool call, tool output and trade action within a round §2.1 app. B.3.1. All capability scores are computed from these traces §2.3.

Base configuration. A ReAct loop over a fixed set of ten tools: market snapshot, candle history, account state, decision history, a news search agent, a sandboxed shell, file read and write, a sandboxed Python runner and trade execution app. B.3.1.

Tool use. The agent can reach 295 extra tools taken from a public API snapshot, which are executed locally; with the ten base tools the candidate pool is 305 app. B.3.2. Five core tools always stay visible app. B.3.2. A meta-tool, select_tools, returns a subset of candidates for the current information need, and the router output is validated before use app. B.3.2.

Persistent memory. Two tools, memory_search and memory_add, store and retrieve reusable experience app. B.3.3. Each entry has the form condition, observation, rule, and carries a market tag app. B.3.3 §2.2. Retrieval uses only same-market entries and ranks them by a weighted sum of embedding similarity and recency app. B.3.3. A new entry is rejected as a near duplicate if its cosine similarity to a same-market entry is at least 0.90 app. B.3.3. The store starts empty for each evaluation app. B.3.3.

Rule following. The policy has three priority levels, R0 above R1 above R2 §2.2 app. A.1.1. R0 covers system limits (leverage, margin, intraday drawdown, single-order size). R1 covers client requirements (exclusions, per-asset concentration, minimum cash). R2 covers six soft preferences (decision stability, cash utilisation, sector affinity, scaling, fee sensitivity, active engagement) app. A.1.1. Before any trade, the agent must write a structured compliance rationale, and a HOLD decision also needs one app. B.3.4 §2.2.

Multi-agent collaboration. A manager with no market or trading tools delegates to five specialists: TradingAgent, NewsAgent, CoderAgent, AnalystAgent and CriticAgent. A separate ExecutionAgent carries out the approved plan app. B.3.5 §2.2. Each specialist answer becomes a source-attributed evidence item app. B.3.5.

Trading outcomes. Account equity is sampled at checkpoints. Returns are sampled at four granularities, from quarter-hour steps up to daily steps, and the headline metrics use the hourly series app. B.1. Overall return compounds the hourly returns app. B.1. Tail loss is the mean of the worst 5% of hourly returns app. B.1.

Capability scores. Each add-on has its own score, defined in app. B.4:

  • Tool use: four LLM-judge dimensions (relevance, invocation efficiency, coverage, faithfulness), plus routing quality, valid-call rate and a hallucination-free rate, combined into a Tool Call Score app. B.4.1 §2.3.
  • Memory: two judge dimensions (content quality and usage quality), plus change in maximum drawdown and change in tail loss against the matched memory-free run, combined into a Memory Score with equal weights app. B.4.2 §2.3.
  • Rules: a programmatic checker gives a hard-rule pass rate and a soft-rule satisfaction; their product is the Rule Satisfaction Score. An LLM auditor scores coverage and conflict handling, and their sum over 20 is the Rule Auditability Score app. B.4.3 §2.3.
  • Multi-agent: role diversity and evidence integration, averaged into an Observable Collaboration Score app. B.4.4 §2.3.

Key formulas are below. They are the appendix definitions, and the Pitfalls section lists what the code does differently.

Pseudocode
# Decision round: one agent invocation every 4 h  [§2.1] [app. B.3.1]
# State (cash, positions) is kept after the round ends  [app. B.3.1]
def decision_round(agent, account, trace_log):
    # Repeat ReAct steps until the round ends or the budget is spent  [app. B.3.1]
    while not round_finished and budget_left():
        # Each tool result is appended as an observation  [app. B.3.1]
        step = agent.step(account, trace_log)
        trace_log.append(step)
        # The round ends on the termination token  [app. B.3.1]
        round_finished = step.emitted_trade_done
    return account
Pseudocode
# Hourly return from account-equity checkpoints  [app. B.1]
def hourly_returns(equity):
    returns = []
    # Consecutive checkpoints give one interval each  [app. B.1]
    for prev, cur in pairwise(equity):
        # Zero return when the starting value is not positive  [app. B.1]
        if prev > 0:
            returns.append((cur - prev) / prev)
        else:
            returns.append(0.0)
    return returns
Pseudocode
# Total return: compound the hourly returns  [app. B.1]
def total_return(returns):
    growth = 1.0
    for r in returns:
        growth *= 1.0 + r
    return growth - 1.0
Pseudocode
# Memory retrieval score for a same-market entry  [app. B.3.3]
def memory_score(similarity, age_days):
    # Recency decays with a 7-day half-life  [app. B.3.3]
    recency = 0.5 ** (age_days / 7.0)
    # Similarity weight 0.8, recency weight 0.2  [app. B.3.3]
    return 0.8 * similarity + 0.2 * recency
Pseudocode
# Write gate: reject near-duplicates of same-market entries  [app. B.3.3]
def memory_add(candidate, store):
    # Highest cosine similarity to existing entries in this market  [app. B.3.3]
    best = max(cosine(candidate, m) for m in store.same_market())
    # At or above 0.90 counts as a duplicate  [app. B.3.3]
    if best >= 0.90:
        return False
    store.add(candidate)
    return True
Pseudocode
# Tool Call Score for one trace  [app. B.4.1]
def tool_call_score(judge_scores, routing, valid_rate, hallu_free):
    # Mean of the four judge scores, each already on [0, 1]  [app. B.4.1]
    judge_mean = sum(judge_scores) / 4
    # Fixed weights from the appendix  [app. B.4.1]
    return (0.55 * judge_mean + 0.20 * routing
            + 0.15 * valid_rate + 0.10 * hallu_free)
Pseudocode
# Role Diversity: u distinct roles, p = share of the top role  [app. B.4.4]
def role_diversity(u, p):
    # One role or fewer earns nothing  [app. B.4.4]
    if u <= 1:
        return 0.0
    # Scaled by role count; penalised only when the top role exceeds 80%  [app. B.4.4]
    return (u - 1) / 3 * min(1.0, (1 - p) / 0.2)
Pseudocode
# Evidence Integration: a sources accepted, b of them cited in the decision  [app. B.4.4]
def evidence_integration(a, b):
    # Share of accepted sources that are used  [app. B.4.4]
    coverage = b / a if a > 0 else 0.0
    # Cross-source term, zero unless two or more sources are cited  [app. B.4.4]
    integration = (b - 1) / (a - 1) if a > 1 and b > 1 else 0.0
    return 0.5 * (coverage + integration)

Hyperparameters

NameValueWhereNotes
Decision intervalevery 4 h, shared grid§2.1positions persist between rounds
Account check cadenceevery 30 sapp. B.1checkpoints written at interval boundaries
Return intervals15 min, 1 h, 4 h, 1 dapp. B.1headline metrics use 1 h
Tail-loss quantileworst 5% of 1 h returnsapp. B.1signed return, closer to zero is better
Evaluation window2026-04-13 to 2026-05-13§3about 30 days, paper trading
Asset universe6 crypto, 17 US equities (23)app. B.2agents restricted to this set
Grid baseline levels5 above and 5 below referenceapp. B.2
Grid spacing1% multiplicativeapp. B.2
Grid capital share80% of equityapp. B.2
Grid minimum orderUSD 5app. B.2
Grid pending orders24 per symbolapp. B.2
Grid cleanup band1.5app. B.2band multiplier for far orders
Extra tool catalogue295 tools (305 with base)app. B.3.2five core tools always visible
Memory embedding modelBAAI/bge-base-en-v1.5app. B.3.3queries and entries
Memory similarity weight α0.8app. B.3.3recency weight 1 − α
Memory half-life7 daysapp. B.3.3
Memory duplicate cut-offcosine ≥ 0.90app. B.3.3same market tag
Manager step budget100 stepsapp. B.3.5
Specialist turn budget20 model turns per callapp. B.3.5
NewsAgent search budget3 search calls per invocationapp. B.3.5
Base round budgetnot statedapp. B.3.1"configured execution budget" only
Tool-judge modelQwen3.5-397B-A17B (primary)app. B.4.1DeepSeek-V4-Pro for cross-judge
Memory judgesQwen3-Max, GPT-5.4, DeepSeek-V3.2 (thinking variant)app. B.4.2scores averaged
Rule auditorQwen3.5-397B-A17B (primary)app. B.4.3DeepSeek-V4-Pro and GPT-5.5 for cross-audit
Tool Call Score weights0.55 J + 0.20 RQ + 0.15 VCR + 0.10 HFRapp. B.4.1J = mean of four judge scores
Memory Score weightsequal, four termsapp. B.4.2MCQ, MUQ, normalised ΔMDD and ΔTL
Rule Satisfaction ScoreHPR × SRSapp. B.4.3hard gate on soft satisfaction
Rule Auditability Score(RC + CH) / 20app. B.4.3each judge score on 1 to 10
Observable Collaboration Scoreequal mean of RD and EIapp. B.4.4
Role-dominance threshold80% of scored-role callsapp. B.4.4sensitivity reported in app. D
Rule thresholds, R0leverage ≤ 5×; margin ≥ 10%; intraday loss > 5% stops; order ≤ 20% of equityapp. A.1.1
Rule thresholds, R1asset ≤ 15% of equity; cash ≥ 10%; excluded small caps (< USD 50M)app. A.1.1
Rule thresholds, R2cash 5–15%; smart-contract share 30–70%; reversal inside 60 min discouraged; scaling over 10% of equityapp. A.1.1fee assumption about 0.15% per trade

Results

Author-reported. Setup: five backbones (GPT-5.4, DeepSeek-V3.2, Gemini-3.1-Pro, Grok-4.20, Qwen3-Max) on the live 30-day paper-trading window §3. Hardware is not stated.

Ranking stability §3.1 fig. 2. Across the window, cumulative-return rankings reverse 21 times (tool use), 19 (memory), 19 (rules) and 29 (multi-agent). Capability rankings reverse 0, 5, 8 and 2 times for the same four settings. The paper names the capability leaders as GPT-5.4 for tool use, Grok-4.20 for memory use, Gemini-3.1-Pro for rule following and DeepSeek-V3.2 for collaboration §3.1.

Cumulative return by backbone and configuration (%) app. D:

ModelPlain ReAct+Tools+Memory+Rules+Team
DeepSeek-V3.2−0.127.091.785.773.60
Grok-4.20−2.077.790.843.09—
GPT-5.43.455.731.995.745.90
Gemini-3.1-Pro3.46—2.143.506.03
Qwen3-Max6.832.550.334.422.44

Tool use §3.2.1 app. D. Judge scores are on 0 to 1 and come from Qwen3.5-397B-A17B over 170 traces per backbone fig. 3. Gemini-3.1-Pro has no tool-use run.

ModelTRIEICEFJudge meanRQVCRHFRTCS
GPT-5.40.8990.6390.9110.8990.8370.9200.9981.0000.894
Qwen3-Max0.8960.7310.9010.8910.8550.7360.9781.0000.864
DeepSeek-V3.20.8120.5810.8340.7450.7430.7850.8251.0000.789
Grok-4.200.7670.5870.8090.7400.7250.8050.6661.0000.760

Information coverage is above 0.8 for all four models, while invocation efficiency is the lowest judge score for each §3.2.1.

Persistent memory §3.2.2 tab. 1:

ModelSearchesAddsStoredMCQMUQΔMDD (pp)ΔTL (pp)ΔPnL full → late (pp)
GPT-5.4332110.9780.762−0.10+0.13−1.46 → −1.80
DeepSeek-V3.271255410.9380.680+2.57+0.40+1.90 → +2.14
Gemini-3.1-Pro31997930.9380.497+0.06+0.05−1.32 → +0.53
Grok-4.2029112120.7600.855+3.30+0.83+2.91 → +3.18
Qwen3-Max41515110.8780.742+1.51+0.38−6.50 → −4.85

All five memory runs lower tail loss against their matched memory-free run, while four of the five also show a shallower maximum drawdown §3.2.2. Storage volume does not track the downstream effect: Grok keeps 12 entries and has the highest MUQ, while Gemini keeps 93 entries and has the lowest MUQ §3.2.2.

Rule following §3.2.3 tab. 2:

ModelFinal return (%)MDD (%)HPRRSSRAS
GPT-5.4+5.7385.1690.6820.6340.677
DeepSeek-V3.2+5.7654.2700.8770.8470.623
Gemini-3.1-Pro+3.5034.9270.8880.8720.882
Grok-4.20+3.0855.1650.8340.7510.775
Qwen3-Max+4.4185.6740.6130.6120.511

The paper reports that DeepSeek-V3.2 and GPT-5.4 end with almost the same return but differ sharply on HPR and RSS §3.2.3. The most frequent hard-rule failure is R1-02, the per-asset concentration limit, mostly for GPT-5.4 and Qwen3-Max §3.2.3.

Multi-agent §3.2.4 tab. 3 fig. 6:

ModelΔReturn vs ReAct (pp)ΔSharp gainsΔMDD (pp)ΔTL (pp)RDEI
GPT-5.4+2.45+3+0.93+0.380.0120.518
DeepSeek-V3.2+3.72+2+0.25+0.260.7600.852
Gemini-3.1-Pro+2.57+2+0.49+0.260.4920.984
Qwen3-Max−4.39+6+2.85+0.260.2820.751

Positive risk changes mean more downside tab. 3. The paper argues that a shared harness produces different collaboration patterns, and that more agents alone do not show real collaboration §3.2.4. DeepSeek-V3.2 has the highest observable collaboration score at 0.806 §3.2.4.

Paper vs code

Code: official repository linked directly by the paper at commit 6ea7434c019fdecd7c52129592f275e224b5919c, MIT licence (copyright Fu Leiming). The paper's \coderepository field names the same URL.

Consistent with the paper:

  • Memory retrieval weights α = 0.8 and a 7-day half-life [backend/services/agent/memory.py#L73-L74, #L91, #L97; backend/config/agent_config.py#L61, #L63].
  • Memory duplicate threshold 0.90 [backend/services/agent/memory_tools.py#L17]. Embedding model BAAI/bge-base-en-v1.5 [backend/config/agent_config.py#L48].
  • Grid baseline settings: 5 levels, 1% step, 80% capital, USD 5 minimum, 24 pending orders, cleanup band 1.5 [backend/services/baselines.py#L196-L201].
  • Five always-available tool names and routing scores on a 0 to 4 scale, with empty requests skipped [backend/services/evaluation/llm_tool_judge.py#L23-L29, #L75-L110]. The paper's per-request routing formula matches this code.
  • Sub-agent turn budget of 20 and NewsAgent search budget of 3 [backend/services/agent/multi_agent_advanced.py#L46, #L1231].
  • Tool-judge dimensions, 0 to 10 scale [backend/services/evaluation/sys_prompts_eval/tool_use_judge.md; docs/evaluation/metrics.md].
  • Rule thresholds in the rule-aware checker: R2-02 cash 5–15% [backend/services/agent/rule_aware/rule_validator.py#L625-L626]; R2-03 30–70% [#L676-L677]; R2-05 fee plus slippage default 0.10% + 0.05% [#L786-L788]; R2-01 reversal window 60 min [#L562].
  • Rule Auditability Score: the auditor's final score is (coverage + conflict) / 20 [backend/eval/offline_llm_audit_rule_informed.py#L13, #L211].
  • The Tool Call Score values match the paper's formula. Recomputing GPT-5.4 and Qwen3-Max from the tool-use table in appendix D (judge and objective columns) gives 0.894 and 0.864, as printed.

Not found in the code (checked by search):

  • The Tool Call Score weights (0.55, 0.20, 0.15, 0.10) as code. The only place they appear is the paper and the appendix.
  • Role Diversity, Evidence Integration and the Observable Collaboration Score. No function with these definitions was found.
  • The MCQ and MUQ LLM-judge scoring. backend/services/evaluation/memory_evaluator.py computes retrieval distribution, diversity and growth instead, which are different quantities.

Differences:

  • Rule Satisfaction. The appendix defines SRS as the unweighted fraction of six R2 preferences satisfied per trade, and RSS as HPR × SRS app. B.4.3. The code's checker returns a continuous 0 to 1 score per R2 rule with quadratic penalties [backend/services/agent/rule_aware/rule_validator.py#L56, #L607, #L647]. The older backend/services/evaluation/rule_evaluator.py#L576-L584 uses different R2 rules (volatility, turnover, sector, cash drag, fee, engagement) with weights 0.15 and 0.20, and combines them with the auditor score using λ = 0.5 [#L556]. No HPR × SRS product was found. Which pipeline produced the paper's SRS was not established from the repository.
  • Multi-agent step budget. The paper states a 100-step manager budget app. B.3.5. The code's default is 15 steps [backend/services/agent/multi_agent_advanced.py#L54], and the benchmark adapter defaults to 30 [backend/benchmark/builtin/agents/advanced_multi_agent.py#L37-L40]. The value used in the paper's runs is not in the repository.
  • Memory retrieval cut-off. The store code tags entries above a 0.6 similarity cut-off [backend/config/agent_config.py#L57, backend/services/agent/memory_pinecone.py#L209]. The paper does not mention this. How the tag affects retrieval was not traced.
  • Memory entries with no timestamp get recency 0.5 in the code [backend/services/agent/memory.py#L93]. The paper does not say this.

Pitfalls

  • One trajectory per cell. Each model and configuration appears to come from a single live run. The main text gives no confidence intervals, so small return gaps come from one market path. Treat them as indicative.
  • Composite weights are the authors' choice. The Tool Call Score (0.55/0.20/0.15/0.10), equal-weight Memory Score and OCS, and the 80% role-dominance threshold all affect the ranking. The appendix reports a sensitivity check for the role-dominance threshold app. D.
  • LLM judges overlap with the compared models. The memory judges are Qwen3-Max, GPT-5.4 and a thinking variant of DeepSeek-V3.2 app. B.4.2, which are three of the five compared backbones. The tool and rule judges are Qwen3.5-397B-A17B, which is not compared, with cross-judge checks in the appendix app. B.4.1 app. B.4.3.
  • Routing relevance is hand-scored. The RQ metric depends on prior 0 to 4 relevance scores for each candidate tool, which are stored in backend/services/evaluation/tool_eval/tool_quality_scores.json. Changing those scores changes RQ and TCS.
  • Return definition. Returns compound one-hour account-equity changes, including market moves on existing positions. Intervals with a non-positive starting value are set to zero app. B.1. This is not per-trade P&L.
  • Memory growth is path-dependent. The store is empty at the start, is scoped to account and market, and grows over the run app. B.3.3. Results with few entries (GPT-5.4 stored one) reflect whether the agent chose to write, not just its retrieval skill.
  • Rule checks use approximate exposure. The paper notes that concentration violations often come from misestimating post-trade exposure, not from ignoring the rule §3.2.3. The checker is therefore sensitive to position bookkeeping.
  • HPR and RSS measure different things. A high HPR does not imply a high RSS, since the soft-rule term is averaged separately §3.2.3. Use both.
  • Collaboration metrics measure participation, not analysis quality. RD and EI describe which roles were called and which evidence was cited. The paper says they do not judge the accuracy of specialist analysis app. B.4.4 §3.2.4.
  • Hidden costs. The tool-use table lists an average token count from about 18.6k to 50.9k and calls per step from 0.95 to 4.21 app. D. Cost differs a lot across models.
  • Base step budget not stated. The round budget for the Base configuration is not given, so a comparison with other configurations depends on an unstated budget app. B.3.1.

Lineage

  • Builds on: ReAct as the shared scaffold §2.2; grid trading as the deterministic baseline app. B.2; a public API snapshot as the source of the extra tool catalogue app. B.3.2.
  • Compared against: the paper's own Base ReAct configuration §2.2 and a deterministic grid strategy app. B.2.
  • Superseded by: not known from this wiki.

Source and license

Adapted from “LiveMACE: Process-Aware Evaluation of LLM Agent Capabilities in Evolving Markets” by Jun Zhao, Leiming Fu, Yanbo Wen, Yiding Wang, Xuantong Liu, Yang Shu, Yuyang Lu, Xuanran Xing, Jingqi Tong, Hao Xu, Qi Zhang, Xuanjing Huang (arXiv:2610.09872), used under CC BY 4.0. Changes: condensed and restructured for implementation, with tables, equations and algorithms taken from the paper and its code, each cited to its source. Not affiliated with or endorsed by the authors. Code citations refer to https://github.com/RaymondVoldemortFu/LiveMACE at commit 6ea7434c019f (MIT). Any quoted code is shown under that license.