How Do AI Agents Spend Your Money? Analyzing and Predicting Token Consumption in Agentic Coding Tasks

In plain words

AI coding agents fix software bugs by reading files, running commands and trying edits. Each step adds more text to the conversation, and the whole history goes back to the model on every step. This paper measures where that text, and therefore the cost, goes. It runs eight large language models through one standard coding-agent setup on real GitHub bug reports and records the tokens each run uses.

The main finding is that most of the bill comes from input: the long history the agent keeps resending. Two runs on the same bug can cost very different amounts. Spending more does not reliably buy a better result, and the costlier runs show more repeated file reads and edits. Models also differ in how many tokens they need for similar work.

The authors also ask each agent to estimate its own token use before it starts. These estimates follow the real usage only loosely, and they tend to come out too low. The authors suggest such estimates could serve as a rough early warning for expensive runs, not as an exact price.

TL;DR

• Eight frontier LLMs run OpenHands on the 500 SWE-bench Verified tasks, four runs per task §2. Agentic coding uses about 3500x the tokens of single-round code reasoning and about 1200x those of multi-round code chat §3 fig. 1. The abstract puts the first gap at 1000x [abstract].
• Input tokens, not output, drive the cost. In a Claude Sonnet 4.5 case study, cache reads are the largest cost in every phase §5.2 fig. 8.
• Runs on the same task differ by up to 30x in total tokens §1. Accuracy rises modestly to a mid cost and then levels off §3 fig. 3.
• Self-prediction correlates weakly with real token use: up to 0.39 for output tokens (Sonnet 4.5), and the models underestimate §6 fig. 10 fig. 11.

Problem

Coding agents are usually billed per token, and the user does not know the total until the task ends. The bill is also owed when the task fails §1. The paper asks whether token use can be predicted before a task runs, so that users, providers and budget alerts can act on it §1.

Method

The study has three parts. First, it collects agent trajectories and analyses where tokens go §2 §3. Second, it compares models on accuracy and token use §4. Third, it breaks one model's cost down by phase and by round §5, and tests self-prediction §6.

1. Agent and benchmark. OpenHands runs on SWE-bench Verified, real GitHub issues with their repositories and tests §2. The agent works in rounds: the model writes a response, makes a tool call and sees the result. The full conversation history is carried forward unchanged to each later round §2.
2. Models and runs. Eight models are run, each four times per task: Claude Sonnet 3.7, Sonnet 4, Sonnet 4.5, GPT-5, GPT-5.2, Qwen3-Coder-480B-A35B-Instruct, Kimi-K2 and Gemini 3 Pro §2. Per-task token metrics average the four runs §2.
3. Baseline tasks. Agentic coding is compared with code reasoning (single turn, no tools) and code chat (multi-turn dialogue) §3.
4. Cost levels. For each task, the four runs are ranked by token cost and grouped as MinCost, LowerCost, UpperCost and MaxCost §3 fig. 3. Repeated file views and edits are counted per run fig. 4.
5. Difficulty. SWE-bench Verified's expert time estimates are grouped into "<15 min", "15 min to 1 hour" and ">1 hour". The 1–4 hour and over-4-hour groups are merged, since only three tasks are in the latter §3. Rank correlation with tokens is Kendall tau-b §3.
6. Model efficiency. Accuracy is plotted against cost. Two subsets are used: tasks solved by every model, and tasks failed by every model §4 fig. 6.
7. Cost decomposition (Claude Sonnet 4.5). The bill is split into four token types: uncached input, output, cache creation and cache read §5.1 app. B. Each run is cut into five phases, named Setup, Explore, Fix, Validate and Closeout, by the agent's behaviour §5.2 tab. 1. A single trajectory is also traced round by round §5.3 tab. 2 fig. 9.
8. Self-prediction. The same agent, with its tools, is told to explore the repository and output a JSON estimate instead of a fix app. C.1. The prompt includes one worked example §6 app. C.2. Each model makes three independent predictions per task §6. The estimate has predicted input, output and total tokens, a confidence in [0, 1], and a per-phase breakdown app. C.1.

The cost of one API round, with explicit prompt caching, is:

# Dollar cost of one agent round, with explicit prompt caching  [app. B]
# Input:  token counts of one API call
# Output: cost in US dollars

def round_cost(prompt, completion, cache_write, cache_read):
    # Uncached input is prompt tokens minus cache reads  [app. B]
    uncached = max(0, prompt - cache_read)
    # Each token type has its own rate per million tokens; values are
    # in the Hyperparameters table  [code: analysis/phase_level_analysis/
    # token_cost_ratio_analysis.py#L42-L47]
    cost = uncached * RATE_INPUT
    # Cache creation is added on top, at the 5-minute write rate
    # [§5.1] [app. B]
    cost += cache_write * RATE_CACHE_WRITE
    # Cache reads are billed at the discounted read rate  [app. B]
    cost += cache_read * RATE_CACHE_READ
    # Generated tokens  [app. B]
    cost += completion * RATE_OUTPUT
    return cost / 1_000_000

Self-prediction and its scoring:

# Self-prediction of token use before execution  [§6] [app. C]
# Input:  one task description and the coding agent with its tools
# Output: predicted input, output and total tokens; prediction cost

def predict_tokens(task, agent):
    # The agent may read files and run commands but must not edit  [app. C.1]
    # It must finish by calling the finish tool with a JSON estimate  [app. C.1]
    estimate = agent.run(task_prompt(task))
    # Parse the JSON fields the prompt asks for  [app. C.1]
    prediction = parse_json(estimate.finish_message)
    # The cost of the prediction run itself  [§6]
    prediction_cost = round_cost_total(estimate.usage)
    return prediction, prediction_cost

# Three runs per model; Pearson r per run, then the mean over runs
# [§6] [code: self_prediction_analysis/calculate_correlations.py#L158-L166]
def score(predicted_by_run, actual):
    # One correlation per run index, between predicted and real tokens
    run_corrs = [pearson(p, actual) for p in predicted_by_run]
    return mean(run_corrs)

Hyperparameters

Results

Author-reported. Setup: OpenHands on SWE-bench Verified (500 tasks), eight models, four runs per task §2.

Without the worked example, correlations of the two models that still followed the format were:

Source: tab. 3 app. D

Paper vs code

Official code: the project page the paper links points to LongjuBai/agent_token_consumption_analysis, owned by the first author. It was cloned at commit 335ecf2. The repository has scripts and prompt templates only; its README says it does not track generated CSV or PNG results README.md. There is no LICENSE file, so the code is cite-only and nothing is quoted from it.

The project page also links a Hugging Face dataset of trajectories (loong0814/openhands_trajectories). It was not fetched or license-checked for this card.

I searched GitHub for agent_token_consumption; the author's second repository, LongjuBai/agent_token_consumption, returned a 504 and was not read.

• Pricing constants. The code uses $3.00 input, $3.75 cache write, $0.30 cache read and $15.00 output per million tokens token_cost_ratio_analysis.py. The paper states none of these values in its text. It cites a pricing page §5.1 and says output costs about 80x cache reads §5.2. The code's rates give 15 / 0.30 = 50x, not 80x.
• Cache-write label. The constant is named cache_write_1h_rate with a comment reading "1h cache" token_cost_ratio_analysis.py. The paper says the 5-minute write rate is used throughout §5.1. The code's value, $3.75, is the one used; the label and the paper's stated rate disagree, and the paper does not give the value.
• Uncached input. The code subtracts cache reads from prompt tokens and charges cache writes separately token_cost_ratio_analysis.py. This matches the paper's formulas app. B.
• Phases are positional in the code. divide_into_phases cuts each run's rounds into five equal-count bins by round index. The bins are named early, early_mid, mid, later_mid and later token_cost_ratio_analysis.py token_cost_ratio_analysis.py. The paper's Setup, Explore, Fix, Validate and Closeout phases are labelled by behaviour §5.2. No labelling code is in the repository, so tab. 1 and the phase figures cannot be reproduced from it.
• Difficulty statistics. The repository has no Kendall tau-b computation. The difficulty script produces box plots multi_model_difficulty_box_plots.py. The tau of 0.32 and the 6.7% and 11.1% figures cannot be reproduced from the release. The figure caption mentions a bootstrap interval, but that text is commented out in the source §3 fig. 5.
• Cost grouping. Before ranking tasks by cost, the code clips per-task cost at the 1st and 99th percentiles multi_model_group_accuracy_analysis.py. The paper does not mention clipping. The per-task MinCost to MaxCost table is read from cost_level_accuracy_chart.csv, which a script not in the repository produces multi_model_cost_level_accuracy_plots.py.
• Regression form. The regression formula in the code has model as a fixed term and no random effect multi_model_cost_level_accuracy_plots.py. The paper calls its analysis mixed-effects fig. 3 fig. 4.
• Self-prediction scoring. The code computes Pearson r separately for each of the three runs, then averages the three calculate_correlations.py calculate_correlations.py. The paper does not say whether it pools or averages. The code also reports correlations on log-transformed values, which the paper does not describe calculate_correlations.py.
• Overhead. The code's overhead is the mean of per-run ratios of prediction cost to task cost calculate_correlations.py. This is not the ratio of means.
• Missing task costs. For Qwen3-Coder, a missing task cost is filled from a different CSV, dataset/qwen/swe_bench_token_cost_aggregated_total_with_accuracy.csv calculate_correlations.py calculate_correlations.py. The paper does not mention this.
• Prompt. The template's phase list and finish fields match the appendix swe_token_estimate.j2 swe_token_estimate.j2. The template includes warning-symbol lines that the printed prompt does not. I did not compare the system prompt line by line.

Pitfalls

• Headline multipliers conflict. The abstract gives 1000x over code reasoning and code chat. The body gives 3500x and 1200x [abstract] §3. The source does not explain the difference.
• Phase breakdown is not reproducible. The five phases are defined by behaviour, but the released code only makes five equal-count bins by round index token_cost_ratio_analysis.py. Treat tab. 1 as the paper's labelling, which an implementer cannot re-run.
• Price ratio. The text says output costs about 80x cache reads §5.2. At the code's rates the ratio is 50x token_cost_ratio_analysis.py. Use the rates you have for your provider, not either number.
• Cache-creation double count. The formula charges cache creation on top of prompt tokens minus cache reads app. B token_cost_ratio_analysis.py. If the provider's prompt count already includes cache-creation tokens, they would be billed twice. The paper does not say which convention the API uses.
• GPT-5 cached price. The formula bills cached tokens at 0.2 times the input rate. The listed prices, $0.125 cached against $1.250 input, give 0.1 app. B. The two disagree within the appendix, so check the provider's price list before using either.
• Cost depends on the price list. Dollar shares depend on prices that change over time. The cache-read result is for Claude Sonnet 4.5 at the prices the paper used, which it links to a pricing page §5.1 §5.2 fig. 8.
• The worked example anchors the estimate. The prompt's example predicts 1,250,000 input tokens and 18,500 output tokens for a different issue app. C.2. The paper argues that underestimation is not caused by the example app. D. That check covers only two models, and correlations drop without the example.
• Few models follow the no-example prompt. The paper says most models did not follow the instruction without the example, so the no-example results cover only Sonnet 4.5 and GPT-5.2 app. D.
• Self-prediction is weak. The best output correlation is 0.39, and Kimi-K2's best input correlation is 0.38 §6. The paper says the estimates are not usable for instance-level pricing §6 §7.
• Failure cost is model-specific. The extra tokens spent on failed tasks differ a lot by model: GPT-5 and GPT-5.2 under 0.5M, Kimi-K2 about 2M §4. A budget alert calibrated on one model will not transfer to another.
• Per-model numbers are in figures. The text gives only some correlations and overheads. The rest are in fig. 10 and fig. 11, which are not transcribed here. Read the figures before quoting per-model values.
• Generalisation. One benchmark, one agent framework, and eight models §7. The paper says the patterns held across the models it tested, not beyond them.
• Version. This card follows arXiv v3, updated 2026-10-02 [raw metadata]. Earlier versions may differ.

Lineage

• Concurrent work: token distribution across multi-agent systems and token pricing in reasoning models, both cited as parallel work §1.
• Related: inverse test-time scaling, where longer reasoning does not improve accuracy, is the context for the accuracy-cost finding §3.
• Related: self-feedback and language-model calibration, cited as grounding for self-prediction §6.
• Builds on: OpenHands, the agent framework §2; SWE-bench Verified, the benchmark §2.
• Points to: budget-aware tool-use policies, cited as a way to enforce runtime token limits §7.