LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena

In plain words

The paper asks whether a language model can forecast real future events, such as who wins a championship or what a price will be on a given day. The authors build a test bench, called Prophet Arena, that keeps collecting live questions from prediction markets, where prices reflect what traders think will happen. Each question is split into stages: find the event, gather the same news and market prices for every model, ask each model for a probability, and score the answer once the event is over.

A forecast is graded in three ways. One asks how close the stated probability was to what happened. One asks whether a "70%" answer comes true about 70% of the time. The third asks whether following the model would have made money against the market's own price. The authors find that the best models are already reasonably good on the first two. On the third, none of the tested models makes money on average. They also find that models recall past events only roughly, pay uneven attention to news sources, and fall behind the market when a result is close to being known.

TL;DR

• A live, modular benchmark for probabilistic forecasts of binary prediction-market questions, with a fixed context shared by all models §2.2.
• Three metric families: Brier score (loss), binned expected calibration error (reliability) and average market return against the market price (economic value) §3.1.
• Author-reported: top models have Brier scores close to the market baseline and lower calibration error, but average return stays below 1 (break-even) for all five highlighted models tab. 2.
• Models are conservative relative to the market, and their reasoning quality differs most in reasoning synthesis and alignment, not in source use §4.2.3, §4.3.
• Implementers should note that the forecasting prompt includes live market prices, so the "market baseline" comparison measures something narrower than it first seems app. E.

Problem

Forecasts are usually judged by accuracy, which ignores how confident a forecast is. Earlier LLM forecasting benchmarks mostly used one metric (Brier score, calibration, or accuracy) and few kept a live stream of questions §3, §2.3. Market-based benchmarks need a baseline, and raw Brier scores are hard to compare across lead times, because some events are much easier to predict close to their resolution §3.2.1, Caveat 2. Measured on accuracy alone, a perfectly calibrated forecast and an overconfident one can score the same §3, Caveat 1.

Method

Overview

The pipeline has three stages §2.2:

1. Event and market extraction. Events come from active prediction markets (Kalshi). Each day, 20 new events are added app. A.1. The evaluation uses events resolved before 11 October 2025 §2.2. Each event has one or more binary markets; markets under one event may be mutually exclusive or not §2.1.
2. Prediction context. Forecast times are scheduled before resolution. Each time gets a context shared by all models: news sources found by one LLM searcher, plus a market snapshot that gives an implied probability for each market app. A.2.
3. Probabilistic forecasting and evaluation. Each model returns a probability for each market and a short rationale. Only the probabilities are scored, after the event resolves §2.2, app. A.3, app. A.4.

Definitions used below: p_{ij} is the model's probability that market j of event i resolves Yes, q_{ij} is the market-implied probability, and o_{ij} \in \{0, 1\} is the outcome §2.1, app. A.4.

Forecast schedule

1. Let \tau_i be the close time and t^{(0)}_i the first forecast time, with \Delta^{(0)}_i = \tau_i - t^{(0)}_i app. A.2.
2. Each later forecast time is halfway between the previous one and \tau_i, so t^{(k)}_i = \tau_i - 2^{-k}\Delta^{(0)}_i app. A.2.
3. A minimum gap \delta_{\min} before the close is enforced; its value is not stated app. A.2.
4. Forecasts made within three hours before resolution are excluded from all later evaluation §3.2.1, Caveat 3.

Context and market snapshot

1. The searcher is an LLM with web search (GPT-4o in all experiments). Its prompt asks for exactly 10 recent sources, each with URL, English title and a three-sentence summary app. A.2, app. E.
2. The market snapshot is taken from the exchange API. It has the last trade price, the Yes ask and the No ask app. A.2.
3. Implied probability: q_{ij,t} is the Yes price, and the No price is 1-q_{ij,t} app. A.2. Prices may not sum to 1 because of fees. The paper says this needs "slight normalization", but the normalization is not described §2.2 footnote.
4. The model prompt contains the event title, the list of allowed outcomes, the source list with rankings, and the market statistics app. E.

Scoring rules

Brier score (loss). For event i with m_i markets, the event score is the mean of squared errors over its markets eq. 1:

BS_i = \frac{1}{m_i} \sum_{j=1}^{m_i} (p_{ij} - o_{ij})^2 \qquad [\text{eq. 1}]

The overall score averages BS_i over n events §3.1.1. Pooling all markets and weighting each by 1/m_i gives the same value app. B.1.

Expected calibration error (reliability). The main text defines ECE in terms of conditional probabilities eq. 2, but that equation is typeset with m_i inside a sum over k. The estimator actually used is binned, with B = 10 bins app. B.2:

# Binned ECE with B = 10 equal-width bins  [app. B.2]
# p, o: one entry per market (pooled over events)
def binned_ece(p, o, bin_count=10):
    total = 0.0
    for b in range(bin_count):
        # bin b covers [b/10, (b+1)/10); the last bin includes 1.0
        members = [j for j in range(len(p)) if in_bin(p[j], b, bin_count)]
        if len(members) == 0:
            continue
        mean_p = mean([p[j] for j in members])
        mean_o = mean([o[j] for j in members])
        total = total + len(members) * abs(mean_o - mean_p)
    return total / len(p)

Average market return. Each market gets a unit budget. The rule for choosing a side is given in two places, and they differ on ties (see Pitfalls) §3.1.3, app. B.6:

# Risk-neutral bet on one binary market with a $1 budget  [§3.1.3]
# p: model probability of Yes, q: market price of Yes, outcome: 0 or 1
def market_return(p, q, outcome):
    if p > q:
        # buy Yes: one dollar buys 1/q shares that pay 1 if Yes
        shares = 1 / q
        won = outcome == 1
    else:
        # buy No: one dollar buys 1/(1-q) shares that pay 1 if No
        shares = 1 / (1 - q)
        won = outcome == 0
    if won:
        return shares
    return 0.0

The average return is the mean of these values over markets. A value of 1 means break-even §3.2.2.

Edge (used in definitions only): yes-edge e = p/q, no-edge \tilde e = (1-p)/(1-q) app. A.4.

Risk-averse betting (extension). With CRRA utility U_\gamma and \gamma \in [0,1] app. B.5:

# Optimal share of the budget on Yes for a CRRA investor  [app. B.5]
# p: belief in Yes, q: Yes price, gamma in (0, 1]
def yes_share(p, q, gamma):
    # exponent on the probability terms
    e = 1 / gamma
    yes_term = q ** (1 - e) * p ** e
    no_term = (1 - q) ** (1 - e) * (1 - p) ** e
    return yes_term / (yes_term + no_term)

For \gamma \to 0 this reduces to the all-in rule of §3.1.3. For \gamma = 1 it is proportional betting, a_Y = p app. B.5. The main results use the risk-neutral rule §3.1.3.

Sharpe ratio. Computed on per-event payoffs R_a against a reference return R_b = 1 (abstaining), using the sample mean and variance over n events app. C.2.

Bootstrap confidence intervals. Symmetric nonparametric intervals of the form [\hat\theta - h, \hat\theta + h], where h is the k-th order statistic of the absolute bootstrap deviations app. B.7:

# Symmetric bootstrap CI around the point estimate  [app. B.7]
# scores: per event or per market; statistic: usually the mean
def symmetric_ci(scores, statistic, reps):
    estimate = statistic(scores)
    deviations = []
    for b in range(reps):
        resample = draw_with_replacement(scores, len(scores))
        deviations.append(abs(statistic(resample) - estimate))
    deviations.sort()
    # index k and the level 1 - alpha: not stated in the paper
    h = deviations[k - 1]
    return (estimate - h, estimate + h)

Probability elicitation

The default is a verbalized probability from a prompt that asks for a JSON answer with a rationale app. E. Alternatives tested on five models app. C.3:

• prompt variants (concise, verbose, LLM-rewritten) and an ensemble of them;
• Bi-direction (the authors' method): elicit p for Yes and p^\circ for No, then use \tfrac12\bigl(p + (1 - p^\circ)\bigr);
• self-consistency over 10 Yes/No samples, unweighted or weighted by stated confidence.

Hyperparameters

Results

Author-reported. Setup: 1,367 events resolved before 11 Oct 2025, all 23 LLMs plus the market baseline, with search and market context shared §3.2.2, app. C.1. Results are bootstrapped 95% intervals for Brier and average return; ECE is a point estimate tab. 2.

Five highlighted models tab. 2:

Full evaluation (ranks omitted; reasoning settings as in the paper) app. C.1:

Summary of the findings the paper states §3.2.2:

• Brier scores sit in a narrow band of about 0.18 to 0.22. Random guessing gives 0.25.
• ECE separates models more clearly. Strong models are at or below 0.05, weaker ones fall between 0.06 and 0.2. Every highlighted model is better calibrated than the market baseline.
• No highlighted model reaches an average return of 1. The top model is at 0.943, and most are below 0.9.
• Model rankings change with the metric used §3.2.2.

Sharpe ratio app. C.2: o3 (reasoning) −0.0131 (highest); GPT-5 (reasoning) −0.0212; Gemini 2.5 Pro (reasoning) −0.0230; Gemini 2.5 Flash (reasoning) −0.0270; GPT-5 (reasoning, High) −0.0435; Claude Sonnet 4 (reasoning) −0.0404; Grok-4 (reasoning) −0.0771; Market Baseline −0.0897; Llama-4-Scout −0.1799. All values are negative.

Robustness to category mix app. B.8. The Brier ranking of the top five models is the same under the original set and two rebalanced sets. The sample sizes in that table are 816, 250 and 152, not 1,367 (see Pitfalls). Under the full rebalance, GPT-5 (reasoning) scores 0.128 and the market baseline 0.149.

Probability elicitation app. C.3. Default versus Bi-direction (Brier / ECE):

Bi-direction improves ECE on four of five models. Self-consistency is worse on Brier score (0.181 to 0.267 in the weighted and unweighted variants). The paper states that no single method wins on every column app. C.3.

Reasoning quality (LLM judge) on a 1 to 5 scale tab. 3, app. C.6:

Full judge table app. C.6: o3 averages 3.81; Gemini 2.5 Pro 3.73; GPT-5 (medium) 3.73; GPT-5 (minimal) 3.61; GPT-4o 2.79; Llama-4-Maverick 2.83. The judge agrees with 170 human-rated predictions to within about half a point on average (mean absolute difference 0.37 to 0.44 on a 0 to 4 scale) app. C.6.1.

Internalized knowledge §4.2.1, fig. 4, app. C.7. On 100 past events, models recall best in Entertainment and worst in Climate and Weather and Politics. Recall is approximate: the example model names the right song but aligns it to a chart dated two days later app. D.

Sources §4.2.2, fig. 5, app. C.8. Adding market data alone is only slightly worse than market data plus news, but the spread across questions is larger without news. Sources can also hurt: on hard-to-recall past events, adding retrieved sources made Brier scores worse.

Conservatism §4.2.3, fig. 6. LLMs give less extreme probabilities than the market for events that resolve Yes. GPT-5 and Grok-4 track the market more closely. Claude Sonnet 4 is sometimes more assertive in the middle range.

Consistency app. C.4. Mutually-exclusive consistency is between 0.993 and 0.996 across models. Logical-chain consistency ranges from 0.901 (Grok-3-Mini, reasoning) to 0.999 (GPT-5, reasoning, High). Both are judged by Gemini 2.5 Flash.

Paper vs code

No official code.

The paper has no code link in its text, its footnotes or the arXiv metadata. The paper's contact domain, prophetarena.co, has no GitHub or code link. GitHub searches for "prophet arena", "LLM-as-a-Prophet", and the organisation name prophetarena (404) turned up only unrelated projects or third-party reimplementations, none of which is used here. The paper names a Hugging Face dataset (Prophet-Arena-Subset-100) §4. It is data, not code, and was not fetched.

Pitfalls

• Betting tie-breaking differs between sections. §3.1.3 buys Yes when p/q \ge (1-p)/(1-q), which is the same as p \ge q. App. B.6 says buy Yes when p > q and No otherwise. The two agree except when p = q exactly. The paper does not say which one was used in the experiments.
• ECE equation is typeset ambiguously. Eq. 2 uses m_i inside a sum over k. The binned estimator in app. B.2 is the one the experiments used. Implement that, with B = 10.
• Brier weighting. The main text (eq. 1) averages events, weighting each market by 1/m_i. Pooling all markets without that weight gives a different number, and events with many markets dominate it app. B.1.
• Implied probability is not a fair price. It is reverse-engineered from last-trade and ask prices. Fees mean Yes and No prices may not sum to 1. The normalization is not described §2.2, app. A.2.
• The forecasting prompt includes the market price. Models see the live market statistics as an input app. E. The "market baseline" is therefore a reference point rather than an independent competitor, and the paper notes that models mostly track it §4.2.3. Results without market data are only covered by the ablation in §4.2.2 on the 100-event subset.
• Near-resolution exclusion is a fixed cutoff. Forecasts within three hours of resolution are removed. Model Brier scores improve as resolution nears, and the market gains on models in that window §3.2.1.
• Abstract versus tables. The abstract mentions "promising market returns". In the paper's own tables, the best average return is 0.943 and no highlighted model reaches 1 (break-even) tab. 2, app. C.1. The paper does not explain how the abstract phrase squares with these numbers.
• Sample sizes. The main text gives 1,367 events. The rebalancing table lists "Original (N=816)" app. B.8. The table caption for consistency says 22 LLMs, but the table has 21 rows app. C.4. Neither mismatch is explained.
• Event mix. Of 1,367 events, 76% are Sports, 8% Entertainment, 7% Politics and 9% Other §3.2.2. Category effects are large §4.2.2.
• Bootstrap details are missing. The number of replicates, the level \alpha and the index k in the algorithm are all not stated app. B.7.
• Sharpe ratio. The formula compares payoffs against R_b = 1, which is the return from keeping the budget unbet app. C.2. The text describes a payoff, not a net return, so the scale is not directly comparable to an excess return.
• Judge versus human scale. The LLM judge uses 1 to 5 in the main table, and the human validation uses 0 to 4 for the mean difference app. C.6. Check which scale you are reading.
• Model versions and sampling. Hosted models are listed without snapshot IDs. Temperature and decoding settings for forecasts are not stated app. A.3. Results can change when a provider updates a model.
• Self-consistency can't reach fine probabilities. Unweighted 10-sample voting gives values at steps of 0.1 app. C.3.
• Mutually exclusive tolerance. The tolerance \epsilon for summing to 1 is not stated app. C.4.
• Live benchmark, fixed dataset. The benchmark keeps adding events, but the paper fixes one snapshot for its tables tab. 2 caption. Results on a newer snapshot will differ.

Lineage

• Builds on: prediction-market prices as a baseline and proper scoring rules, both cited in §3.1 (the paper names no card-linked prior method).
• Compared against: MIRAI, ForecastBench, FutureBench, FutureX tab. 1, §2.3.
• Superseded by: not known from content already in this wiki.