Rethinking Multimodal Time-Series Forecasting Evaluation

In plain words

Forecasting a number such as a commodity price or a search-interest curve from its past values is a classic task. Newer forecasters are often pretrained models that read only the numbers. A person forecasting the same quantity would also use what is happening in the world: a holiday, an oil-supply decision, a change in a related price. Feeding that kind of text to a language model alongside the numbers is the multimodal setting.

The paper argues that the benchmarks used to compare such systems are weak. Many are small, many are made up, and some may already be in the training data of the models being tested. The authors build a new benchmark from real series and real news-style events. They date every event carefully and keep a pipeline that can be re-run with newer data later.

When they test a range of forecasters on it, the usual ranking changes. Plain averages of two forecasters do better than fancier schemes in which a language model edits a numeric forecast. Some earlier benchmarks seem to favour language models more than real data does.

TL;DR

TimesX is a benchmark of 190 real-world variables from 19 domains. Each variable has four kinds of text context: metadata, calendar features, covariate summaries and time-stamped events. The events come from a multi-agent web pipeline with time isolation, and the benchmark is meant to be refreshed as models' knowledge cut-offs move. Zero-shot, a plain average of TimesFM-2.5 and Gemini-2.0-Flash has the best author-reported geometric-mean MASE (0.619). Agentic revision methods do not beat that average §4.2.

Problem

Existing multimodal forecasting benchmarks have three problems, according to the paper §1. First, they are small or synthetic, so their scores may not carry over to real data §3.1. Second, pretrained models may have seen the test data, and benchmarks have a short shelf life as cut-offs move §3.1. Third, the textual context is of limited types and uneven quality, so a low score cannot tell a weak method from weak context §1 §3.1.

To limit contamination, the paper keeps only evaluation horizons that start after each evaluated model's knowledge cut-off. The default cut-off is 2024-07-01 §4.1 app. "Knowledge Cutoff of Evaluated Models".

Method

The paper has two parts: the benchmark construction §3 and a zero-shot evaluation with composed solutions §4. There is no new forecasting model.

Numeric series §3.2. Weekly series are Google Search Trends values across 12 domains. Daily series are commodity prices from a market-data API and major USD exchange rates from a currency-rates API. Both APIs can be re-queried to extend the data §3.2. All series in one frequency share timestamps.

Text context §3.2. Each variable gets four kinds of context:

1. Metadata: a fixed template with the variable name and unit, its domain, its frequency and the prediction window app. "Details on Metadata and Calendar Context Construction".
2. Calendar: holidays overlapping the forecast horizon, taken from the Python holidays library and written as text app. "Details on Metadata and Calendar Context Construction".
3. Covariates: for each other series in the same domain, the mean, median, maximum and minimum over the history (with dates), and an overall trend direction, written as text app. "Covariate Context Construction".
4. Time-stamped events: short event descriptions aligned to the variable's time windows, built by the agent pipeline below §3.2.

Event agent §3.2 app. "Details of Textual Event Construction". Four LLM roles run in sequence, each with time-bounded web search and lightweight crawlers:

1. Hypothesizer: finds peaks or unusual moves and proposes candidate events with draft dates and source URLs, adding hypotheses until enough peaks are covered or a step cap is reached.
2. Verifier: re-fetches evidence under the same time bound, extracts an announcement date and an occurrence date, assigns a temporal type from a fixed set, and drops claims without an accessible source.
3. Enricher: fills missing details (actors, places, magnitudes, dates) with a bounded number of extra searches, each adding at most one source.
4. Synthesizer: writes the final event text, fixes timestamps and discards events that stay in doubt.

Every query has an explicit upper time bound. Verification rejects evidence first published after that bound. Only events with a source URL, a stored access date and a supporting quote survive.

Evaluation protocol §4.1. Each variable has a look-back window of 96 and a forecast horizon of 12. The rolling-window stride is 4 for weekly and 12 for daily data. Each prompt gets all metadata, the calendar and covariate text, and the K = 10 most recent events whose announcement dates fall strictly before the first forecast timestamp. The evaluation uses forecast horizons that start on or after 2024-07-01 unless stated otherwise. The benchmark runs offline; web access is limited to the build-time agent §4.1.

Models and composed solutions §4.1. Three time-series foundation models are used: TimesFM-2.5, Moirai-2.0 and Sundial. Three LLMs are used: Gemini-2.0-Flash, GPT-4o and DeepSeek-V3. Four composed solutions are tested. AvgEns averages two forecasts with equal weights; the pairs in the results are TimesFM with Moirai, GPT-4o or Gemini. TextRev asks an LLM to revise a TimesFM forecast in text. CodeRev asks an LLM to write and run Python that revises it. FuncRev lets the LLM pick from a fixed set of forecast-adjustment functions §4.1 app. "Implementation Details of Methods".

Metric app. "Evaluation Metrics". Each window's MASE is the mean absolute error scaled by the in-history seasonal difference eq. 1 eq. 2. A seasonal-naive forecast gives the reference eq. 3. For each variable, the summed model MASE is divided by the summed seasonal-naive MASE eq. 4. The primary score is the geometric mean of these ratios over variables eq. 5. Average rank over variables is a secondary score eq. 6. Seasonality m is 12 for monthly, 4 for weekly and 7 for daily data.

# Normalised MASE and geometric-mean score  [eq. 1] [eq. 2] [eq. 4] [eq. 5]
# Input:  per variable: windows, each with history, actual and forecast
# Output: geometric mean over variables (lower is better)

def gm_score(variables, m):
    ratios = []
    for var in variables:
        model_sum = 0.0
        naive_sum = 0.0
        for win in var.windows:
            # In-history seasonal scale, from the history only  [eq. 1]
            scale = seasonal_scale(win.history, m)
            # Per-window MASE, for the model and for seasonal naive  [eq. 2-3]
            model_sum += mase(win.actual, win.forecast, scale)
            naive_sum += mase(win.actual, snaive(win.history, m), scale)
        # Ratio of the two sums for this variable  [eq. 4]
        ratios.append(model_sum / naive_sum)
    # Geometric mean across variables  [eq. 5]
    return geometric_mean(ratios)

# Equal-weight ensemble of a TFM and an LLM  [§4.1]
# Input:  tfm_forecast (H,), llm_forecast (H,)
# Output: forecast (H,)

def avg_ensemble(tfm_forecast, llm_forecast):
    # The paper sets both weights to 0.5 and does not tune them  [§4.1]
    return 0.5 * tfm_forecast + 0.5 * llm_forecast

# Select textual events for one forecast window  [§4.1]
# Input:  events (each with an announcement date), horizon_start
# Output: up to K = 10 events for the prompt

def select_events(events, horizon_start, k=10):
    # Keep only events announced strictly before the horizon  [§4.1]
    eligible = [e for e in events if e.announced < horizon_start]
    # Take the most recent k of them  [§4.1]
    eligible.sort(key=lambda e: e.announced, reverse=True)
    return eligible[:k]

# Event construction for one time block  [app. "Details of Textual Event Construction"]
# Input:  block B, peaks P_B, coverage threshold theta, step cap K_max
# Output: hypothesis set H_B

def build_block_events(block, peaks, theta=0.9, k_max=3):
    hypotheses = []
    for step in range(k_max):
        # Ask the LLM for more hypotheses for this block, with time bounds
        hypotheses += propose_hypotheses(block, hypotheses)
        # Share of peaks with a temporally aligned, relevant hypothesis
        if coverage(hypotheses, peaks) >= theta:
            break
    return hypotheses

Hyperparameters

Results

Author-reported. Setup: TimesX, 2,434 forecasting samples over 19 domains and 190 variables, horizons on or after 2024-07-01 app. "Evaluation Samples Statistics" §4.1. Values are the MASE column of the main table, in the paper's normalised form (SeasonalNaive = 1.000); each entry is the mean over 10 runs tab. 4. Rank is the average rank over variables.

Other reported results.

The paper reports the ablations as follows. Adding all four context types to Gemini gives about 16% lower MASE than metadata alone §4.3. The paper saw no gain from covariates or events added alone §4.3. Calendar context helps most in Shopping, and event context in RMC, SAM, SHVM and Curr §4.4. Reasoning models (GPT-5, Gemini-2.5-Flash, DeepSeek-R1) show no clear advantage over the earlier LLMs on post-January-2025 data §4.2. A probe with one retrieved past example, plus an explicit no-change option, gives TextRev a geometric-mean MASE of 0.707 on the last-three subset, where plain TextRev scores 0.782 §4.6 tab. 7.

Paper vs code

The code link points to TimesX/dataset_agent in the google-research repository (Apache-2.0, LICENSE at the repo root). The snapshot at 0eebfd4 contains only the event-construction agent and its YAML configs. It contains no forecasting models, no MASE/GM/CRPS code, no AvgEns, TextRev, FuncRev or CodeRev code, and no released dataset files. A text search of the .py, .md and .yaml files for TimesFM, Moirai, AvgEns, TextRev, FuncRev and CodeRev found no matches. The benchmark numbers therefore cannot be re-run from the code in the repository.

Within the event agent, the following differences were found:

• Models: the paper uses Gemini 2.5 Pro plus Gemini 2.5 Flash for event construction app. "Details of Textual Event Construction". The shipped configs set gemini-3.5-flash for all five LLM roles US-1.yaml.
• Early stopping: the paper says Phase 1 uses coverage-based early stopping §3.2 app. "Details of Textual Event Construction". The shipped batch config turns it off US-1.yaml. The code default threshold is 0.95 module_00_setup.py, while the paper's θ is 90% and the batch config sets 0.9 US-1.yaml.
• Iteration cap: the paper's K_max = 3 matches initial_event_iterations: 3 US-1.yaml. The smoke config uses 1 smoke_1.yaml.
• Peak detection: the paper says only "a standard peak detector with fixed hyperparameters" app. "Details of Textual Event Construction". The code's trend channel uses an absolute threshold of 75 or the 90th percentile, with a 3-day suppression radius module_06_peak_discovery.py US-1.yaml. The paper does not describe this dual-channel discovery.
• Not checked: the Enricher step cap L_max, the ±k-day search window and the leakage rule that refuses post-t_e evidence were not traced in the code.

Pitfalls

• Table 2 Gemini row. The stated change of 14.81% does not match its own values. From 0.514 to 0.594 the change is 15.56%. The other three rows match their values tab. 2. Treat the Gemini percentage as unreliable.
• Count mismatches. The text says 190 variables, but the GM formula says "In our release, D = 200" app. "Evaluation Metrics". The paper gives 2,434 samples in the appendix and "about 2.5K" in §4.1. The training split is stated as 2018–2022 in §3.3 and as 2018–2023 in §4.6.
• Aggregation. Table 4 is labelled "MASE" and the paper's main text calls it normalised MASE. Use the geometric mean of per-variable ratios as defined in eq. 5, not the arithmetic mean.
• Subset tables are not comparable. Table 7 uses only the last three evaluation instances per variable. Its AvgEns value of 0.712 is not the 0.619 from the main table. The claim that TextRev-ICL-C "surpasses" AvgEns is made only on that subset §4.6.
• Stochastic LLMs. Ten seeded runs are averaged tab. 4. The main table gives no spread, interval or significance test, so small differences (for example 0.619 against 0.627) should not be read as settled.
• Leakage control depends on the cut-off. The default cut-off is 2024-07-01 and the stated model cut-offs are earlier app. "Knowledge Cutoff of Evaluated Models". The Time-MMD comparison uses the 2021-06-30 to 2024-04-01 period app. "Additional results for context–quality study on Time-MMD dataset". Results on other periods may change.
• Event construction is not reproducible from the paper alone. Live runs depend on external quotas and search results, and the README says event counts are not deterministic README.md. Running the agent needs Google Custom Search and Gemini API keys README.md.
• Audit. A manual audit of 50 samples found 94% exact date matches, 4% conservative later offsets and 2% earlier offsets §3.2. This is a small sample for a 190-variable benchmark.
• Third-party content. The event texts are built from web pages and cite their URLs. The paper gives no statement on redistribution of that text, and the paper's CC BY 4.0 licence covers the authors' own work only. Check the terms of each source before publishing event text from the dataset.
• Cost. The $0.7 per variable per block figure is an author estimate for one model pair and one pricing period §3.2.
• Forecasting-side details are thin. Decoding settings are "recommended by OpenRouter" with no values, and the paper does not say how forecasts are parsed from LLM text. The FuncRev function list is described in groups; reimplementing it requires the appendix list.
• Ensemble weights. Equal weights are fixed; the paper gives no sweep of the weight. The "simple ensembles win" claim rests on pairs of forecasters, two of them TFMs only.
• Refresh is planned, not shown. The paper plans to update the dataset every three months with versioned releases §3.3. It reports no second release.

Lineage

• Compared against: CiK (synthetic context benchmark) §2 §4.1; Time-MMD §2 §3.2; MTBench, ChatTime and MoTime tab. 1; GIFT-Eval for the metric §4.1 app. "Evaluation Metrics".
• Models evaluated: TimesFM-2.5, Moirai-2.0, Sundial, Gemini-2.0-Flash, GPT-4o and DeepSeek-V3 §4.1.
• Builds on: the MASE and geometric-mean protocol of GIFT-Eval and Chronos Benchmark 2 app. "Evaluation Metrics".
• Superseded by: not stated.