Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools

Shunya Nagashima

cs.LGSep 23, 2026

Forecast Workflow Bench: Evaluating Language-Model Decisions with Budgeted Forecast Tools

Authors: Shunya Nagashima

Abstract

Time-series foundation models (TSFMs) provide forecasts for operational decisions, but accuracy alone does not determine their value. Evaluating agents that use these models requires measuring decision quality and forecast cost. FWBench evaluates this capability on 1,251 electricity and cycle-hire cases using fixed forecast tools and simulated capacity contracts. Agents select models, histories and horizons, then submit capacities to minimize a stated loss-cost objective. We evaluated two hosted and eight local configurations, including small language models, and tested local models with and without TSFMs. GPT-6 Astra bought inexpensive short-horizon forecasts selectively, using 2.5% of the budget, and outperformed fixed policies when the saved decisions were scored with three loss-cost weightings. FWBench enables reproducible evaluation of how language models select and use time-series forecasts to make decisions under cost constraints.

Explore similar work

Foundation models for electricity price forecasting and battery arbitrage: Can they replace market-specific forecasting models?
Aug 31, 2026Arkadiusz Lipiecki, Rafał WeronElectricity Price Forecasting Llm-Based Forecasting Methods
Expert-Guided Forecast Editing for Time-Series Foundation Models
Jul 22, 2026Hung Le, Minh Hoang Nguyen, Manh Nguyen +2Time Series Foundation Models Load Forecasting
$t_0$ : A Time-Series Foundation Model for Forecasting with Context
Sep 21, 2026Lucas Meyer, Claudio Sole, Huikan Xiang +6

Aug 31, 2026 · cs.LGJ/K move · Enter open · S save

Foundation models for electricity price forecasting and battery arbitrage: Can they replace market-specific forecasting models?

Arkadiusz Lipiecki, Rafał Weron

Department of Computational Social Science, Wrocław University of Science and Technology, Poland · Department of Operations Research and Business Intelligence, Wrocław University of Science and Technology, Poland · Center for Research in Energy (CoRE), Aarhus University, 8000 Aarhus C, Denmark

Foundation models promise accurate forecasts with little or no task-specific training, but whether they can replace models designed specifically for electricity price forecasting remains unclear. We compare nine variants from five foundation model families, evaluated in zero-shot mode, with two state-of-the-art electricity price forecasting benchmarks in Germany, Poland, and Spain over 2021-2025. Their performance is assessed in terms of point and probabilistic forecasting accuracy, as well as economic value in battery energy storage arbitrage. Only the TabPFN models consistently and significantly outperform the benchmarks across all three markets and all statistical measures. However, this statistical dominance does not translate directly into economic dominance: TabPFN performs best under unlimited bids and riskier quantile-based strategies, whereas the Distributional Deep Neural Network benchmark is more profitable when risk tolerance is lower. Thus, foundation models cannot universally replace market-specific models, and their value depends on both model architecture and the decision problem.

Electricity Price Forecasting Llm-Based Forecasting Methods Tabular Prior-Data Fitted Network Foundation Model

Open paper Ask about it PDF

Abstract

Explore similar work

Foundation models for electricity price forecasting and battery arbitrage: Can they replace market-specific forecasting models?

Expert-Guided Forecast Editing for Time-Series Foundation Models

t0t_0t0​: A Time-Series Foundation Model for Forecasting with Context

$t_0$ : A Time-Series Foundation Model for Forecasting with Context