AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

Liqin Ye; Rongzhi Zhang; Xin Liu; Zheng Li; Yuan He; Qingyu Yin; Priyanka Nigam; Chao Zhang

cs.LGOct 15, 2026

AlphaBet: Reinforcement Learning for LLM Event Forecasting Agents

Authors: Liqin Ye, Rongzhi Zhang, Xin Liu, Zheng Li, Yuan He, Qingyu Yin, Priyanka Nigam, Chao Zhang

Organizations: Georgia Institute of Technology · Amazon

Abstract

Forecasting an unresolved event requires finding relevant evidence and deciding how strongly it supports each outcome. Yet outcome-based training of Large Language Model (LLM) forecasters often begins with preassembled evidence at a single prediction date, leaving evidence acquisition outside the learning loop. We introduce AlphaBet, an agentic reinforcement learning (RL) framework that trains a single LLM policy to search, reason, and report probabilities across multiple pre-resolution dates. Each dated forecast is scored by a proper scoring rule against the eventual outcome, allowing one resolved event to supervise decisions under different information states. We decompose forecasting regret into an evidence-acquisition gap and a probability-reporting gap, and show that Group Relative Policy Optimization (GRPO)'s standard-deviation normalization can distort the reporting target under multi-class proper scores. On GLM-4.5-Air, AlphaBet reduces the Brier score from 0.731 to 0.626 on Forecast-Dojo and from 0.612 to 0.528 on FutureX. Controlled ablations attribute these gains to multi-date training and show that they come from improvements in both information acquisition and probability reporting. Our empirical results validate that GRPO's std normalization degrades proper-scoring-rule performance while leaving top-outcome accuracy largely unchanged.

Figures & tables

Figure 1: AlphaBet improves forecasting on Forecast-Dojo and FutureX (March 2026 to May 2026). Dashed arrows connect GLM-4.5-Air to AlphaBet

{}_{\text{{\color[rgb]{0,0,0}memory-free}}}

. The lower right corresponds to higher accuracy and lower Brier score. Accuracy gains are in percentage points, and Brier reductions are relative to the base model. Table 4.1 reports full results and confidence intervals.

Figure 2: Overview of AlphaBet . One shared policy searches, reads, and reasons over the corpus available at each forecast date before reporting probabilities. Each report is scored against the eventual outcome, which is hidden during rollout. Training uses per-date feedback with mean-centered GRPO; a notebook optionally carries context between dates.

	Forecast-Dojo			FutureX
Model	Acc. (%) $\uparrow$	Brier $\downarrow$	$-$ Info- $\alpha$ $\downarrow$	Acc. (%) $\uparrow$	Brier $\downarrow$
Uniform Reference

Table 1: Main results. Bold and underlined values mark the best and second-best point estimates, respectively, within the GLM-4.5-Air block. Gray ranges give 95% confidence intervals (event-clustered bootstrap for Forecast-Dojo; item bootstrap for FutureX).

Probabilistic Forecasting

Probabilistic forecasting predicts distributions or events with quantified uncertainty to support reliable decisions under uncertainty.

See all 230 papers

Reinforcement Learning for LLM-based Event Forecasting
Jun 14, 2026Amit Arnold LevyEvent Time Prediction RL for Language Models
University of Oxford
Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents
Sep 24, 2026Liqin Ye, Haorui Wang, Fardin Ahmed +8LLM Agent Evaluation Forecasting Benchmarks
Georgia Institute of Technology · Amazon · University of Florida
Do Your Own Research: Learning to Forecast by Learning to Search
Oct 1, 2026Yusuf Afifi, Artur Kiulian, Anton Polishko +3Agentic RL Forecasting Benchmarks
Future Principle

Jun 14, 2026J/K move · Enter open · S save

Reinforcement Learning for LLM-based Event Forecasting

Amit Arnold Levy

University of Oxford

Event Time Prediction RL for Language Models Probabilistic Forecasting RL Fine-Tuning

Open paper AskPDF arXiv

We use Group Relative Policy Optimization (GRPO), a recently devised sample and memory efficient reinforcement learning method, to finetune pretrained LLMs in the range of 1.5B to 14B parameters equipped with the ability to get current information through the use of a Wikipedia revisions tool, or news summaries, to forecast real events beyond the knowledge cutoff of the LLM, as well as problems made to simulate different aspects of the dynamics of that training. We use the results of these experiments to comment on the scaling capability of LLMs for forecasting, as well as classify how judgmental forecasting fits into the verifiable/unverifiable domain taxonomy, considering the impact of the inherent aleatoric uncertainty when forecasting future events (e.g. the roll of a die). As a result of the GRPO training, we manage to bring a 1.5B parameter transformer (Qwen 2.5 1.5B) to forecasting performance superior to Claude Sonnet 3.5 over the same dataset as measured by cross entropy from the market agreed probabilities. We also discuss various dead ends on the path to this result.

Abstract

Probabilistic Forecasting

Explore similar work

Reinforcement Learning for LLM-based Event Forecasting

Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agents

Do Your Own Research: Learning to Forecast by Learning to Search