Learning to Bet for Horizon-Aware Anytime-Valid Testing

Ege Onur Taga; Samet Oymak; Shubhanshu Shekhar

stat.MEMar 20, 2026

Learning to Bet for Horizon-Aware Anytime-Valid Testing

Authors: Ege Onur Taga, Samet Oymak, Shubhanshu Shekhar

Organizations: Department of Electrical and Computer Engineering, University of Michigan, Ann Arbor, USA.

Abstract

We develop horizon-aware anytime-valid tests and confidence sequences for bounded means under a strict deadline $N$ . Using the betting/e-process framework, we cast horizon-aware betting as a finite-horizon optimal control problem with state space $(t, \log W_t)$ , where $t$ is the time and $W_t$ is the test martingale value. We first show that in certain interior regions of the state space, policies that deviate significantly from Kelly betting are provably suboptimal, while Kelly betting reaches the threshold with high probability. We then identify sufficient conditions showing that outside this region, more aggressive betting than Kelly can be better if the bettor is behind schedule, and less aggressive can be better if the bettor is ahead. Taken together these results suggest a simple phase diagram in the $(t, \log W_t)$ plane, delineating regions where Kelly, fractional Kelly, and aggressive betting may be preferable. Guided by this phase diagram, we introduce a Deep Reinforcement Learning approach based on a universal Deep Q-Network (DQN) agent that learns a single policy from synthetic experience and maps simple statistics of past observations to bets across horizons and null values. In limited-horizon experiments, the learned DQN policy yields state-of-the-art results.

Figures & tables

Figure 1: A representative phase diagram in the

(t,\log W_{t})

plane, illustrating the qualitative partition of the plane into regimes where Kelly, aggressive, and conservative betting may be preferred under a finite horizon

N

and significance level

\alpha

. Formal results implying such a partition are presented in Section 3 .

Panel of Figure 2 — Figure 2: The phase diagram of optimal actions in $(t,\log W_{t})$ plane demonstrating the change of optimal policies with respect to problem difficulty. $X_{i}$ drawn from a two-component Beta mixture: $X_{i}\sim\mathrm{Beta}(2.4,3.6)$ with probability $1/2$ and $X_{i}\sim\mathrm{Beta}(4.8,7.2)$ with probability $1/2$ , which has mean $\mu_{X}=0.40$ and varying variance across realizations. Experimental details are in Section 4.1 .

Panel of Figure 3 — Figure 3: We compare the probability of rejection by time $t$ with $N=100$ , $\alpha=0.05$ , and null mean $m=0.45$ . We simulate 5,000 length- $N$ sequences $\{X_{i}\}$ from a two-component Beta mixture. We define the hopeless region as the set of states where the optimal policy attains $\mathbb{P}(\text{reject by }N)<10^{-4}$ . In each of (a) and (b) , the right panel shows the DQN’s most frequently selected action, illustrating the learned phase diagram adapts across data-generating distributions. Linear- $\epsilon$ baseline is implemented with $\eta=1/2$ and the Hedge over $\epsilon$ is implemented with varying $\eta$ . All-in refers to $\lambda_{\max}$ . The details are provided in Section 4.2 . (a): $X_{i}\sim\mathrm{Beta}(2.4,3.6)$ with probability $1/2$ and $X_{i}\sim\mathrm{Beta}(4.8,7.2)$ with probability $1/2$ . (b): $X_{i}\sim\mathrm{Beta}(0.4,0.6)$ w.p. $1/2$ and $X_{i}\sim\mathrm{Beta}(0.8,1.2)$ w.p. $1/2$ .

Panel of Figure 4 — Figure 4: We compare confidence sequences under different data-generating distributions with $\alpha=0.05$ following Section 3.1 . DQN produces the narrowest confidence intervals at deadline $t=N=100$ and also yields tight intervals in the early steps. (a) $X_{i}\sim\mathrm{Beta}(2,6)$ w.p. $1/2$ and $X_{i}\sim\mathrm{Beta}(4,12)$ w.p. $1/2$ . (b) $X_{i}\sim\mathrm{Beta}(0.4,0.6)$ w.p. $1/2$ and $X_{i}\sim\mathrm{Beta}(0.8,1.2)$ w.p. $1/2$ . (c) $X_{i}\sim\mathrm{Beta}(6.5,3.5)$ w.p. $1/2$ and $X_{i}\sim\mathrm{Beta}(13,7)$ w.p. $1/2$ .

Panel of Figure 5 — Figure 5: We plot the lower confidence bounds across different Beta distributions and deadlines $N$ . When a method’s curve passes through a point $(x,y)$ , it means that over 5000 repetitions, the lower confidence bound was below $x$ in a fraction $y$ of the runs. The vertical line indicates the mean of the distribution, and the horizontal line indicates the $(1-\alpha/2)$ quantile. Lower curves indicate better performance across methods. Note that in all cases, at $x=\mu_{X}$ we observe $y>1-\alpha/2$ , consistent with the validity of the methods.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Setting
Actions	$\{\widehat{\lambda}/2,\ \widehat{\lambda},\ \lambda_{\mathrm{end}}\}$
Feature dimension	$22$
Network	MLP (256, 128) with ReLU activation
Discount	$\gamma=1$
Loss	Huber Loss
Optimizer	AdamW, lr $=3\cdot 10^{-4}$

Appendix

Table 1: DQN training hyperparameters used in our main runs.

Panel of Figure 6 — Figure 6: Reward-shaping comparison in the Beta-mixture world. Curves show the cumulative rejection probability $P(\tau\leq t)$ , where $\tau$ is the first threshold-crossing time. Both panels use $m=0.45$ , $\mu_{X}=0.40$ , $N=100$ , and $\alpha=0.05$ ; only the concentration $c$ varies. We compare the original DQN, trained with binary deadline reward $\mathbf{1}\{\tau\leq N\}$ , with two reward-shaped variants: DQN-EB, with early-bonus reward $\mathbf{1}\{\tau\leq N\}[1+(1-\tau/N)]$ , and DQN-U, with urgency reward $\mathbf{1}\{\tau\leq N\}(1-\tau/N)$ , along with the existing baselines.

Panel of Figure 7 — Figure 7: We evaluate the DQN policy on out-of-distribution samples drawn from a logit-normal distribution. A random variable $P\in(0,1)$ is said to follow a logit-normal distribution with parameters $(\mu_{X},\sigma_{X})$ if $\mathrm{logit}(P)=\log\!\left(\frac{P}{1-P}\right)\sim\mathcal{N}(\mu_{X},\sigma_{X}^{2}).$ Equivalently, if $X\sim\mathcal{N}(\mu_{X},\sigma_{X}^{2})$ , then $P=\frac{e^{X}}{1+e^{X}}$ has a logit-normal distribution. The policy was trained on the Beta and Beta-mixture families, and is evaluated here on the logit-normal distribution. For (a) , $\mu_{X}=0.30$ and $\sigma_{X}=1.0$ ; for (b) , $\mu_{X}=0.55$ and $\sigma_{X}=1.2$ . In both cases, DQN achieves higher power, even though the evaluation distribution is out of distribution relative to the training distributions.

Panel of Figure 8 — Figure 8: Instead of focusing on Beta/Beta-mixture distributions, we universally train a DQN policy on Bernoulli distribution families (for all $\mu_{X},m\in[0,1]$ ). We evaluate for Bernoulli distributions with $\mu_{X}=0.10,\mu_{X}=0.30$ and $\mu_{X}=0.50$ with $\alpha=0.05$ . Just like for Beta families, DQN learned policy yields power improvements in the Bernoulli case too.

Panel of Figure 9 — Figure 9: Real-data evaluation on CpG methylation beta values from GEO sample GSM838506 (adipose-derived stem cells, donor 1; GSE33896, Illumina HumanMethylation27 BeadChip). (a) The empirical distribution is heavily right-skewed with a bimodal structure. (b) Rejection curves for $H_{0}\colon\mu=0.30$ with randomly $N=100$ subsampled observations and $\alpha=0.05$ . (c) Type-I error curves for $H_{0}\colon\mu=\hat{\mu}_{X}$ with $N=100$ and $\alpha=0.05$ . We conducted 5,000 trials to create both rejection and Type-I error curves.

Panel of Figure 10 — Figure 10: Real-data evaluation on CpG methylation beta values from GEO sample GSM838510 (in vitro induced osteocytes, donor 1; GSE33896). (a) The distribution shape is similar to the undifferentiated stem cells but with a slightly higher mean. (b) Rejection curves for $H_{0}\colon\mu=0.30$ with $N=100$ and $\alpha=0.05$ . (c) Type-I error curves for $H_{0}\colon\mu=\hat{\mu}_{X}$ with $N=100$ and $\alpha=0.05$ . We conducted 5,000 trials to create both rejection and Type-I error curves.

Panel of Figure 11 — Figure 11: Real-data evaluation on CpG methylation beta values from GEO sample GSM838517 (rhabdomyosarcoma cell line CRL-7731; GSE33896). (a) The cancer cell line exhibits a more pronounced bimodal distribution with higher mean and the largest standard deviation among all samples. (b) Rejection curves for $H_{0}\colon\mu=0.39$ with $N=100$ and $\alpha=0.05$ . (c) Type-I error curves for $H_{0}\colon\mu=\hat{\mu}_{X}$ with $N=100$ and $\alpha=0.05$ . We conducted 5,000 trials to create both rejection and Type-I error curves.

Panel of Figure 12 — Figure 12: Real-data evaluation on daily average relative humidity from the USCRN station at Yuma, AZ (desert climate, 2007–2025). (a) The distribution is right-skewed with low mean. (b) Rejection curves for $H_{0}\colon\mu=0.24$ with $N=100$ and $\alpha=0.05$ . (c) Type-I error curves for $H_{0}\colon\mu=\hat{\mu}_{X}$ with $N=100$ and $\alpha=0.05$ . We conducted 5,000 trials to create both rejection and Type-I error curves.

Panel of Figure 13 — Figure 13: Real-data evaluation on daily average relative humidity from the USCRN station at Millbrook, NY (temperate Northeast climate, 2004–2025). (a) The distribution is approximately symmetric with a mild left skew. (b) Rejection curves for $H_{0}\colon\mu=0.65$ with $N=100$ and $\alpha=0.05$ . (c) Type-I error curves for $H_{0}\colon\mu=\hat{\mu}_{X}$ with $N=100$ and $\alpha=0.05$ . We conducted 5,000 trials to create both rejection and Type-I error curves.

Panel of Figure 14 — Figure 14: Real-data evaluation on daily average relative humidity from the USCRN station at Boulder, CO (mountain climate, 2003–2025). (a) The distribution is nearly symmetric with the highest variance among all USCRN stations tested. (b) Rejection curves for $H_{0}\colon\mu=0.52$ with $N=100$ and $\alpha=0.05$ . (c) Type-I error curves for $H_{0}\colon\mu=\hat{\mu}_{X}$ with $N=100$ and $\alpha=0.05$ . We conducted 5,000 trials to create both rejection and Type-I error curves.

Panel of Figure 15 — Figure 15: We compare two DQN policies: one trained with the original 3-action space, consisting of $\widehat{\lambda}_{t}(m)/2$ , $\widehat{\lambda}t(m)$ , and $\lambda{\max}$ , and another with an expanded 9-action space, namely $\widehat{\lambda}_{t}(m)/8$ , $\widehat{\lambda}_{t}(m)/4$ , $\widehat{\lambda}_{t}(m)/2$ , $\widehat{\lambda}_{t}(m)$ , $\widehat{\lambda}_{t}(m)$ , $\widehat{\lambda}_{t}(m)\times\frac{5}{4}$ , $\widehat{\lambda}_{t}(m)\times\frac{6}{4}$ , $\widehat{\lambda}t(m)\times\frac{7}{4}$ , and $\lambda{\max}$ . The distribution in the plot is Beta and $\alpha=0.05$ . We observe that the larger action space yields similar performance but does not lead to an overall improvement. Therefore, for simplicity, we use the 3-action baseline in the paper.

Panel of Figure 16 — Figure 16: We use the same DQN policy as in the paper and evaluate it at longer deadlines, namely $N\in{250,300,350}$ . The data-generating setting is the same Beta-mixture model used in Figure 3 of the main submission. The plots show that DQN maintains improved power under these extended horizons.

Panel of Figure 17 — Figure 17: We note that DQN is flexible enough to accommodate a wide range of $\alpha$ values. Here, we set $\alpha=0.01$ . The data-generating setting is the same Beta-mixture model used in Figure 3 of the main submission. The plots show that DQN also improves power at $\alpha=0.01$ .

Panel of Figure 18 — Figure 18: In all figures, the Type I error remains below $\alpha=0.05$ , as ensured by the anytime-validity of all methods. At the same time, DQN comes closer to the threshold than the competing methods, suggesting that it uses the permitted Type I error budget more efficiently, which in turn helps explain its stronger power.

Figure 19: The validation hit rate in comparison with the training hit rate during DQN training.

Panel of Figure 20 — Figure 20: $X_{i}\sim\mathrm{Beta(conc\mu_{X},conc(1-\mu_{X}))}$ with conc=6, $\mu_{X}=0.4$ , $N=200$ with $\alpha=0.05$ , and $m=0.41$

Panel of Figure 21 — Figure 21: $X_{i}\sim\mathrm{Beta(conc\mu_{X},conc(1-\mu_{X}))}$ with conc=1, $\mu_{X}=0.22$ , $N=100$ with $\alpha=0.05$ , and $m=0.28$

Panel of Figure 22 — Figure 22: $X_{i}\sim\mathrm{Beta(conc\mu_{X},conc(1-\mu_{X}))}$ with conc=0.5, $\mu_{X}=0.71$ , $N=100$ with $\alpha=0.05$ , and $m=0.66$

Explore similar work

KellyBench: A Benchmark for Long-Horizon Sequential Decision Making
Apr 30, 2026Thomas Grady, Kip Parker, Iliyan Zarov +3Sequential Decision Making Long-Horizon Agent Evaluation
General Reasoning, Inc.
Multi-Armed Sequential Hypothesis Testing by Betting
Mar 18, 2026Ricardo J. Sandoval, Ian Waudby-Smith, Michael I. JordanSequential Hypothesis Testing Statistical Hypothesis Testing
University of California, Berkeley · École Normale Supérieure & Inria, Paris
Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation
May 1, 2026Andrzej Ruszczynski, Tiangang ZhangMarkov Decision Processes Reinforcement Learning
Department of Management Science and Information Systems, Rutgers University

Apr 30, 2026J/K move · Enter open · S save

KellyBench: A Benchmark for Long-Horizon Sequential Decision Making

Thomas Grady, Kip Parker, Iliyan Zarov, Henry Course+2

General Reasoning, Inc.

Sequential Decision Making Long-Horizon Agent Evaluation AI Agent Benchmarks LLM Agents Financial Decision-Making

Open paper AskPDF arXiv

Language models are saturating benchmarks for procedural tasks with narrow objectives. But they are increasingly being deployed in long-horizon, non-stationary environments with open-ended goals. In this paper we introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets. Agents are placed in a sequential simulation of the 2023-24 English Premier League season and tasked with maximising their long-term bankroll growth. They are given detailed historical data, including advanced statistics, lineups, and public odds. To succeed they must build machine learning models, identify edge in public markets, and adapt as the environment changes over time. We find that all frontier models evaluated lose money on average over the course of the season for five seeds. The best performing model achieves an average return of -8%, and many models experiencing ruin across seeds. To judge strategy sophistication, we use a human expert rubric to grade each model and find their approaches to be unsophisticated compared to human baselines; Claude Opus 4.6 achieves a rubric score of 26.5%, which means there is significant room for improvement. KellyBench is available as an open-access API endpoint at https://openreward.ai/GeneralReasoning/KellyBench.

Abstract

Explore similar work

KellyBench: A Benchmark for Long-Horizon Sequential Decision Making

Multi-Armed Sequential Hypothesis Testing by Betting

Reinforcement Learning with Markov Risk Measures and Multipattern Risk Approximation