We develop horizon-aware anytime-valid tests and confidence sequences for bounded means under a strict deadline N. Using the betting/e-process framework, we cast horizon-aware betting as a finite-horizon optimal control problem with state space (t,logWt), where t is the time and Wt is the test martingale value. We first show that in certain interior regions of the state space, policies that deviate significantly from Kelly betting are provably suboptimal, while Kelly betting reaches the threshold with high probability. We then identify sufficient conditions showing that outside this region, more aggressive betting than Kelly can be better if the bettor is behind schedule, and less aggressive can be better if the bettor is ahead. Taken together these results suggest a simple phase diagram in the (t,logWt) plane, delineating regions where Kelly, fractional Kelly, and aggressive betting may be preferable. Guided by this phase diagram, we introduce a Deep Reinforcement Learning approach based on a universal Deep Q-Network (DQN) agent that learns a single policy from synthetic experience and maps simple statistics of past observations to bets across horizons and null values. In limited-horizon experiments, the learned DQN policy yields state-of-the-art results.
Figures & tables
Figure 1: A representative phase diagram in the (t,logWt) plane, illustrating the qualitative partition of the plane into regimes where Kelly, aggressive, and conservative betting may be preferred under a finite horizon N and significance level α . Formal results implying such a partition are presented in Section 3 .
Figure 2: The phase diagram of optimal actions in (t,logWt) plane demonstrating the change of optimal policies with respect to problem difficulty. Xi drawn from a two-component Beta mixture: Xi∼Beta(2.4,3.6) with probability 1/2 and Xi∼Beta(4.8,7.2) with probability 1/2 , which has mean μX=0.40 and varying variance across realizations. Experimental details are in Section 4.1 .
Figure 3: We compare the probability of rejection by time t with N=100 , α=0.05 , and null mean m=0.45 . We simulate 5,000 length- N sequences {Xi} from a two-component Beta mixture. We define the hopeless region as the set of states where the optimal policy attains P(reject by N)<10−4 . In each of (a) and (b) , the right panel shows the DQN’s most frequently selected action, illustrating the learned phase diagram adapts across data-generating distributions. Linear- ϵ baseline is implemented with η=1/2 and the Hedge over ϵ is implemented with varying η . All-in refers to λmax . The details are provided in Section 4.2 . (a): Xi∼Beta(2.4,3.6) with probability 1/2 and Xi∼Beta(4.8,7.2) with probability 1/2 . (b): Xi∼Beta(0.4,0.6) w.p. 1/2 and Xi∼Beta(0.8,1.2) w.p. 1/2 .
Figure 4: We compare confidence sequences under different data-generating distributions with α=0.05 following Section 3.1 . DQN produces the narrowest confidence intervals at deadline t=N=100 and also yields tight intervals in the early steps. (a) Xi∼Beta(2,6) w.p. 1/2 and Xi∼Beta(4,12) w.p. 1/2 . (b) Xi∼Beta(0.4,0.6) w.p. 1/2 and Xi∼Beta(0.8,1.2) w.p. 1/2 . (c) Xi∼Beta(6.5,3.5) w.p. 1/2 and Xi∼Beta(13,7) w.p. 1/2 .
Figure 5: We plot the lower confidence bounds across different Beta distributions and deadlines N . When a method’s curve passes through a point (x,y) , it means that over 5000 repetitions, the lower confidence bound was below x in a fraction y of the runs. The vertical line indicates the mean of the distribution, and the horizontal line indicates the (1−α/2) quantile. Lower curves indicate better performance across methods. Note that in all cases, at x=μX we observe y>1−α/2 , consistent with the validity of the methods.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Setting
Actions
{λ/2,λ,λend}
Feature dimension
22
Network
MLP (256, 128) with ReLU activation
Discount
γ=1
Loss
Huber Loss
Optimizer
AdamW, lr =3⋅10−4
Appendix
Table 1: DQN training hyperparameters used in our main runs.
Figure 6: Reward-shaping comparison in the Beta-mixture world. Curves show the cumulative rejection probability P(τ≤t) , where τ is the first threshold-crossing time. Both panels use m=0.45 , μX=0.40 , N=100 , and α=0.05 ; only the concentration c varies. We compare the original DQN, trained with binary deadline reward 1{τ≤N} , with two reward-shaped variants: DQN-EB, with early-bonus reward 1{τ≤N}[1+(1−τ/N)] , and DQN-U, with urgency reward 1{τ≤N}(1−τ/N) , along with the existing baselines.
Figure 7: We evaluate the DQN policy on out-of-distribution samples drawn from a logit-normal distribution. A random variable P∈(0,1) is said to follow a logit-normal distribution with parameters (μX,σX) if logit(P)=log(1−PP)∼N(μX,σX2). Equivalently, if X∼N(μX,σX2) , then P=1+eXeX has a logit-normal distribution. The policy was trained on the Beta and Beta-mixture families, and is evaluated here on the logit-normal distribution. For (a) , μX=0.30 and σX=1.0 ; for (b) , μX=0.55 and σX=1.2 . In both cases, DQN achieves higher power, even though the evaluation distribution is out of distribution relative to the training distributions.
Figure 8: Instead of focusing on Beta/Beta-mixture distributions, we universally train a DQN policy on Bernoulli distribution families (for all μX,m∈[0,1] ). We evaluate for Bernoulli distributions with μX=0.10,μX=0.30 and μX=0.50 with α=0.05 . Just like for Beta families, DQN learned policy yields power improvements in the Bernoulli case too.
Figure 9: Real-data evaluation on CpG methylation beta values from GEO sample GSM838506 (adipose-derived stem cells, donor 1; GSE33896, Illumina HumanMethylation27 BeadChip). (a) The empirical distribution is heavily right-skewed with a bimodal structure. (b) Rejection curves for H0:μ=0.30 with randomly N=100 subsampled observations and α=0.05 . (c) Type-I error curves for H0:μ=μ^X with N=100 and α=0.05 . We conducted 5,000 trials to create both rejection and Type-I error curves.
Figure 10: Real-data evaluation on CpG methylation beta values from GEO sample GSM838510 (in vitro induced osteocytes, donor 1; GSE33896). (a) The distribution shape is similar to the undifferentiated stem cells but with a slightly higher mean. (b) Rejection curves for H0:μ=0.30 with N=100 and α=0.05 . (c) Type-I error curves for H0:μ=μ^X with N=100 and α=0.05 . We conducted 5,000 trials to create both rejection and Type-I error curves.
Figure 11: Real-data evaluation on CpG methylation beta values from GEO sample GSM838517 (rhabdomyosarcoma cell line CRL-7731; GSE33896). (a) The cancer cell line exhibits a more pronounced bimodal distribution with higher mean and the largest standard deviation among all samples. (b) Rejection curves for H0:μ=0.39 with N=100 and α=0.05 . (c) Type-I error curves for H0:μ=μ^X with N=100 and α=0.05 . We conducted 5,000 trials to create both rejection and Type-I error curves.
Figure 12: Real-data evaluation on daily average relative humidity from the USCRN station at Yuma, AZ (desert climate, 2007–2025). (a) The distribution is right-skewed with low mean. (b) Rejection curves for H0:μ=0.24 with N=100 and α=0.05 . (c) Type-I error curves for H0:μ=μ^X with N=100 and α=0.05 . We conducted 5,000 trials to create both rejection and Type-I error curves.
Figure 13: Real-data evaluation on daily average relative humidity from the USCRN station at Millbrook, NY (temperate Northeast climate, 2004–2025). (a) The distribution is approximately symmetric with a mild left skew. (b) Rejection curves for H0:μ=0.65 with N=100 and α=0.05 . (c) Type-I error curves for H0:μ=μ^X with N=100 and α=0.05 . We conducted 5,000 trials to create both rejection and Type-I error curves.
Figure 14: Real-data evaluation on daily average relative humidity from the USCRN station at Boulder, CO (mountain climate, 2003–2025). (a) The distribution is nearly symmetric with the highest variance among all USCRN stations tested. (b) Rejection curves for H0:μ=0.52 with N=100 and α=0.05 . (c) Type-I error curves for H0:μ=μ^X with N=100 and α=0.05 . We conducted 5,000 trials to create both rejection and Type-I error curves.
Figure 15: We compare two DQN policies: one trained with the original 3-action space, consisting of λt(m)/2 , λt(m) , and λmax , and another with an expanded 9-action space, namely λt(m)/8 , λt(m)/4 , λt(m)/2 , λt(m) , λt(m) , λt(m)×45 , λt(m)×46 , λt(m)×47 , and λmax . The distribution in the plot is Beta and α=0.05 . We observe that the larger action space yields similar performance but does not lead to an overall improvement. Therefore, for simplicity, we use the 3-action baseline in the paper.
Figure 16: We use the same DQN policy as in the paper and evaluate it at longer deadlines, namely N∈250,300,350 . The data-generating setting is the same Beta-mixture model used in Figure 3 of the main submission. The plots show that DQN maintains improved power under these extended horizons.
Figure 17: We note that DQN is flexible enough to accommodate a wide range of α values. Here, we set α=0.01 . The data-generating setting is the same Beta-mixture model used in Figure 3 of the main submission. The plots show that DQN also improves power at α=0.01 .
Figure 18: In all figures, the Type I error remains below α=0.05 , as ensured by the anytime-validity of all methods. At the same time, DQN comes closer to the threshold than the competing methods, suggesting that it uses the permitted Type I error budget more efficiently, which in turn helps explain its stronger power.
Figure 19: The validation hit rate in comparison with the training hit rate during DQN training.
Figure 20: Xi∼Beta(concμX,conc(1−μX)) with conc=6, μX=0.4 , N=200 with α=0.05 , and m=0.41
Figure 21: Xi∼Beta(concμX,conc(1−μX)) with conc=1, μX=0.22 , N=100 with α=0.05 , and m=0.28
Figure 22: Xi∼Beta(concμX,conc(1−μX)) with conc=0.5, μX=0.71 , N=100 with α=0.05 , and m=0.66
Language models are saturating benchmarks for procedural tasks with narrow objectives. But they are increasingly being deployed in long-horizon, non-stationary environments with open-ended goals. In this paper we introduce KellyBench, an environment for evaluating sequential decision-making in sports betting markets. Agents are placed in a sequential simulation of the 2023-24 English Premier League season and tasked with maximising their long-term bankroll growth. They are given detailed historical data, including advanced statistics, lineups, and public odds. To succeed they must build machine learning models, identify edge in public markets, and adapt as the environment changes over time. We find that all frontier models evaluated lose money on average over the course of the season for five seeds. The best performing model achieves an average return of -8%, and many models experiencing ruin across seeds. To judge strategy sophistication, we use a human expert rubric to grade each model and find their approaches to be unsophisticated compared to human baselines; Claude Opus 4.6 achieves a rubric score of 26.5%, which means there is significant room for improvement. KellyBench is available as an open-access API endpoint at https://openreward.ai/GeneralReasoning/KellyBench.
We consider a variant of sequential testing by betting where, at each time step, the statistician is presented with multiple data sources (arms) and obtains data by choosing one of the arms. We consider the composite global null hypothesis P that all arms are null in a certain sense (e.g. all dosages of a treatment are ineffective) and we are interested in rejecting P in favor of a composite alternative Q where at least one arm is non-null (e.g. there exists an effective treatment dosage). We posit an optimality desideratum that we describe informally as follows: even if several arms are non-null, we seek e-processes and sequential tests whose performance are as strong as the ones that have oracle knowledge about which arm generates the most evidence against P. Formally, we generalize notions of log-optimality and expected rejection time optimality to more than one arm, obtaining matching lower and upper bounds for both. A key technical device in this optimality analysis is a modified upper-confidence-bound-like algorithm for unobservable but sufficiently "estimable" rewards. In the design of this algorithm, we derive nonasymptotic concentration inequalities for optimal wealth growth rates in the sense of Kelly [1956]. These may be of independent interest.
Ricardo J. Sandoval, Ian Waudby-Smith, Michael I. Jordan
University of California, Berkeley · École Normale Supérieure & Inria, Paris
For a risk-averse finite-horizon Markov Decision Problem, we introduce a special class of Markov coherent risk measures, called mini-batch measures. We also define the class of multipattern risk-averse problems that generalizes the class of linear systems. We use both concepts in a feature-based Q-learning method with multipattern Q-factor approximation and we prove a high-probability regret bound of O(H2NHK), where H is the horizon, N is the mini-batch size, and K is the number of episodes. We also propose an economical version of the Q-learning method that streamlines the policy evaluation (backward) step. The theoretical results are illustrated on a stochastic assignment problem and a short-horizon multi-armed bandit problem.
Andrzej Ruszczynski, Tiangang Zhang
Department of Management Science and Information Systems, Rutgers University