JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

Nabil Omi; Eric Bae; Chung Yik Edward Yeung; Siddhartha Sen; Ali Farhadi

cs.LGSep 12, 2026

JumpStart Your Policy Learning with Lessons from 160,000 Training Runs

Authors: Nabil Omi, Eric Bae, Chung Yik Edward Yeung, Siddhartha Sen, Ali Farhadi

Organizations: University of Washington · Microsoft

Abstract

Reliable progress in offline policy learning depends on careful reporting, well-tuned baselines, and evaluation across diverse conditions. Prior work has shown that results can be sensitive to reporting choices, hyperparameter tuning, and dataset properties, but these sources of variability have not been systematically investigated together at the scale needed to understand how they shape conclusions. To address this gap, we present a large-scale empirical study of offline reinforcement and imitation learning, training over 160,000 policies across 114 datasets. At this scale, no algorithm dominates: aggregate performance among the strongest methods is often close, but the leaders differ substantially across environments. We find that proper hyperparameter tuning frequently reshuffles perceived algorithm rankings and that benchmark composition can produce conflicting conclusions. We also study hyperparameter sensitivity and transfer across environments, identifying a simple strategy for deriving strong default configurations. We use our findings to develop a dataset-conditioned recommender that provides task-specific algorithm recommendations for practitioners. Finally, we release JumpStart: a resource suite containing every trained policy, per-model scores and hyperparameters, strong baselines across all environments, training and evaluation code, and an extensible website for retrieving, analyzing, and contributing results. Together, these resources aim to make offline policy-learning research more reliable and enable future work beyond the scope of this study.

Figures & tables

Figure 1: Our tuned models substantially outperform published baselines on Minari and change the relative ordering of algorithms. Original paper implementations use the authors’ recommended or closest available hyperparameters. Further details are provided in Appendix B .

Study	Datasets or tasks	Methods / variants	Policies counted	Continuous	Discrete	Trained models	Policy metadata	Training code	Extensible leaderboard
agarwal2020optimistic (Atari)	60 games	6	$1{,}800^{\dagger}$	–	✓	–	–	✓	–
paine2020hyperparameterselectionofflinereinforcement	10 tasks	3	$2{,}560^{\dagger}$	✓	–	–	–	–	–
mandlekar2021matterslearningofflinehuman	8 tasks	6	NR	✓	–	$\triangle$	$\triangle$	✓	–
schweighofer2021dataset	150 (6 envs.)	9	6,750	–	✓	–	–	✓	–
d3rlpy	17 datasets/tasks	12	$1{,}400^{\dagger}$	✓	✓	–	✓	✓	–
tarasov2023revisiting	30 datasets	10	$1{,}200^{\dagger}$	✓	–	–	✓	✓	–

Table 1: Comparison with selected large-scale empirical studies in offline policy learning. Prior-work counts describe the principal offline benchmarks; JumpStart counts include the full hyperparameter-tuning corpus.

Figure 2: An 18-environment subset of the diverse tasks used in the study. A full list of environments and datasets can be found in Appendix A .

	BC	BCP	BCQ	CQL	DT	IQL	ReBRAC	ACT	VQ-BeT	Diffusion	Total
Continuous	11,400	11,400	11,400	11,400	11,400	11,302	11,401	10,830	8,833	10,590	109,956
Discrete	11,400	11,400	8,687	10,021	11,400	–	–	–	–	–	52,908
Total	22,800	22,800	20,087	21,421	22,800	11,302	11,401	10,830	8,833	10,590	162,864

Table 2: Number of trained policies by algorithm and action space.

Algorithm	Top 10%	All trials	Enrichment
ACT	32.8%	35.3%	$-2.5$ pp
BC	33.5%	33.9%	$-0.4$ pp
BCP	41.6%	40.3%	$+1.3$ pp
BCQ	85.4%	85.2%	$+0.2$ pp
CQL	83.3%	78.5%	$+4.8$ pp
DT	70.1%	69.9%	$+0.1$ pp

Table 3: Search-space boundary occupancy. We report the percentage of trials containing at least one hyperparameter within 5% of either search-range boundary. Enrichment is the difference between the top 10% of trials and all trials.

Figure 3: A visual example of optimality gap and the IQM overlaid on a survival chart for behavior cloning on our suite.

Figure 4: 15-dataset subset leaderboard constructions with different Kendall’s

\tau

to the full-environment leaderboard. Even with 15-20 different datasets, it is possible to have rankings in different orders, with the top 3 (gold, silver, bronze outlined) varying substantially. Letters A-J indicate different algorithms.

Figure 5: Tuning progress in continuous-control. Most of the improvement in best observed normalized score occurs within 25–50 trials, whereas agreement with the full-budget algorithm ranking improves more gradually. Confidence intervals are across datasets, where peak performance can vary substantially.

Figure 6: Tuning progress in discrete-control. Agreement with the full-budget ranking increases rapidly during the first 10–25 trials, while the best observed normalized score continues to improve with additional tuning. Confidence intervals are across datasets, where peak performance can vary substantially.

Figure 7: Variability across hyperparameter configurations and random seeds. The left panel reports the median per-cell coefficient of variation for each algorithm, and the right panel summarizes 731 algorithm–dataset cells. Hyperparameter variability exceeds seed-wise variability in 92% of cells, or 83% after removing outlier configurations. Within the top 25% of configurations, however, variability is comparable to that observed across seeds.

Figure 8: Comparison between continuous-control leaderboards constructed from the top-quartile mean of the hyperparameter sweep and from independently seeded runs. IQM and optimality-gap estimates are nearly identical, and the resulting algorithm rankings have Kendall correlations of

\tau=0.94

for IQM and

\tau=0.83

for optimality gap.

Figure 9: Selected benchmark subsets that favor different algorithms. We sample 2,000 subsets at each tested size (

n\in\{10,12,15,20\}

). Among these candidates, subsets containing 12, 10, and 12 environment–dataset pairs place ACT, DT, and BCQ first by a margin, respectively, under both IQM and optimality gap.

Figure 10: Agreement between subset and full-suite leaderboards as a function of subset size. Lines show the mean Kendall’s

\tau

over randomly sampled subsets, and shading shows one standard deviation. Agreement increases and variability across subsets decreases as more environment–dataset pairs are included.

Figure 11: Probability of displaying a winner conflicting with the true top algorithm as the benchmark size increases.

Figure 12: Agreement between algorithm rankings across expert sources. The left and center panels report pairwise Kendall’s $\tau$ under optimality gap and IQM. The right panel reports mean off-diagonal agreement before and after controlling for the log number of datasets in each source. The diamond with whiskers represents the count-matched random baseline, with whiskers representing 1 standard deviation.

Figure 13: Overall algorithm performance on continuous- and discrete-control datasets. We report interquartile mean (IQM; higher is better) and optimality gap with threshold

\gamma=1

(lower is better). Markers denote point estimates, while boxes and whiskers summarize bootstrap uncertainty. ACT leads the continuous-control aggregate, whereas BCP leads the discrete-control aggregate under both metrics.

Figure 14: Continuous-control performance separated by environment collection. The leading algorithm changes across collections: ACT leads on AntMaze and Kitchen, BCQ on PointMaze, DT on Adroit, and IQL or ACT on MuJoCo depending on the aggregation metric.

Figure 15: Continuous-control performance separated by the source of the trajectories in each dataset. The leading algorithm changes from ACT on human and RL + planner data, to DT on cloned data, IQL on RL data, and BCQ on scripted data.

Figure 16: Hyperparameter sensitivity across the 57 continuous-control environments. For each algorithm–environment pair, we compute the absolute Spearman correlation

|\rho_{h}|

between each hyperparameter and trial score and normalize these magnitudes as

s_{h}=|\rho_{h}|/\sum_{j}|\rho_{j}|

. Each vertical bar represents one environment and shows the resulting sensitivity shares. The bottom panel reports the median cumulative share of the one through five most influential hyperparameters within each environment; the displayed value is the median top-one share. Environment ordering is chosen separately within each algorithm for readability.

Figure 17: Hyperparameter sensitivity across the 54 Atari environments, computed as in Figure 16 . The sensitivity profiles are more consistent than in continuous control: learning rate is the most influential hyperparameter for every algorithm, with particularly strong concentration for CQL and BCQ.

Figure 18: Performance of existing d3rlpy defaults, median-transfer defaults, and fully tuned JumpStart top-quartile results. The evaluation contains five folds of five held-out environments, yielding 25 environments and 140 matched algorithm–environment cells. Each fold’s test environments are excluded when constructing its transferred defaults. Error bars are

95\%

bootstrap confidence intervals.

Figure 19: Marginal support for candidate defaults across five folds. Within each fold, its five final test environments are excluded, and each remaining environment is held out in turn. A value is supported when it lies inside the

80\%

highest-density region (HDR) of hyperparameter values among the target environment’s top

25\%

of trials. We compare our median-transfer defaults with the existing d3rlpy defaults and the exact coverage of a value drawn from the configured search distribution. The dotted line denotes the

80\%

probability mass contained by the HDR by construction, not an expected transfer rate. Error bars are

95\%

environment-bootstrap confidence intervals.

Figure 20: Algorithm-recommendation performance under leave-one-environment-family-out validation. The global-mean baseline recommends the fold-local leaderboard to every target, while the learned models condition on observable dataset features and are tuned entirely within each outer training split. Higher top-three overlap and lower best-of-three regret are better. Error bars show

95\%

environment-clustered bootstrap confidence intervals.

Figure 21: Impurity-based feature importance for the random-forest recommender. Expert source and mean episode length provide the strongest predictive signal, followed by observation dimensionality, task domain, and properties of the return distribution.

Figure 22: Transfer to minority expert-source categories. The top row removes each target source from training; the bottom row additionally removes all training datasets from environment families represented in the corresponding test set. “Index” aggregates the 25 datasets across Human, BC-clone, RL-plus-planner, and Scripted sources. Error bars show

95\%

environment-clustered bootstrap confidence intervals.

Figure 23: Recommendation quality as test features are randomly marked unknown. The tree models remain effective with half of the features missing. When fewer than

35\%

are available, the recommender explicitly falls back to the fold-local global leaderboard, causing all methods to converge to the baseline at high missingness.

Figure 24: Reliability of ensemble agreement as a recommendation-confidence measure. Left: empirical top-three frequency against the fraction of trees placing an algorithm in the top three. Right: Brier score and calibration error. Random forest is better calibrated.

Parameter	Range
Learning rate	$\mathrm{LogU}[10^{-5},10^{-2}]$
Weight decay	$\mathrm{LogU}[10^{-4},10^{-1}]$
Gradient clip	$\mathrm{U}[0.1,2]$
Dropout	$\mathrm{U}[0,0.3]$
Hidden units	$\{128,256,512\}\times 2$
Policy type	{det., stoch.}

Table 4: Hyperparameter search spaces for continuous-control algorithms. Fixed parameters are omitted.

Parameter	Range
Learning rate	$\mathrm{LogU}[10^{-5},10^{-2}]$
Weight decay	$\mathrm{LogU}[10^{-4},10^{-1}]$
Gradient clip	$\mathrm{U}[0.1,2]$
Dropout	$\mathrm{U}[0,0.3]$
Hidden units	$\{128,256,512\}\times 2$
Regularization	$\mathrm{U}[0,1]$

Table 5: Hyperparameter search spaces for discrete-control algorithms. Fixed parameters are omitted.

Hyperparameter	$\widetilde{\|\rho\|}$	Share (%)
Hidden units	$0.206$	$22.5$
Learning rate	$0.190$	$19.2$
Dropout	$0.171$	$19.8$
Weight decay	$0.127$	$12.1$
Policy type	$0.094$	$10.7$
Grad-clip norm	$0.092$	$8.7$

Table 8: Hyperparameter sensitivity in continuous control. Within each environment,

\rho

is the Spearman correlation between the hyperparameter and trial score, and Share is its L1-normalized magnitude,

|\rho_{h}|/\sum_{j}|\rho_{j}|

. Entries report the median magnitude and share across 57 environments; rows are ordered by median

|\rho|

. Because each statistic is aggregated independently, the displayed shares need not sum to 100%.

Hyperparameter	$\widetilde{\|\rho\|}$	Share (%)
Learning rate	$0.233$	$25.7$
Dropout	$0.161$	$21.1$
Hidden units	$0.097$	$12.0$
Reg. factor	$0.092$	$11.0$
Weight decay	$0.084$	$10.4$
Grad-clip norm	$0.076$	$9.7$

Table 9: Hyperparameter sensitivity in discrete control (Atari). Within each environment,

\rho

is the Spearman correlation between the hyperparameter and trial score, and Share is its L1-normalized magnitude,

|\rho_{h}|/\sum_{j}|\rho_{j}|

. Entries report the median magnitude and share across 54 environments; rows are ordered by median

|\rho|

. Because each statistic is aggregated independently, the displayed shares need not sum to 100%.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 25 — Figure 25: Per-environment hyperparameter-sensitivity breakdowns for the continuous-control algorithms. Each heatmap reports relative absolute Spearman associations between hyperparameters and trial performance. Zoom in for environment and hyperparameter labels.

Panel of Figure 26 — Figure 26: Hyperparameter-sensitivity breakdowns (continued).

Panel of Figure 27 — Figure 27: Hyperparameter-sensitivity breakdowns (continued).

Panel of Figure 28 — Figure 28: Hyperparameter-sensitivity breakdowns (continued).

Panel of Figure 29 — Figure 29: Hyperparameter-sensitivity breakdowns (continued).

Offline RL

Offline reinforcement learning learns decision-making policies from fixed datasets without interacting with the environment during training.

See all 290 papers

Fully Offline Reinforcement Learning
May 28, 2025Mattie Fellows, Clarisse Wibault, Uljad Berdica +3Bayesian RL Offline RL
Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford
Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness
Sep 29, 2026Zhiwei Wang, Yanxi Chen, Yaliang Li +1Policy Gradient Methods Off-Policy RL
Department of Mathematical Sciences Tsinghua University · Alibaba Group
Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies
Feb 27, 2026Xiang Li, Yuheng Zhang, Nan JiangNatural Policy Gradient Offline RL
Nanjing University · UIUC

May 28, 2025J/K move · Enter open · S save

Fully Offline Reinforcement Learning

Mattie Fellows, Clarisse Wibault, Uljad Berdica, Johannes Forkel+2

Foerster Lab for AI Research (FLAIR), Department of Engineering Science, University of Oxford · Machine Learning Research Group, Department of Engineering Science, University of Oxford

Bayesian RL Offline RL Policy Learning Hyperparameter Optimization Model-Based RL

Open paper AskPDF arXiv

Offline RL (ORL) promises safe and sample-efficient deployment but existing methods rely on undocumented online interactions for hyperparameter tuning and lack reliable fully offline estimates of initial online performance. We introduce SOReL, a fully offline Bayesian model-based RL method that learns a posterior over dynamics, estimates policy value via predictive uncertainty, and enables complete offline hyperparameter selection. We further propose TOReL, which extends this tuning framework to arbitrary model-free and model-based ORL algorithms. We provide a regret analysis showing that Bayesian offline RL achieves the minimax-optimal parametric rate under standard regularity conditions. Together, our methods establish a practical and theoretically grounded framework for fully offline RL.

Abstract

Offline RL

Explore similar work

Fully Offline Reinforcement Learning

Fine-Tuning on Self-Generated and Reward-Weighted Data: Learning Dynamics, Convergence Rates, and Benefits of Off-Policyness

Beyond State-Wise Mirror Descent: Offline Policy Optimization with Parametric Policies