Delphos: A reinforcement learning framework for assisting discrete choice model specification

Gabriel Nova; Stephane Hess; Sander van Cranenburgh

econ.GNJun 6, 2025

Delphos: A reinforcement learning framework for assisting discrete choice model specification

Authors: Gabriel Nova, Stephane Hess, Sander van Cranenburgh

Organizations: CityAI Lab, Transport and Logistics Group, Delft University of Technology, The Netherlands · Choice Modelling Centre - Institute for Transport Studies, University of Leeds, England

Abstract

We introduce Delphos, a deep reinforcement learning framework for assisting discrete choice model specification process. Delphos aims to support the modeller by providing automated, data-driven suggestions for model specifications, thereby reducing the effort required to develop and refine utility functions. Delphos conceptualises model specification as a sequential decision-making problem, inspired by the way human choice modellers iteratively construct models through a series of reasoned specification decisions. In this setting, an agent learns to specify candidate model specifications by choosing a sequence of modelling actions, such as adding alternative specific constants, accommodating both generic and alternative-specific taste parameters, applying non-linear transformations to attributes, and including interactions with covariates. Each resulting candidate model is estimated and evaluated using a reward function defined by the modeller, which can reflect statistical model fit as well as behavioural expectations. Specifically, Delphos uses a Deep Q-Network to learn how individual specification decisions contribute to the eventual quality of the resulting model and, in turn, which sequences of modelling decisions tend to produce well-performing candidates. We evaluate Delphos on both simulated and empirical datasets using alternative reward functions. In simulated cases, learning curves, Q-value patterns, and performance metrics show that Delphos learns effective specification strategies while exploring only a small fraction of the feasible modelling space. We further apply the framework to two empirical datasets to benchmark and demonstrate its practical use. These experiments illustrate the ability of Delphos to generate competitive, behaviourally plausible models and highlight the potential of this adaptive, learning-based framework to assist the model specification process.

Figures & tables

Figure 1: Classical RL framework. Adaptation from Sutton & Barto ((2018))

Figure 2: Framework for assisting discrete choice model specification. While Delphos encodes the current specification, selects a modelling action, and submits the model for estimation, the environment returns modelling outcomes as a reward signal. Transitions are stored in a replay buffer and used to update the policy and target networks.

Modelling outcome	Formulation	Bound
Log-likelihood	$\displaystyle\max\left(0,\frac{LL-LL_{0}}{N}\right)$	$\displaystyle[0,\log J]$
AIC	$\displaystyle\max\left(0,\frac{LL-LL_{0}}{N}-\frac{K}{N}\right)$	$\displaystyle\left[0,\log J-\frac{K}{N}\right]$
BIC	$\displaystyle\max\left(0,\frac{LL-LL_{0}}{N}-\frac{K\log N}{2N}\right)$	$\displaystyle\left[0,\log J-\frac{K\log N}{2N}\right]$
McFadden’s pseudo- $R^{2}$	$\displaystyle\max\left(0,1-\frac{LL}{LL_{0}}\right)$	$\displaystyle[0,1]$
Adjusted McFadden’s pseudo- $R^{2}$	$\displaystyle\max\left(0,1-\frac{LL-K}{LL_{0}}\right)$	$\displaystyle\left[0,1+\frac{K}{N\log J}\right]$

Table 1: Scaling of the modelling outcomes used in the reward function.

Figure 3: Agent’s sequential decision-making process. Nodes represent internal states (current specifications). Dashed arrows indicate feasible actions, while solid arrows show the actions selected by the agent until termination.

Experiment	$1A$		$1B$		$1C$
Variable	Transf.	Taste	Transf.	Taste	Transf.	Taste	Covariate
ASC 1	Linear		Linear		Linear		cov 1
ASC 2	Linear		Linear		Linear		cov 1
$x_{1}$	Linear	Generic	Log	Specific	Linear	Specific
$x_{2}$	Log	Generic	Box-Cox	Generic	Log	Specific	cov 2
$x_{3}$	Linear	Generic	Linear	Specific	Linear	Generic

Table 2: Utility specifications and modelling spaces for simulated choice datasets (

1A

–

1C

Figure 4: Learning process on simulated dataset $1A$ . The upper panel displays the learning curve, showing the rolling mean and min–max range of the reward signal. The lower panel reports rolling novelty, measured as the share of previously unseen specifications.

Figure 5: Action selection distribution in simulated dataset $1A$ . Bars show the average frequency with which each action is selected in the early and late training across runs. Numbers report the average $\bar{\rho}^{2}$ for specifications in which the action was used. The full list of actions and their corresponding definitions is provided in the Appendix E .

Exp	Model space	Models estimated by agent	Converged models	First recovery	Reward (last 100)	$\Delta$ Reward
1A	8.19 $\times 10^{3}$	1.89 $\times 10^{3}$	97.42%	1.46 $\times 10^{3}$	0.611	-0.002
1B	8.19 $\times 10^{3}$	2.34 $\times 10^{3}$	98.45%	1.63 $\times 10^{3}$	0.753	-0.009
1C	1.59 $\times 10^{6}$	3.92 $\times 10^{3}$	78.48%	3.73 $\times 10^{3}$	0.654	-0.023

Table 3: Modelling size, exploration coverage, and performance on simulated experiments

	Search Metrics			In-Sample AIC			Out-of-Sample LL
	#Param.	Converged	Time (min)	Mean ( $\pm$ std)	Best	Worst	Best	Worst
Delphos	22 ( $\pm$ 8)	645 ( $\pm$ 65)	24 ( $\pm$ 3)	7,957 ( $\pm$ 394)	7,454	9,128	-879	-1,073
SA	13 ( $\pm$ 7)	1,344 ( $\pm$ 21)	9 ( $\pm$ 2)	8,431 ( $\pm$ 472)	7,463	9,410	-913	-1,163
VNS	8 ( $\pm$ 4)	1,443 ( $\pm$ 14)	104 ( $\pm$ 23)	8,843 ( $\pm$ 385)	7,744	9,401	-948	-1,163

Table 4: Search space performance across 10 independent seeds.

Figure 6: Evolution of the best-so-far AIC across methods on the Swissmetro dataset.

Figure 7: Pareto front of Delphos, SA, VNS on the Swissmetro dataset.Bold lines indicate the Pareto front of the best-performing seed for each method, while shaded regions span the area between the best and worst seeds across the 10 independent runs.

Figure 8: Pareto front of Delphos candidates on the DECISIONS dataset. Grey dots show all converged specifications proposed during training. Stars indicate the non-dominated candidates, and circles show the best AIC and best log-likelihood specifications.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	$1A$		$1B$		$1C$
ASC 1	$\beta_{\text{ASC1}}$	$-0.95$	$\beta_{\text{ASC1}}$	$-0.95$	$\beta_{\text{ASC1,cov1}=0}$	$-0.90$
					$\beta_{\text{ASC1,cov1}=1}$	$0.43$
ASC 2	$\beta_{\text{ASC2}}$	$0.84$	$\beta_{\text{ASC2}}$	$0.84$	$\beta_{\text{ASC2,cov1}=0}$	$-0.83$
					$\beta_{\text{ASC2,cov1}=1}$	$0.43$
$x_{1}$	$\beta_{1}$	$-0.72$	$\beta_{1}$	$-1.65$	$\beta_{1,1}$	$-0.50$
					$\beta_{1,2}$	$-0.40$

Appendix

Table 5: True parameter values for each simulated case

Hyperparameter	Level values
Number of hidden layers $(\mathbf{L})$	$\{2,\,3\}$
Hidden units per layer $(\mathbf{U})$	$\{64,\,128,\,256,\,512\}$
Learning rate $(\boldsymbol{\alpha})$	$\{1e-3,\,5e-4,\,1e-4\}$
Discount factor $(\boldsymbol{\gamma})$	$\{0.95,\,0.99\}$
Replay buffer size $(\mathbf{BS})$	$\{5{,}000,\,10{,}000,\,20{,}000\}$
Target network update frequency $(\mathbf{TU})$	$\{5,\,10,\}$

Appendix

Table 6: Hyperparameter ranges.

Exp	L	U	$\boldsymbol{\alpha}$	$\boldsymbol{\gamma}$	BS	TU	E	AUC	Final–100
$1A$ - conf. 1	2	256	1e–4	0.99	5k	10	50k	0.552	0.612
$1A$ - conf. 2	3	256	1e–4	0.99	5k	5	50k	0.553	0.612
$1A$ - conf. 3	2	64	1e–4	0.99	10k	10	5k	0.538	0.611
S 1 - conf. 4	2	64	1e–4	0.99	10k	10	10k	0.548	0.611
$1A$ - conf. 5	2	256	1e–4	0.95	10k	5	30k	0.538	0.610
$1B$ - conf. 1	2	128	1e–4	0.99	10k	10	50k	0.708	0.753

Appendix

Table 7: Performance of sampled hyperparameter configurations.

Figure 9: Learning curve showing the reward signal ( $\bar{\rho}^{2}$ ) across 10 runs

Figure 10: Learning curve showing the reward signal ( $\bar{\rho}^{2}$ ) across 10 runs

Action ID	Operation	Variable	Transformation Type	Action
0	terminate	0	-	( end )
1	add	1	linear	( add , 1, linear )
2	change	1	linear	( change , 1, linear )
3	add	1	log	( add , 1, log )
4	change	1	log	( change , 1, log )
5	add	1	box-cox	( add , 1, box-cox )

Appendix

Table 8: Action space for simulated experiment

1A

Parameter	Transformation	Covariate	Value (Rob. s.e.)
$\beta_{\text{train},\,\text{age}=1}$		age = 1	$1.483\,(0.706)$
$\beta_{\text{train},\,\text{age}=2}$		age = 2	$0.515\,(0.371)$
$\beta_{\text{train},\,\text{age}=3}$		age = 3	$0.232\,(0.399)$
$\beta_{\text{train},\,\text{age}=4}$		age = 4	$0.746\,(0.482)$
$\beta_{\text{train},\,\text{age}=5}$		age = 5	$2.168\,(0.499)$
$\beta_{\text{sm},\,\text{age}=1}$		age = 1	$0.682\,(0.879)$

Appendix

Table 9: Best-AIC specification on Swissmetro dataset

Parameter	Transformation	Covariate	Value (Rob. s.e.)
$\beta_{\text{car},\,\text{edu}=1}$		edu = 1	$-0.521\,(0.394)$
$\beta_{\text{car},\,\text{edu}=2}$		edu = 2	$-0.594\,(0.348)$
$\beta_{\text{car},\,\text{edu}=3}$		edu = 3	$-0.887\,(0.248)$
$\beta_{\text{car},\,\text{edu}=4}$		edu = 4	$-1.654\,(0.201)$
$\beta_{\text{car},\,\text{edu}=5}$		edu = 5	$-1.661\,(0.237)$
$\beta_{\text{car},\,\text{edu}=6}$		edu = 6	$-2.038\,(0.275)$

Appendix

Table 10: Best-LL_Nparams specification on Decisions dataset.

Discrete Choice Modeling

Discrete choice modeling studies how people select among alternatives and how preferences can be estimated from observed choices.

See all 44 papers

Multitask Reinforcement Learning for Assisting Choice Model Specification
Sep 16, 2026Gabriel Nova, Stephane Hess, Sander Van CranenburghDiscrete Choice Modeling Deep Sets
CityAI Lab, Transport and Logistics Group, Delft University of Technology, The Netherlands · Institute for Transport Studies and Choice Modelling Centre, University of Leeds, UK
Bayesian Deep Learning for Discrete Choice
May 23, 2025Daniel F. Villarraga, Ricardo A. DazianoBayesian Neural Networks Bayesian Inference
School of Civil and Environmental Engineering, Cornell University, 220 Hollister Hall, Ithaca, NY 14853, USA
An analysis of machine learning approaches for enhancing decision-making in complex discrete choice tasks
Jul 30, 2026Sheng Lun Christine Cao, Destenie Nock, Alex DavisDiscrete Choice Modeling User Preference Modeling
Department of Electrical and Software Engineering, University of Calgary, Calgary, AB, Canada · Department of Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Civil Engineering, Carnegie Mellon University, Pittsburgh, PA, USA +1

Sep 16, 2026J/K move · Enter open · S save

Multitask Reinforcement Learning for Assisting Choice Model Specification

Gabriel Nova, Stephane Hess, Sander Van Cranenburgh

CityAI Lab, Transport and Logistics Group, Delft University of Technology, The Netherlands · Institute for Transport Studies and Choice Modelling Centre, University of Leeds, UK

Discrete Choice Modeling Deep Sets Multi-Task RL

Open paper AskPDF arXiv

Discrete choice model specification is a time-consuming task in which modellers often specify and estimate multiple models while balancing goodness-of-fit, parsimony, and behavioural plausibility. We present Delphos, a multitask reinforcement learning framework that learns transferable specification strategies across transport choice datasets. Delphos frames model specification as a sequential decision-making problem in which it applies a sequence of modelling actions and receives feedback from an estimation environment based on model performance and convergence. To transfer modelling decisions across datasets with different sets of variables, Delphos represents utility specifications as sets of modelling terms using a DeepSet-Q architecture, allowing a shared specification policy to learn across multiple datasets. Trained on nine transport choice datasets, Delphos consistently outperforms independently trained single-task agents, indicating that sharing modelling experience improves learning efficiency and helps identify promising sequences of modelling decisions with fewer unsuccessful estimation attempts. When applied without further training to the unseen Swissmetro and Decisions datasets, the same agent identifies competitive specifications in less than 20 minutes on a standard CPU. It achieves a higher log-likelihood per observation than the VNS metaheuristic on Swissmetro and performance comparable to a published MNL specification developed by expert modellers on Decisions. These findings show that accumulating and reusing modelling experience enables Delphos to function as an intelligent assistant for discrete choice model specification. It reduces manual trial-and-error while allowing modellers to retain control over model diagnosis, refinement, and final selection.

Abstract

Discrete Choice Modeling

Explore similar work

Multitask Reinforcement Learning for Assisting Choice Model Specification

Bayesian Deep Learning for Discrete Choice

An analysis of machine learning approaches for enhancing decision-making in complex discrete choice tasks