Delphos: A reinforcement learning framework for assisting discrete choice model specification

Gabriel Nova; Stephane Hess; Sander van Cranenburgh

econ.GNJun 6, 2025

Delphos: A reinforcement learning framework for assisting discrete choice model specification

Authors: Gabriel Nova, Stephane Hess, Sander van Cranenburgh

Organizations: CityAI Lab, Transport and Logistics Group, Delft University of Technology, The Netherlands · Choice Modelling Centre - Institute for Transport Studies, University of Leeds, England

Abstract

We introduce Delphos, a deep reinforcement learning framework for assisting discrete choice model specification process. Delphos aims to support the modeller by providing automated, data-driven suggestions for model specifications, thereby reducing the effort required to develop and refine utility functions. Delphos conceptualises model specification as a sequential decision-making problem, inspired by the way human choice modellers iteratively construct models through a series of reasoned specification decisions. In this setting, an agent learns to specify candidate model specifications by choosing a sequence of modelling actions, such as adding alternative specific constants, accommodating both generic and alternative-specific taste parameters, applying non-linear transformations to attributes, and including interactions with covariates. Each resulting candidate model is estimated and evaluated using a reward function defined by the modeller, which can reflect statistical model fit as well as behavioural expectations. Specifically, Delphos uses a Deep Q-Network to learn how individual specification decisions contribute to the eventual quality of the resulting model and, in turn, which sequences of modelling decisions tend to produce well-performing candidates. We evaluate Delphos on both simulated and empirical datasets using alternative reward functions. In simulated cases, learning curves, Q-value patterns, and performance metrics show that Delphos learns effective specification strategies while exploring only a small fraction of the feasible modelling space. We further apply the framework to two empirical datasets to benchmark and demonstrate its practical use. These experiments illustrate the ability of Delphos to generate competitive, behaviourally plausible models and highlight the potential of this adaptive, learning-based framework to assist the model specification process.

Figures & tables

Figure 1: Classical RL framework. Adaptation from Sutton & Barto ((2018))

Figure 2: Framework for assisting discrete choice model specification. While Delphos encodes the current specification, selects a modelling action, and submits the model for estimation, the environment returns modelling outcomes as a reward signal. Transitions are stored in a replay buffer and used to update the policy and target networks.

Modelling outcome	Formulation	Bound
Log-likelihood	$\displaystyle\max\left(0,\frac{LL-LL_{0}}{N}\right)$	$\displaystyle[0,\log J]$
AIC	$\displaystyle\max\left(0,\frac{LL-LL_{0}}{N}-\frac{K}{N}\right)$	$\displaystyle\left[0,\log J-\frac{K}{N}\right]$
BIC	$\displaystyle\max\left(0,\frac{LL-LL_{0}}{N}-\frac{K\log N}{2N}\right)$	$\displaystyle\left[0,\log J-\frac{K\log N}{2N}\right]$
McFadden’s pseudo- $R^{2}$	$\displaystyle\max\left(0,1-\frac{LL}{LL_{0}}\right)$	$\displaystyle[0,1]$
Adjusted McFadden’s pseudo- $R^{2}$	$\displaystyle\max\left(0,1-\frac{LL-K}{LL_{0}}\right)$	$\displaystyle\left[0,1+\frac{K}{N\log J}\right]$

Table 1: Scaling of the modelling outcomes used in the reward function.

Figure 3: Agent’s sequential decision-making process. Nodes represent internal states (current specifications). Dashed arrows indicate feasible actions, while solid arrows show the actions selected by the agent until termination.

Experiment	$1A$		$1B$		$1C$
Variable	Transf.	Taste	Transf.	Taste	Transf.	Taste	Covariate
ASC 1	Linear		Linear		Linear		cov 1
ASC 2	Linear		Linear		Linear		cov 1
$x_{1}$	Linear	Generic	Log	Specific	Linear	Specific
$x_{2}$	Log	Generic	Box-Cox	Generic	Log	Specific	cov 2
$x_{3}$	Linear	Generic	Linear	Specific	Linear	Generic

Table 2: Utility specifications and modelling spaces for simulated choice datasets (

1A

–

1C

Figure 4: Learning process on simulated dataset $1A$ . The upper panel displays the learning curve, showing the rolling mean and min–max range of the reward signal. The lower panel reports rolling novelty, measured as the share of previously unseen specifications.

Figure 5: Action selection distribution in simulated dataset $1A$ . Bars show the average frequency with which each action is selected in the early and late training across runs. Numbers report the average $\bar{\rho}^{2}$ for specifications in which the action was used. The full list of actions and their corresponding definitions is provided in the Appendix E .

Exp	Model space	Models estimated by agent	Converged models	First recovery	Reward (last 100)	$\Delta$ Reward
1A	8.19 $\times 10^{3}$	1.89 $\times 10^{3}$	97.42%	1.46 $\times 10^{3}$	0.611	-0.002
1B	8.19 $\times 10^{3}$	2.34 $\times 10^{3}$	98.45%	1.63 $\times 10^{3}$	0.753	-0.009
1C	1.59 $\times 10^{6}$	3.92 $\times 10^{3}$	78.48%	3.73 $\times 10^{3}$	0.654	-0.023

Table 3: Modelling size, exploration coverage, and performance on simulated experiments

	Search Metrics			In-Sample AIC			Out-of-Sample LL
	#Param.	Converged	Time (min)	Mean ( $\pm$ std)	Best	Worst	Best	Worst
Delphos	22 ( $\pm$ 8)	645 ( $\pm$ 65)	24 ( $\pm$ 3)	7,957 ( $\pm$ 394)	7,454	9,128	-879	-1,073
SA	13 ( $\pm$ 7)	1,344 ( $\pm$ 21)	9 ( $\pm$ 2)	8,431 ( $\pm$ 472)	7,463	9,410	-913	-1,163
VNS	8 ( $\pm$ 4)	1,443 ( $\pm$ 14)	104 ( $\pm$ 23)	8,843 ( $\pm$ 385)	7,744	9,401	-948	-1,163

Table 4: Search space performance across 10 independent seeds.

Figure 6: Evolution of the best-so-far AIC across methods on the Swissmetro dataset.

Figure 7: Pareto front of Delphos, SA, VNS on the Swissmetro dataset.Bold lines indicate the Pareto front of the best-performing seed for each method, while shaded regions span the area between the best and worst seeds across the 10 independent runs.

Figure 8: Pareto front of Delphos candidates on the DECISIONS dataset. Grey dots show all converged specifications proposed during training. Stars indicate the non-dominated candidates, and circles show the best AIC and best log-likelihood specifications.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Experiment	$1A$		$1B$		$1C$
ASC 1	$\beta_{\text{ASC1}}$	$-0.95$	$\beta_{\text{ASC1}}$	$-0.95$	$\beta_{\text{ASC1,cov1}=0}$	$-0.90$
					$\beta_{\text{ASC1,cov1}=1}$	$0.43$
ASC 2	$\beta_{\text{ASC2}}$	$0.84$	$\beta_{\text{ASC2}}$	$0.84$	$\beta_{\text{ASC2,cov1}=0}$	$-0.83$
					$\beta_{\text{ASC2,cov1}=1}$	$0.43$
$x_{1}$	$\beta_{1}$	$-0.72$	$\beta_{1}$	$-1.65$	$\beta_{1,1}$	$-0.50$
					$\beta_{1,2}$	$-0.40$

Appendix

Table 5: True parameter values for each simulated case

Hyperparameter	Level values
Number of hidden layers $(\mathbf{L})$	$\{2,\,3\}$
Hidden units per layer $(\mathbf{U})$	$\{64,\,128,\,256,\,512\}$
Learning rate $(\boldsymbol{\alpha})$	$\{1e-3,\,5e-4,\,1e-4\}$
Discount factor $(\boldsymbol{\gamma})$	$\{0.95,\,0.99\}$
Replay buffer size $(\mathbf{BS})$	$\{5{,}000,\,10{,}000,\,20{,}000\}$
Target network update frequency $(\mathbf{TU})$	$\{5,\,10,\}$

Appendix

Table 6: Hyperparameter ranges.

Exp	L	U	$\boldsymbol{\alpha}$	$\boldsymbol{\gamma}$	BS	TU	E	AUC	Final–100
$1A$ - conf. 1	2	256	1e–4	0.99	5k	10	50k	0.552	0.612
$1A$ - conf. 2	3	256	1e–4	0.99	5k	5	50k	0.553	0.612
$1A$ - conf. 3	2	64	1e–4	0.99	10k	10	5k	0.538	0.611
S 1 - conf. 4	2	64	1e–4	0.99	10k	10	10k	0.548	0.611
$1A$ - conf. 5	2	256	1e–4	0.95	10k	5	30k	0.538	0.610
$1B$ - conf. 1	2	128	1e–4	0.99	10k	10	50k	0.708	0.753

Appendix

Table 7: Performance of sampled hyperparameter configurations.

Figure 9: Learning curve showing the reward signal ( $\bar{\rho}^{2}$ ) across 10 runs

Figure 10: Learning curve showing the reward signal ( $\bar{\rho}^{2}$ ) across 10 runs

Action ID	Operation	Variable	Transformation Type	Action
0	terminate	0	-	( end )
1	add	1	linear	( add , 1, linear )
2	change	1	linear	( change , 1, linear )
3	add	1	log	( add , 1, log )
4	change	1	log	( change , 1, log )
5	add	1	box-cox	( add , 1, box-cox )

Appendix

Table 8: Action space for simulated experiment

1A

Parameter	Transformation	Covariate	Value (Rob. s.e.)
$\beta_{\text{train},\,\text{age}=1}$		age = 1	$1.483\,(0.706)$
$\beta_{\text{train},\,\text{age}=2}$		age = 2	$0.515\,(0.371)$
$\beta_{\text{train},\,\text{age}=3}$		age = 3	$0.232\,(0.399)$
$\beta_{\text{train},\,\text{age}=4}$		age = 4	$0.746\,(0.482)$
$\beta_{\text{train},\,\text{age}=5}$		age = 5	$2.168\,(0.499)$
$\beta_{\text{sm},\,\text{age}=1}$		age = 1	$0.682\,(0.879)$

Appendix

Table 9: Best-AIC specification on Swissmetro dataset

Parameter	Transformation	Covariate	Value (Rob. s.e.)
$\beta_{\text{car},\,\text{edu}=1}$		edu = 1	$-0.521\,(0.394)$
$\beta_{\text{car},\,\text{edu}=2}$		edu = 2	$-0.594\,(0.348)$
$\beta_{\text{car},\,\text{edu}=3}$		edu = 3	$-0.887\,(0.248)$
$\beta_{\text{car},\,\text{edu}=4}$		edu = 4	$-1.654\,(0.201)$
$\beta_{\text{car},\,\text{edu}=5}$		edu = 5	$-1.661\,(0.237)$
$\beta_{\text{car},\,\text{edu}=6}$		edu = 6	$-2.038\,(0.275)$

Appendix

Table 10: Best-LL_Nparams specification on Decisions dataset.

Discrete Choice Modeling

Discrete choice modeling studies how people select among alternatives and how preferences can be estimated from observed choices.

See all 44 papers

Sep 16, 2026

Multitask Reinforcement Learning for Assisting Choice Model Specification

Discrete choice model specification is a time-consuming task in which modellers often specify and estimate multiple models while balancing goodness-of-fit, parsimony, and behavioural plausibility. We present Delphos, a multitask reinforcement learning framework that learns transferable specification strategies across transport choice datasets. Delphos frames model specification as a sequential decision-making problem in which it applies a sequence of modelling actions and receives feedback from an estimation environment based on model performance and convergence. To transfer modelling decisions across datasets with different sets of variables, Delphos represents utility specifications as sets of modelling terms using a DeepSet-Q architecture, allowing a shared specification policy to learn across multiple datasets. Trained on nine transport choice datasets, Delphos consistently outperforms independently trained single-task agents, indicating that sharing modelling experience improves learning efficiency and helps identify promising sequences of modelling decisions with fewer unsuccessful estimation attempts. When applied without further training to the unseen Swissmetro and Decisions datasets, the same agent identifies competitive specifications in less than 20 minutes on a standard CPU. It achieves a higher log-likelihood per observation than the VNS metaheuristic on Swissmetro and performance comparable to a published MNL specification developed by expert modellers on Decisions. These findings show that accumulating and reusing modelling experience enables Delphos to function as an intelligent assistant for discrete choice model specification. It reduces manual trial-and-error while allowing modellers to retain control over model diagnosis, refinement, and final selection.

Gabriel Nova, Stephane Hess, Sander Van Cranenburgh

CityAI Lab, Transport and Logistics Group, Delft University of Technology, The Netherlands · Institute for Transport Studies and Choice Modelling Centre, University of Leeds, UK

Discrete Choice Modeling Deep Sets Multi-Task RL

Ask PDF

May 23, 2025

Bayesian Deep Learning for Discrete Choice

Discrete choice models (DCMs) are used to analyze individual decision-making in contexts such as transportation choices, political elections, and consumer preferences. DCMs play a central role in applied econometrics by enabling inference on key economic variables, such as marginal rates of substitution, rather than focusing solely on predicting choices on new unlabeled data. However, while traditional DCMs offer high interpretability and support for point and interval estimation of economic quantities, these models often underperform in predictive tasks compared to deep learning (DL) models. Despite their predictive advantages, DL models remain largely underutilized in discrete choice due to concerns about their lack of interpretability, unstable parameter estimates, and the absence of established methods for uncertainty quantification. Here, we introduce a deep learning model architecture specifically designed to integrate with approximate Bayesian inference methods, such as Stochastic Gradient Langevin Dynamics (SGLD). Our proposed model collapses to behaviorally informed hypotheses when data is limited, mitigating overfitting and instability in underspecified settings while retaining the flexibility to capture complex nonlinear relationships when sufficient data is available. We demonstrate our approach using SGLD through a Monte Carlo simulation study, evaluating both predictive metrics--such as out-of-sample balanced accuracy--and inferential metrics--such as empirical coverage for marginal rates of substitution interval estimates. Additionally, we present results from two empirical case studies: one using revealed mode choice data in NYC, and the other based on the widely used Swiss train choice stated preference data.

Daniel F. Villarraga, Ricardo A. Daziano

School of Civil and Environmental Engineering, Cornell University, 220 Hollister Hall, Ithaca, NY 14853, USA

Bayesian Neural Networks Bayesian Inference Discrete Choice Modeling

Ask PDF

Jul 30, 2026

An analysis of machine learning approaches for enhancing decision-making in complex discrete choice tasks

Discrete choice modeling is a common tool used for preference elicitation during policy-making, but this is typically done through parametric models. Machine learning can push the boundaries of discrete choice modeling for policy-based preference elicitation by adopting a data-driven approach or learning individual preferences. However, there is limited knowledge of how well machine learning methods can estimate individual discrete choice rules under individual heterogeneity, especially in the context of challenges often experienced during preference elicitation. This study evaluates four machine learning models (multinomial logistic regression, generalized additive model, twinned neural network, and Gaussian process) with respect to their capacity to learn and predict five choice rules that are important in the behavioral and social sciences (linear strong utility, monotonic strong utility, ideal point, lexicographic semiorder, and multiattribute linear ballistic accumulator). Monte Carlo experiments were performed to assess model performance when increasing a) the number of attributes in the choice alternatives, b) the number of training choice sets, and c) the choice rule's determinism. The simulation results demonstrated that semi-parametric and non-parametric models generally outperform parametric models across all choice rules and experimental contexts. Model performance also generally improves by 6% to 96% and 0% to 55%, respectively, with an increase in training choice sets and choice rule determinism. A case study using real energy policy preference data was also conducted, where TNN performed best with a BIC of 13.351. This work demonstrated the viability and limitations of semi-parametric and non-parametric models in the context of policy-centric discrete choice modeling and showed how the choice task context should drive model selection.

Sheng Lun Christine Cao, Destenie Nock, Alex Davis

Department of Electrical and Software Engineering, University of Calgary, Calgary, AB, Canada · Department of Engineering and Public Policy, Carnegie Mellon University, Pittsburgh, PA, USA · Department of Civil Engineering, Carnegie Mellon University, Pittsburgh, PA, USA +1

Discrete Choice Modeling User Preference Modeling Preference Elicitation

Ask PDF

Abstract

Discrete Choice Modeling

Explore similar work

Multitask Reinforcement Learning for Assisting Choice Model Specification

Bayesian Deep Learning for Discrete Choice

An analysis of machine learning approaches for enhancing decision-making in complex discrete choice tasks