A Bayesian latent class reinforcement learning framework to capture adaptive, feedback-driven travel behaviour
Organizations: Choice Modelling Centre & Institute for Transport Studies, University of Leeds · DTU Management, Technical University of Denmark · Institute of Transport and Logistics Studies, University of Sydney · College of Business, Al Yamamah University, Riyadh, Saudi Arabia
Abstract
Many travel decisions involve a degree of experience formation, where individuals learn their preferences over time. At the same time, there is extensive scope for heterogeneity across individual travellers, both in their underlying preferences and in how these evolve. The present paper puts forward a Latent Class Reinforcement Learning (LCRL) model that allows analysts to capture both of these phenomena. We apply the model to a driving simulator dataset and estimate the parameters through Variational Bayes. We identify three distinct classes of individuals that differ markedly in how they adapt their preferences: the first displays context-dependent preferences with context-specific exploitative tendencies; the second follows a persistent exploitative strategy regardless of context; and the third engages in an exploratory strategy combined with context-specific preferences.
Figures & tables
| High Learning Rate ( = 0.9) | Low Learning Rate ( = 0.1) | |
| 25 (not updated) | 25 (not updated) | |
| = 29.5 | = 25.5 | |
| 0.97 | 0.38 | |
| 0.03 | 0.62 |
| Number of Participants | Number of Choices | ||||
| Count | % | Reliable Route | Unreliable Route | ||
| Overall Count | — | 83 | — | 635 | 1025 |
| Gender | Female | 41 | 49.40% | 380 | 440 |
| Male | 42 | 50.60% | 255 | 585 | |
| Age | 1: 29 or less | 21 | 25.30% | 162 | 258 |
| 2: 30 to 39 | 22 | 26.51% | 168 | 272 | |
| Model | K | Nb. Param | LL | AIC | BIC |
| BL | 1 | 1 | -1,104.38 | 2,210.76 | 2,213.18 |
| RL | 1 | 6 | -962.91 | 1,937.82 | 1,952.33 |
| LCCM | 2 | 10 | -913.96 | 1,847.92 | 1,872.11 |
| LCCM | 3 | 18 | -848.03 | 1,732.06 | 1,775.60 |
| LCRL | 2 | 18 | -870.16 | 1,776.32 | 1,819.86 |
| LCRL | 3 | 30 | -803.51 | 1,667.02 | 1,739.59 |
| RL | LCCM | LCRL | |||||
| Variable | Class 1 | Class 2 | Class 3 | Class 1 | Class 2 | Class 3 | |
| RL Choice Model | LCCM Benchmark | Class-Specific RL Choice Model | |||||
| – | – | – | |||||
| – | – | – | |||||
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | k | Bias | NRMSE | Correlation | |
| 0 | 0.018 | 0.086 | 0.901 | 0.804 | |
| 1 | -0.038 | 0.107 | 0.883 | 0.746 | |
| 0 | -0.062 | 0.096 | 0.893 | 0.790 | |
| 1 | -0.060 | 0.083 | 0.916 | 0.828 | |
| 0 | 0.014 | 0.141 | 0.880 | 0.772 | |
| 1 | 0.002 | 0.117 | 0.919 | 0.837 |
| Parameter | Bias | NRMSE | Correlation | |
| -0.024 | 0.041 | 0.973 | 0.946 | |
| 0.023 | 0.052 | 0.965 | 0.930 | |
| 0.006 | 0.057 | 0.981 | 0.958 | |
| 0.032 | 0.068 | 0.977 | 0.946 | |
| -0.020 | 0.082 | 0.944 | 0.880 | |
| -0.042 | 0.069 | 0.944 | 0.886 |