Authors: Yingkang Luo, Hua-Dong Xiong, Hanbo Xie, Caroline Phelps, Madeline Harms, Ryan Smith, James Waltz, Robert Wilson
Organizations: College of Computing, Georgia Institute of Technology · School of Psychology, Georgia Institute of Technology · Department of Psychology, University of Minnesota Duluth · Laureate Institute for Brain Research · School of Medicine, University of Maryland
Decision making often requires balancing exploitation of familiar options with exploration of uncertain alternatives. One strategy for exploration is random exploration, in which choices become more variable and less tied to the option that currently appears best. However, it remains unclear whether this variability reflects genuine stochasticity or instead a deterministic structure that traditional cognitive models fail to capture. We examined this question using the Horizon Task in an aggregated dataset of 1,392 participants completing over 1,270,695 trials. In this task, exploration becomes more valuable when participants have more future choices, and choice variability increases under longer horizons. We tested whether more expressive neural network models could recover deterministic structure from this additional variability. Although neural networks provided better overall fits to behavior than the traditional model, long-horizon choices remained consistently less predictable than short-horizon choices. These findings show that limited model expressiveness cannot fully account for the increased variability associated with random exploration, suggesting random exploration truly is substantially random.
Figures & tables
Figure 1: The Horizon Task. (A) The first four instructed trials, during which participants are required to select the highlighted green option between the two slot machines. The structure of the forced trials is identical across horizon conditions, differing only in the total number of trials. (B, C) Following the instructed trials, participants complete free-choice trials until the end of the game. During this phase, they choose freely between the two slot machines. In the short-horizon condition (B), participants make one additional free choice, whereas in the long-horizon condition (C), they make six additional free choices. (D) Illustration of the uncertainty conditions. On the left, the instructed trials reveal an unequal amount of information per machine [1 3]. On the right, the instructed trials reveal an equal amount of information per machine [2 2]. Analyses focus on the first free-choice trial, highlighted in green.
Figure 2: Choice curves of the Horizon Task. (A, B) Choice curves showing the probability of selecting an option as a function of the difference in observed mean rewards. (A) Probability of choosing the more informative option in the unequal uncertainty condition (i.e., the machine played once during the forced trials). The observed mean reward difference is the reward of the more informative option minus that of the less informative option. (B) Probability of choosing the left option in equal condition as a function of the observed mean reward difference between the left and right options. (C, D, E) Parameter estimates from logistic regression fits across horizon conditions. (C) Information bonus in the unequal uncertainty condition. (D, E) Decision noise parameters in the unequal and equal uncertainty conditions, respectively. *** denotes a significant difference between horizon conditions at p<0.001 .
Baseline fitting procedure
H1 accuracy
H6 accuracy
H1–H6 difference
Global fit
0.807
0.732
0.075
Individual fits
0.799
0.737
0.061
First-game fit
0.881
0.767
0.113
Table 1: Horizon-specific test accuracy of the logistic regression baselines.
Model
NLL
Pseudo- R2
ΔR2 vs. Baseline
MLP
0.4440
0.3593
0.1426
RNN
0.4604
0.3356
0.1189
LSTM
0.4282
0.3821
0.1655
GRU
0.4350
0.3722
0.1556
TinyGRU
0.4769
0.3118
0.0952
Transformer
0.4323
0.3761
0.1595
Table 2: Model comparison using NLL and McFadden’s pseudo- R2 .
Figure 3: Test accuracy by horizon across neural network architectures. (A) Global analysis, where models predicted choices without participant identity. (B) Subject-level analysis, where participant identity was included as an additional input. Each point shows the test accuracy of one architecture, computed separately for H1 and H6 after restoring the checkpoint with the lowest validation loss. Horizontal error bars indicate 95% Wilson confidence intervals. H1 accuracy is shown in blue and H6 accuracy in red. Dashed vertical lines indicate the corresponding horizon-specific test accuracy of the logistic regression baseline. Abbreviations: MLP, multilayer perceptron; RNN, recurrent neural network; LSTM, long short-term memory; GRU, gated recurrent unit; H1, Horizon 1; H6, Horizon 6.
Figure 4: Predictive performance in the first game. Test accuracy was compared between H1 and H6 using only the first game played by each participant. Each point shows the test accuracy of one neural network architecture after restoring the checkpoint with the lowest validation loss, computed separately for the H1 and H6 conditions. Horizontal error bars indicate 95% Wilson confidence intervals for test accuracy. H1 accuracy is shown in blue and H6 accuracy is shown in red. Dashed vertical lines indicate the corresponding horizon-specific test accuracy of the first-game logistic regression baseline. Abbreviations: MLP, multilayer perceptron; RNN, recurrent neural network; LSTM, long short-term memory; GRU, gated recurrent unit; H1, Horizon 1; H6, Horizon 6.
Figure 5: Cross-game history analysis. Predictive accuracy was compared between H1 and H6 for recurrent models that incorporated cross-game history through the persistence parameter β . Each point shows the test accuracy of one architecture after restoring the checkpoint with the lowest validation loss, computed separately for the H1 and H6 conditions. Horizontal error bars indicate 95% Wilson confidence intervals for test accuracy. H1 accuracy is shown in blue and H6 accuracy is shown in red. Dashed vertical lines indicate the corresponding horizon-specific test accuracy of the individual-fit logistic regression baseline. Abbreviations: RNN, recurrent neural network; LSTM, long short-term memory; GRU, gated recurrent unit; H1, Horizon 1; H6, Horizon 6.
Figure 6: Predictive accuracy across H6 free-choice trials. The red line shows Seq2Seq model accuracy across the six free-choice trials in the H6 condition, from C4 to C9 . The dashed blue line indicates the H1 benchmark accuracy. Predictive accuracy increased across later H6 trials as the remaining horizon shortened, but final-trial H6 performance remained below the H1 benchmark.
Total
Gender
Age
Location
Paper
N
Group
F
M
O
n
Mean
SD
City, State
Harms et al. (2024)
123
Teens
25
29
0
54
11.31
1.10
Madison, WI
Young
14
43
0
57
19.33
2.63
Midlife
1
1
0
2
37.50
0.71
Unknown
0
0
10
10
–
–
Phelps et al. (2025) Mizell et al. (2024)
349
Young
84
65
2
151
21.43
2.87
Tucson, AZ
Table 3: Participant demographics across datasets stratified by age group
Figure 7: MLP Architecture. The model received a single concatenated vector of sequential and static inputs. Sequential inputs consisted of choice–reward pairs (Ct,Rt) from the instructed trials ( t∈{0,1,2,3} ), while static inputs encoded horizon length (H) and uncertainty condition (UC). The architecture consisted of two FC layers (32 and 16 units, respectively) with ReLU activations. The output layer applied a softmax function to generate choice probabilities for the first free-choice trial. C4 denotes the first free-choice response, as trial indexing began at 0.
Figure 8: RNN-based Architectures. The vanilla RNN and GRU used 64 hidden units, whereas the LSTM used 128 hidden units. The TinyGRU variant had two hidden units. (a) The recurrent layer received only sequential inputs. The final hidden state was concatenated with static inputs. (b) The recurrent layer received both sequential inputs and static inputs at each time step. (a, b) In both architectures, the resulting representation was passed through two FC layers, identical to those used in the MLP architecture, to generate a softmax prediction of the first free-choice response.
Figure 9: Transformer-based Architectures. Sequential inputs were first projected into a 64-dimensional representation with sinusoidal positional encoding. (a) The Transformer encoder had four attention heads and a feedforward network with 128 hidden units using ReLU activation. (b) The attention-only variant consisted of eight attention heads.
Figure 10: Seq2Seq Architecture. The model implemented an encoder–decoder recurrent framework. (a) In the teacher-forcing variant, the decoder received the ground-truth choice at each time step during training to predict the subsequent choice ( t∈{4,…,9} ). (b) In the student-forcing variant, the decoder used its own previous prediction as input to generate the next choice.