Using predictive models to explain cognition requires more than accurate behavioral predictions. Input ablations offer an appealing route: remove information from a model and interpret the resulting performance change as evidence of its importance for behavior. Yet this inference assumes that the model's dependence on information reflects the dependence of the process generating the behavior. We test it in two synthetic sequential bandit tasks with known generating policies, where past choices can remain informative when feedback is unavailable to a predictor. We compare GRUs and Transformers trained from scratch, a fine-tuned LLaMA model, and cognitive models across systematically varied reward contributions. Our analyses distinguish prediction after training without reward observations from the response of a fixed predictor to donor-reward replacement. Three findings emerge. First, in the restless task, neural models trained without rewards predict held-out choices better than four simple training-fitted behavioral baselines. Second, under matched donor replacement, accurate predictors can respond much less than the known generator. Third, at some reward weights, neural networks predict better than a pooled reinforcement-learning model but have less faithful changes in choice probabilities; the model ordering differs between the two tasks. These independent-test results separate information sufficient for prediction from response fidelity under a specified ablation in sequential choice. They motivate validating model-ablation responses independently of predictive performance before using them to infer how the observed behavior was generated.
Figures & tables
Figure 1: Reward-history ablation and claims about sequential behavior. (A) At evaluation, past rewards are replaced while observed choices and fitted model parameters remain fixed. The generic model diagram shows how a small prediction change can invite an inference of weak reward dependence in the process generating behavior. The arrow and question mark mark the inference being tested. (B) Task A (four-armed restless bandit) combines reward learning with a last-choice kernel. Task B (spatially correlated bandit) combines reward generalization with local choice tendencies. Seven weights control the relative contributions to the known generating policies. (C) Neural and symbolic predictors are fitted to synthetic trajectories and evaluated under matched original and donor-replaced reward histories. Neural models have separate full-input and choice-only fits; symbolic predictors comprise RW + softmax and the fitted generator. The oracle uses true generating parameters. Bars show mean test-set increases in negative log likelihood at reward weight one in Task A over 20 shared donor assignments. Neural models and RW use pooled training; fitted generators use individual prefix calibration (Section 4 ). A retains a validation-trial Transformer example to illustrate the operation; C reports aggregate test responses. Blue marks choices; gold marks rewards. Oracle replay conditions on observed choices throughout.
Figure 2: Choice-only prediction retains information in the restless task. Restless-task NLL after full-input (left) or choice-only (right) training. The oracle observes rewards in both panels; it is not a choice-only oracle. Dotted line: uniform random prediction. Curves use the same 250 test participants per weight and trials 151–200. Bands are participant-bootstrap intervals conditional on three GRU/Transformer fits and one LLaMA fit. Four train-fitted behavioral comparisons appear in Figure S3 .
Figure 3: Better restless-task prediction need not imply a more faithful ablation response. Original and donor-replaced NLL (a,b), their difference (c), and probability-response error relative to the oracle (d; Eq. 3 ). Actual choices are retained; 20 donor derangements replace preceding chosen rewards. High-weight neural predictors have better intact NLL than pooled RW but larger response error. Original/donor panels share limits. Bands are participant-bootstrap intervals; seed and calibration qualifications are as in Section 4 .
Figure 4: The spatial task shows a different predictor ordering. Spatial-task test evaluation on maps 7–8: 50 participants per weight, 40 scored choices each, and 20 matched donor derangements. The initial location/reward cue remains intact. Panel definitions match Figure 3 ; dotted lines mark log25 . Unlike in the high-weight restless task, LLaMA outperforms pooled RW in both original prediction and response fidelity at all tested nonzero weights. Only fixed-shape, causally audited LLaMA evaluations are included.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure S1: Behavioral checks and bounded recovery evidence. Left: repetition probabilities in existing training records, including conditioning on preceding reward. Middle: pooled plug-in conditional entropies, not held-out information estimates. Top right: mean fitted reward weights from 250 restless validation-prefix fits and a separate spatial pilot with 10 training participants per weight. Dashed line denotes equality; means can conceal substantial individual error and nonidentifiability (Figure S2 ). Bottom right: spatial pilot NLL on two maps withheld from individual fitting, comparing local-only, reward-only, and mixture candidates. Curves are descriptive; no training-seed or inferential bands are implied.
Figure S2: Parameter recovery is not uniform. (a) Mean estimated reward weight and identity line; (b) weight mean absolute error (MAE); (c) restless learning-rate MAE; (d) restless kernel-sign accuracy, with chance marked at .5 . Restless metrics use validation-participant prefix fits ( n=250 per weight); spatial estimates use a training-participant pilot ( n=10 per weight). Inactive parameters at w=0 or w=1 are omitted, not scored as zero error. Curves are descriptive without uncertainty bands; mean agreement in (a) need not imply accurate individual recovery in (b).
Setting
GRU
Transformer
Embedding/model width
64 / 256 hidden
256
Layers
2
6
Attention heads / feed-forward width
Not applicable
8 / 1,024
Dropout
.1
.1
Optimizer
AdamW
AdamW
Learning rate / weight decay
.001 / .01
.0003 / .01
Appendix
Table 1: Scratch-trained neural configurations. Seeds are 11, 22, and 33 in each weight/input condition. Batch sizes are effective sequence batch sizes.
Figure S3: Choice-only prediction exceeds four simple training-fitted controls. Each panel repeats the Figure 2 choice-only neural curves and adds the named behavioral baseline. All baseline parameters, including lag, are selected on training data and frozen at test evaluation. Comparisons use the same 250 participants and trials 151–200. Uniform random prediction is dotted. Neural mean NLL is lower in every displayed model/weight comparison; this is not a multiplicity-adjusted significance claim.
Reward weight
Full input
Choice only
Choice only − full
0.0
2.279
2.280
0.001
0.1
2.386
2.393
0.006
0.3
2.448
2.525
0.077
0.5
2.389
2.512
0.123
0.7
2.273
2.441
0.169
0.9
2.117
2.339
0.222
Appendix
Table 2: Spatial LLaMA prediction on 50 test participants, all 160 choices each. One full-input and one choice-only adapter per weight. Values are NLL in nats.
Figure S4: A neural-generator sensitivity control. Validation evaluation with a frozen previously trained GRU torso and reset output heads; each curve averages three generator seeds. Four panels match the main response comparisons, but the horizontal axis is a reward-visible versus constant-reward policy-mixture weight, not the original psychological utility weight. Students are freshly fitted; no student parameters are shared with the teacher. This experiment does not use LLaMA and is not a fully random-generator control.
Figure S5: Trained Transformer torso with reset output heads. Validation curves average three head seeds, without individual-seed traces or uncertainty bands. Panel definitions match the trained-GRU control. The new output layer is untied from the learned token embedding, leaving the torso unchanged. Original and donor NLL share limits; dotted lines mark uniform prediction. This is a trained-representation control, not a fully random Transformer.
Figure S6: Fully random GRU generator. All weights, not just the output head, are initialized independently of behavioral data. Validation curves average three generator seeds, with one student seed per cell. Panel definitions and scoring window match the trained-torso control. Dotted lines mark uniform NLL; original and donor panels share limits. The oracle’s very small effect sets the scale of this control, not evidence of successful neural recovery.
Figure S7: Fully random Transformer generator. Validation protocol and panel definitions match Figure S6 ; all 21 cells are included. Each student is trained on this teacher’s own generated choices, not the symbolic generator’s choices. No individual-seed traces or uncertainty bands are shown. Axis ranges differ from the random-GRU figure because the generators induce different prediction and response scales.
Figure S8: The scope of reward replacement changes the measured effect. Columns vary which preceding rewards are replaced; rows report NLL change and response error against the oracle under that same operation. Curves average three GRU/Transformer seeds and use one LLaMA SFT seed; oracle and individual correct-family fits use the same participants and donors. Shared limits within each row aid comparison. These are descriptive means from the existing seven-weight validation analysis.
Figure S9: Restless-task reward-edit effects depend on the model and operation. Seven-weight validation sweep of fixed full-input GRU (a), Transformer (b), and LLaMA SFT (c) models, followed by the true-parameter oracle (d) and prefix-fitted generator (e). Each point averages 250 participants on trials 151–200, over three GRU/Transformer seeds or one LLaMA SFT seed. The donor curves retain the 20 validation assignments, not the main test assignments. Constant 50 is replayed through both symbolic policies; deletion and placeholders have no symbolic response without a missing-input convention. The choice-history rewrite applies only to LLaMA. Vertical limits match within each row; rows have different scales. Means are shown without seed traces or uncertainty bands.
Researchers must often choose between Bayesian or neural network models of behavior, two paradigms with complementary strengths and weaknesses. An ideal paradigm would facilitate testing many kinds of representations and inductive biases; Bayesian models make this easy, while neural networks do not. Similarly, an ideal paradigm would avoid over-simplifications; neural networks make this easy, while Bayesian models do not. Here, we introduce Bayesian distillation with Behavioral Tuning (BBT) as an approach to getting the best of both traditions. BBT offers a simple recipe for model building: first, a neural network is trained to mimic a Bayesian model through synthetic data, and second, the network is fine-tuned on human behavior to capture additional structure and nuance. Across four case studies in human concept learning, we find that BBT outperforms traditional approaches at predicting human behavior while also revealing psychological insights, resulting in models that can both mimic Bayesian priors and capture heuristics and biases that violate simple modeling assumptions.
Brenden M. Lake, Akshay K. Jagadish, Guangyuan Jiang
Departments of Computer Science and Psychology, Princeton University · Princeton AI Lab, Princeton University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study how pretraining and midtraining supply the information and computation that make reward adaptation effective. In sequential state computation and contextual memory, we characterize mechanisms that agree on every training reward yet demand different held-out answers. Task-independent source observations resolve this ambiguity. We construct finite sampled Adam paths from specified random initializations through source prediction and reward adaptation in the same parameters, proving how prediction acquires execution or retrieval and rewards learn their task-specific use. Experiments with pretrained Qwen2.5 checkpoints test this division of labor. Across eight worlds, Sequential models trained with correct source and first-operation supervision reach 82.61% success, versus 44.15% for a private-random source control. Memory replay preserves retrieval during reward adaptation, and an independent eight-world confirmation achieves 75.32% task success versus 49.86% after matched alternative-retrieval training. GSM8K and HotpotQA separate accuracy at reward entry, subsequent gain and final performance. Together, these results connect information acquisition, executable computation and reward-guided task learning.
Chiwun Yang, Xiaoyu Li
City University of Hong Kong · University of New South Wales
Reward models are a standard tool to score responses from LLMs. Reward models are built to rank responses to a fixed prompt sampled from a single model, for example to choose the best of n sampled responses. In this paper, we study whether scores from response-level reward models lifted to score a model's suitability for a prompt, prior to seeing responses from that model. Specifically, we show that it is straightforward to predict the expected reward that an LLM would earn from the reward model under repeated sampling. Further, we show that these expected reward predictions are precise and discriminative enough to support an application to a model routing protocol that routes prompts to models at inference time to maximize reward while controlling computational cost. We demonstrate the performance of this routing procedure on the open-perfectblend dataset, using a model pool composed of Llama3.1-Instruct 8B/70B, Gemma2-IT 9B/27B, and Gemma1-IT 7B models. Our simple expected reward prediction--based routing (ERP) outperforms baselines that route prompts to models with the best average performance within each prompt's category, and explains the success of more complex routing protocols that implicitly estimate an expected reward. Our approach has the added advantage of being trivially extensible as new models are added to the pool.
Kenan Hasanaliyev, Silas Alberti, Jenny Hamer +5
Stanford University · Inception Labs · Cognition Labs +2