More accurate behavioral predictions with hybrid Bayesian-connectionist models
Authors: Brenden M. Lake, Akshay K. Jagadish, Guangyuan Jiang
Organizations: Departments of Computer Science and Psychology, Princeton University · Princeton AI Lab, Princeton University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
Researchers must often choose between Bayesian or neural network models of behavior, two paradigms with complementary strengths and weaknesses. An ideal paradigm would facilitate testing many kinds of representations and inductive biases; Bayesian models make this easy, while neural networks do not. Similarly, an ideal paradigm would avoid over-simplifications; neural networks make this easy, while Bayesian models do not. Here, we introduce Bayesian distillation with Behavioral Tuning (BBT) as an approach to getting the best of both traditions. BBT offers a simple recipe for model building: first, a neural network is trained to mimic a Bayesian model through synthetic data, and second, the network is fine-tuned on human behavior to capture additional structure and nuance. Across four case studies in human concept learning, we find that BBT outperforms traditional approaches at predicting human behavior while also revealing psychological insights, resulting in models that can both mimic Bayesian priors and capture heuristics and biases that violate simple modeling assumptions.
Figures & tables
Figure 1: The Bayesian and connectionist paradigms occupy opposite ends of a spectrum. Our Bayesian distillation with Behavioral Tuning (BBT) approach moves along this spectrum in two stages. First, a neural network is pretrained to mimic a Bayesian model through Bayesian distillation (marked as “start” in the figure). The network is then fine-tuned on human behavioral data, shifting it towards the connectionist end of the spectrum until performance on held-out data plateaus (marked as “stop”). Depending on the nature of the behavioral data, the model may stop anywhere along the spectrum. Note that this is a model-fitting procedure and not a hypothesis about cognitive development.
Figure 2: The recipe for BBT modeling involves 1) distilling a Bayesian model into a neural network (via synthetic data from forward sampling; no Bayesian inference required), and then 2) fine-tuning the network on human behavioral data (potentially very limited). During a training “episode”, the network receives a query input xm+1 (here, on the right, the network is queried on a small blue triangle) and a study set (objects Xm and labels Ym ), which are concatenated and presented together as input. The network then produces a predicted query output ym+1 (e.g., “no”) which is compared to a target.
Figure 3: The number game task and model predictions. A) Given a set of numbers that a mystery computer program produces, (e.g., Xm={60} ), the task is to predict which other numbers the program is likely to generate (e.g., 20? 53?). The main text describes the three scenarios and how one might reason about them. B) Results for the study set Xm={66,78} . The y-axis shows the proportion of “yes” answers for each query 1 to 100 (is it likely to generate this number next?) for humans and models. Also shown are the Pearson correlations ( r ) and root-mean-squared-error (RMSE) between the human and model judgments. C) Results for the study set Xm={8,80,48} . D) PCA of embeddings after distillation only (left) and full BBT training (right), extracted from decoder layer 2. Each point in the scatter corresponds to a support set Xm (averaged over all queries), marked with the color of its most likely hypothesis (via sparse approximation) as fit to the model judgments. After fine-tuning with BBT, the influence of “evens”/“odds” (blue) and “ending in n” (purple) hypotheses dramatically expands.
Model
Number concepts
Logical concepts
Shepard concepts
Compositional rules
Bayesian
-31,332.6
-79,023.3
-5,554.4
Distillation only
-31,435.7
-77,249.9
-5,601.1
-466.4
Fine-tuning only
-31,465.2
-106,926.3
-5,924.7
-1,653.6
BBT
-28,157.9
-69,423.8
-5,260.8
-343.6
Table 1: Model predictions evaluated through the log-likelihood of held-out human behavior. The bold shows the best scoring model (higher is better). All models had lapse rates that were fit, as described in Section M.4 of the Methods. Neural networks (Distillation only, Fine-tuning only, and BBT) were trained for five runs, with the reported run selected through a separate validation set.
Figure 4: The logical concept learning task and model predictions. A) Given N sets of objects and their category labels (positive examples marked with ∗ ), the task is to predict the labels for the (N+1) th set. Here, the correct rule is “largest blue object in the set.” B) Human and model judgments on the final 8 object sets when learning the rule in Panel A. The y-axis shows the proportion of “yes” answers per object. The x-axis shows the object stimuli, with the object sets divided by vertical lines. Also shown are the Pearson correlations ( r ) and root-mean-squared-error (RMSE) between human and model judgments. C) Judgments on the last 8 sets when learning the rule “is the only blue or green object in the set.” D) PCA of model embeddings after the distillation (left) and human fine-tuning (right) stages of BBT training, extracted from decoder layer 3. Each point is a different learning episode containing 24 object sets as study (averaging over 20 different query sets), marked with the color of its most likely hypothesis as fit to the model judgments via sparse approximation. After BBT fine-tuning, the model blurs the separation between Boolean vs. non-Boolean episodes, and the simplest hypotheses (true, false, individual features) expand their influence.
Figure 5: The Shepard concept learning task and model predictions. A) Participants are presented N objects sequentially, they are tasked to choose between two categories (1 and 2) on each trial and receive feedback indicating whether their response was correct. Each task is repeated over multiple blocks. In the example shown, the correct rule is “all yellow objects belong to category 1”. B & C) Learning curves over blocks for humans, distillation only networks, and BBT for young (B) and older (C) participants, data taken from the Badham et al. [ 3 ] study. The y-axis shows the proportion of correct choices and x-axis is tasks blocks, with the four Shepard category structures (Type-I to Type-IV) shown as distinct lines. D) PCA of model embeddings after the distillation (left), BBT (conditioned on young embedding; middle), and BBT (conditioned on older group embedding; right) extracted from decoder layer 3. Each point is a different learning episode containing 16 objects in support and all unique objects as query, marked with the color of its most likely hypothesis from four out of six Shepard’s category learning structures as fit to the model judgments via sparse approximation. The first component in BBT (old) can clearly separate Types II-IV (orange, green, and red) and Type I (blue), whereas in BBT (young) younger participants only separate Type II from Type III, and the second principal component separates Type I (blue) from other types for both BBT conditioned on young and old. Although both BBT (young) and BBT (old) show stronger representation of Type I hypothesis compared to Distillation only, the increase is stronger in younger than older.
Figure 6: Few-shot learning of compositional instructions. The task involves responding to strings of pseudowords (inputs) with sequences of colors (outputs). A) Participants learned these 14 study instructions which are composed of four primitives and three functions (heading were not provided to the participants). B) After learning the study instructions, participants were asked to respond to 10 query instructions (4 of 10 are shown here). The most frequent 4 responses to each query instruction are shown, labeled in parentheses with the count for people and the percent of samples for the models. Superscripts indicate which responses are correct ( ∗ ) or consistent with one-to-one biases (1-to-1) or iconic concatenation biases (IC).
Extended Data Figure 1: Interpreting the Bayesian model of the number game as a multilayer perceptron. Input layer : The study set Xm is represented as a count vector whose i th entry records the number of times integer i appears in Xm , similar to a “bag of words” input vector. For instance, if “4” appears twice in Xm , input4=2 . Hidden layer : The activation of each hidden unit h equals the posterior probability of the corresponding hypothesis, P(h∣Xm) . The net input to the unit corresponding to each hypothesis ( neth ) is the dot product between the input vector and weights Lh,: that reflect the log-likelihood, Lh,i=log∣h∣1 if i∈h , otherwise −∞ . The weights Lh,: for the hypothesis “evens” are shown on the arrows. The biases for the hidden units are the log prior, logP(h) . The layer has a softmax activation function. Output : The network simultaneously outputs the posterior predictive probability for every possible query i , outputi=(1−λ)P(i∈h∣Xm)+λ0.5 where the lapse rate λ is optimized. Prior-fit training : For behavioral fine-tuning, the network is initialized at the original Bayesian model parameters. Only the 5084 prior logits corresponding to P(h) and λ are trained with L-BFGS, maximizing the log-likelihood of the human data with a regularizer (0.001) that encourages a higher-entropy prior. Full-fit training : This model extends the prior-fit model by including the log-likelihood matrix L∈R5084×101 trainable. Each row is parameterized by unconstrained logits and normalized using log-softmax, using softened variants of their Bayesian model values for initialization. The indicator function in the output layer is relaxed to reflect that membership is now graded rather than binary, replacing \mathbbm1[i∈h] with eLh,i/(maxjeLh,j). The code for training these models is released as well.
Extended Data Figure 2: Analysis of number game predictions given a program that accepts {66,78} . The proportion of participants to answer “yes” for each query 1 to 100 is shown, along with model predictions. Using sparse hypothesis decomposition for interpretation, the three hypotheses with the largest influence in BBT’s predictions are shown, left to right across panels. The numbers included in each hypothesis are highlighted in orange.
Extended Data Figure 3: Analysis of number game predictions given a program that accepts {8,80,48} . Otherwise refer to the caption in Extended Data Fig. 2 .
Extended Data Figure 4: Analysis of how hypotheses change between the distillation and fine-tuning stage. A) For the number game, the episodes in Fig. 3 D are categorized by their most likely hypothesis via sparse approximation (Section M.2 of the Methods) for the two models, BBT and Distillation only. The difference in frequency (BBT - Distillation only) is shown on the y-axis. B-i) This is the same analysis for learning logical concepts based on the episodes in Fig. 4 D). Note that the following classes are not included for having an absolute difference less than 0.001: and, or, implies, and if and only if. B-ii) Histogram of the difference in function calls in the hypothesis expressions, showing that fine-tuning leads to fewer function calls, defined as the count of open parentheses.
Run
BBT
Distillation only
Difference
1
-7,929.6
-8,411.3
481.7
2
-7,616.9
-8,235.0
618.1
3
-7,759.5
-8,292.6
533.1
4
-7,769.2
-8,376.9
607.7
5
-7,522.5
-8,092.1
569.6
Extended Data Table 1: Predicting human behavior for novel logical concepts. Bayesian distillation was run five times, and each of these network runs was subsequently fine-tuned with BBT. Performance is reported as the overall log-likelihood as well as the difference in log-likelihood between the models (BBT - Bayesian distillation). All models have fit lapse rates (Section M.4 of the Methods).
Extended Data Figure 5: Analysis of logical concept learning for the concept “largest blue object in the set.” The proportion of participants to answer “yes” for each query object is shown, along with model predictions. Using sparse hypothesis decomposition, the two hypotheses with the largest influence in BBT’s predictions are shown, left to right across panels. The objects included in each hypothesis are highlighted in orange.
Extended Data Figure 6: Analysis of logical concept learning for the concept “is the only blue or green object.” Otherwise refer to the caption in Extended Data Fig. 5 .
Extended Data Figure 7: Analysis of how hypotheses change between the distillation and fine-tuning stage. For Shepard category learning, the episodes in (B and D) are categorized by their most likely hypothesis via sparse approximation for the two models (Section M.2 of the Methods), BBT (young on the left and old on the right) and Distillation only. The results for BBT are shown separately into young and old participants by conditioning on age-specific embedding. PCA of model embeddings for distillation only (left) and human BBT (right) extracted from decoder layer 3. Each point is a different learning episode containing 16 objects in support and all unique objects as query, marked with the color of its (B) most likely hypothesis or (D) number of literals in the rule sampled from the probabilistic context-free grammar as fit to the model judgments via sparse approximation. The difference in frequency (BBT - Distillation only) of different hypothesis (A) and number of literals in a given hypothesis (C), whenever there was a difference observed in the sparse approximation analysis.
Extended Data Figure 8: A & B) Learning curve over blocks for humans, distillation-only, and BBT for young participants (A) and older participants (B), data taken from the Badham et al. [ 3 ] study. The y-axis shows the proportion of correct choices and x-axis is tasks blocks, with the four Shepard category structures (Type-I to Type-IV) shown as distinct lines. The distillation was done from a rational rules model with outlier probability set to 0.12 (i.e. b=2 ). As a result, the distillation only model, unlike the one used in main figure which was rational rules model with outlier probability set to 0.01, displays a clear ordering in difficulty (Type I<Type II<Type III<Type IV) when simulated on synthetically generated Shepard tasks. In addition, BBT ( b=2 ) can reproduce the exact ordering of learning difficulties observed in humans, including the swap between type IV and type II observed in older participants, unlike distillation-only model and fine-tuning only model.
Extended Data Figure 9: Three episodes for the compositional instruction learning task, including the test episode (A) and two fine-tuning episodes (B). Human participants and models are provided with the study instructions, which map linguistic expressions (pseudowords) to output sequences (color circles). The task is to produce the output sequences for the query instructions (the outputs are shown here, but not to participants). Each episode is generated by a grammar of rewrite rules, as explained in Section M.8 .
McCoy & Griffiths (2025, henceforth M&G) suggest that a Bayesian prior can be distilled into Artificial Neural Networks (ANNs) through Model-Agnostic Meta-Learning (MAML, Finn et al., 2017). They support this empirically by showing that meta-trained networks demonstrate formal language learning abilities comparable to Yang & Piantadosi (2023)'s Bayesian learner, significantly outperforming standard ANNs. We point out that under the standard interpretation of a prior, M&G's procedure does not actually instill one; it merely initializes network weights favorably, leaving the objective function unchanged. We then consider a more permissive interpretation, where the system as a whole can be seen as implementing a Bayesian learner even without an explicit prior in the objective. We show that this interpretation faces nontrivial challenges. Finally, we assess how well MAML approximates the empirical results of Bayesian learning, showing that unlike genuine Bayesian learners, M&G's model overfits and generalizes poorly to unseen data.
Using predictive models to explain cognition requires more than accurate behavioral predictions. Input ablations offer an appealing route: remove information from a model and interpret the resulting performance change as evidence of its importance for behavior. Yet this inference assumes that the model's dependence on information reflects the dependence of the process generating the behavior. We test it in two synthetic sequential bandit tasks with known generating policies, where past choices can remain informative when feedback is unavailable to a predictor. We compare GRUs and Transformers trained from scratch, a fine-tuned LLaMA model, and cognitive models across systematically varied reward contributions. Our analyses distinguish prediction after training without reward observations from the response of a fixed predictor to donor-reward replacement. Three findings emerge. First, in the restless task, neural models trained without rewards predict held-out choices better than four simple training-fitted behavioral baselines. Second, under matched donor replacement, accurate predictors can respond much less than the known generator. Third, at some reward weights, neural networks predict better than a pooled reinforcement-learning model but have less faithful changes in choice probabilities; the model ordering differs between the two tasks. These independent-test results separate information sufficient for prediction from response fidelity under a specified ablation in sequential choice. They motivate validating model-ablation responses independently of predictive performance before using them to infer how the observed behavior was generated.
Hanbo Xie
School of Psychological and Brain Sciences Georgia Institute of Technology Atlanta, GA 30332, USA
Foundation models have been increasingly applied to behavioral science domains such as psychology, sociology, and economics. While these models show promise in tasks such as survey response prediction and human-subject experiment simulation, there remains no systematic understanding of how well they perform across diverse behavioral science tasks. We introduce BehaviorBench, a comprehensive benchmark that evaluates foundation models along four core capabilities: (1) behavior prediction and simulation, (2) strategic decision-making, (3) subject-trait inference, and (4) behavioral knowledge application. Crucially, BehaviorBench evaluates model outputs at both the individual and distributional levels, capturing not only per-subject accuracy but also population-level alignment, an essential requirement for behavioral validity. Our evaluation shows that BehaviorBench remains challenging for leading general-purpose LLMs and behavior foundation models that are specifically trained with behavioral data. We find that individual-level and distributional performance do not always align. General-purpose LLMs tend to underestimate the diversity of human responses, whereas behavior foundation models often lag behind at individual-level prediction. Our investigation further demonstrates how fine-tuning on diverse behavioral data can improve both individual-level prediction and distributional alignment, balancing these two objectives. Our results highlight the importance of evaluation at both individual and distributional levels, establishing BehaviorBench as a foundation for developing and assessing behaviorally aligned AI systems. Our BehaviorBench and models can be accessed via https://umich-foreseer.github.io/behaviorbench/
Jin Huang, Yutong Xie, Wanli Song +5
University of Michigan · MobLab · Stanford University +1