cs.LGAug 23, 2026
SaveMore accurate behavioral predictions with hybrid Bayesian-connectionist models
Organizations: Departments of Computer Science and Psychology, Princeton University · Princeton AI Lab, Princeton University · Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology
Abstract
Researchers must often choose between Bayesian or neural network models of behavior, two paradigms with complementary strengths and weaknesses. An ideal paradigm would facilitate testing many kinds of representations and inductive biases; Bayesian models make this easy, while neural networks do not. Similarly, an ideal paradigm would avoid over-simplifications; neural networks make this easy, while Bayesian models do not. Here, we introduce Bayesian distillation with Behavioral Tuning (BBT) as an approach to getting the best of both traditions. BBT offers a simple recipe for model building: first, a neural network is trained to mimic a Bayesian model through synthetic data, and second, the network is fine-tuned on human behavior to capture additional structure and nuance. Across four case studies in human concept learning, we find that BBT outperforms traditional approaches at predicting human behavior while also revealing psychological insights, resulting in models that can both mimic Bayesian priors and capture heuristics and biases that violate simple modeling assumptions.
Figures & tables
Figure 1: The Bayesian and connectionist paradigms occupy opposite ends of a spectrum. Our Bayesian distillation with Behavioral Tuning (BBT) approach moves along this spectrum in two stages. First, a neural network is pretrained to mimic a Bayesian model through Bayesian distillation (marked as “start” in the figure). The network is then fine-tuned on human behavioral data, shifting it towards the connectionist end of the spectrum until performance on held-out data plateaus (marked as “stop”). Depending on the nature of the behavioral data, the model may stop anywhere along the spectrum. Note that this is a model-fitting procedure and not a hypothesis about cognitive development.
Figure 2: The recipe for BBT modeling involves 1) distilling a Bayesian model into a neural network (via synthetic data from forward sampling; no Bayesian inference required), and then 2) fine-tuning the network on human behavioral data (potentially very limited). During a training “episode”, the network receives a query input (here, on the right, the network is queried on a small blue triangle) and a study set (objects and labels ), which are concatenated and presented together as input. The network then produces a predicted query output (e.g., “no”) which is compared to a target.
Figure 3: The number game task and model predictions. A) Given a set of numbers that a mystery computer program produces, (e.g., ), the task is to predict which other numbers the program is likely to generate (e.g., 20? 53?). The main text describes the three scenarios and how one might reason about them. B) Results for the study set . The y-axis shows the proportion of “yes” answers for each query 1 to 100 (is it likely to generate this number next?) for humans and models. Also shown are the Pearson correlations ( ) and root-mean-squared-error (RMSE) between the human and model judgments. C) Results for the study set . D) PCA of embeddings after distillation only (left) and full BBT training (right), extracted from decoder layer 2. Each point in the scatter corresponds to a support set (averaged over all queries), marked with the color of its most likely hypothesis (via sparse approximation) as fit to the model judgments. After fine-tuning with BBT, the influence of “evens”/“odds” (blue) and “ending in n” (purple) hypotheses dramatically expands.
| Model | Number concepts | Logical concepts | Shepard concepts | Compositional rules |
|---|---|---|---|---|
| Bayesian | -31,332.6 | -79,023.3 | -5,554.4 | |
| Distillation only | -31,435.7 | -77,249.9 | -5,601.1 | -466.4 |
| Fine-tuning only | -31,465.2 | -106,926.3 | -5,924.7 | -1,653.6 |
| BBT | -28,157.9 | -69,423.8 | -5,260.8 | -343.6 |
Table 1: Model predictions evaluated through the log-likelihood of held-out human behavior. The bold shows the best scoring model (higher is better). All models had lapse rates that were fit, as described in Section M.4 of the Methods. Neural networks (Distillation only, Fine-tuning only, and BBT) were trained for five runs, with the reported run selected through a separate validation set.
Figure 4: The logical concept learning task and model predictions. A) Given sets of objects and their category labels (positive examples marked with ), the task is to predict the labels for the th set. Here, the correct rule is “largest blue object in the set.” B) Human and model judgments on the final 8 object sets when learning the rule in Panel A. The y-axis shows the proportion of “yes” answers per object. The x-axis shows the object stimuli, with the object sets divided by vertical lines. Also shown are the Pearson correlations ( ) and root-mean-squared-error (RMSE) between human and model judgments. C) Judgments on the last 8 sets when learning the rule “is the only blue or green object in the set.” D) PCA of model embeddings after the distillation (left) and human fine-tuning (right) stages of BBT training, extracted from decoder layer 3. Each point is a different learning episode containing 24 object sets as study (averaging over 20 different query sets), marked with the color of its most likely hypothesis as fit to the model judgments via sparse approximation. After BBT fine-tuning, the model blurs the separation between Boolean vs. non-Boolean episodes, and the simplest hypotheses (true, false, individual features) expand their influence.
Figure 5: The Shepard concept learning task and model predictions. A) Participants are presented objects sequentially, they are tasked to choose between two categories (1 and 2) on each trial and receive feedback indicating whether their response was correct. Each task is repeated over multiple blocks. In the example shown, the correct rule is “all yellow objects belong to category 1”. B & C) Learning curves over blocks for humans, distillation only networks, and BBT for young (B) and older (C) participants, data taken from the Badham et al. [ 3 ] study. The y-axis shows the proportion of correct choices and x-axis is tasks blocks, with the four Shepard category structures (Type-I to Type-IV) shown as distinct lines. D) PCA of model embeddings after the distillation (left), BBT (conditioned on young embedding; middle), and BBT (conditioned on older group embedding; right) extracted from decoder layer 3. Each point is a different learning episode containing 16 objects in support and all unique objects as query, marked with the color of its most likely hypothesis from four out of six Shepard’s category learning structures as fit to the model judgments via sparse approximation. The first component in BBT (old) can clearly separate Types II-IV (orange, green, and red) and Type I (blue), whereas in BBT (young) younger participants only separate Type II from Type III, and the second principal component separates Type I (blue) from other types for both BBT conditioned on young and old. Although both BBT (young) and BBT (old) show stronger representation of Type I hypothesis compared to Distillation only, the increase is stronger in younger than older.
Figure 6: Few-shot learning of compositional instructions. The task involves responding to strings of pseudowords (inputs) with sequences of colors (outputs). A) Participants learned these 14 study instructions which are composed of four primitives and three functions (heading were not provided to the participants). B) After learning the study instructions, participants were asked to respond to 10 query instructions (4 of 10 are shown here). The most frequent 4 responses to each query instruction are shown, labeled in parentheses with the count for people and the percent of samples for the models. Superscripts indicate which responses are correct ( ∗ ) or consistent with one-to-one biases (1-to-1) or iconic concatenation biases (IC).
Extended Data Figure 1: Interpreting the Bayesian model of the number game as a multilayer perceptron. Input layer : The study set is represented as a count vector whose th entry records the number of times integer appears in , similar to a “bag of words” input vector. For instance, if “4” appears twice in , . Hidden layer : The activation of each hidden unit equals the posterior probability of the corresponding hypothesis, . The net input to the unit corresponding to each hypothesis ( ) is the dot product between the input vector and weights that reflect the log-likelihood, if , otherwise . The weights for the hypothesis “evens” are shown on the arrows. The biases for the hidden units are the log prior, . The layer has a softmax activation function. Output : The network simultaneously outputs the posterior predictive probability for every possible query , where the lapse rate is optimized. Prior-fit training : For behavioral fine-tuning, the network is initialized at the original Bayesian model parameters. Only the 5084 prior logits corresponding to and are trained with L-BFGS, maximizing the log-likelihood of the human data with a regularizer (0.001) that encourages a higher-entropy prior. Full-fit training : This model extends the prior-fit model by including the log-likelihood matrix trainable. Each row is parameterized by unconstrained logits and normalized using log-softmax, using softened variants of their Bayesian model values for initialization. The indicator function in the output layer is relaxed to reflect that membership is now graded rather than binary, replacing with The code for training these models is released as well.
Extended Data Figure 2: Analysis of number game predictions given a program that accepts . The proportion of participants to answer “yes” for each query 1 to 100 is shown, along with model predictions. Using sparse hypothesis decomposition for interpretation, the three hypotheses with the largest influence in BBT’s predictions are shown, left to right across panels. The numbers included in each hypothesis are highlighted in orange.
Extended Data Figure 3: Analysis of number game predictions given a program that accepts . Otherwise refer to the caption in Extended Data Fig. 2 .
Extended Data Figure 4: Analysis of how hypotheses change between the distillation and fine-tuning stage. A) For the number game, the episodes in Fig. 3 D are categorized by their most likely hypothesis via sparse approximation (Section M.2 of the Methods) for the two models, BBT and Distillation only. The difference in frequency (BBT - Distillation only) is shown on the y-axis. B-i) This is the same analysis for learning logical concepts based on the episodes in Fig. 4 D). Note that the following classes are not included for having an absolute difference less than 0.001: and, or, implies, and if and only if. B-ii) Histogram of the difference in function calls in the hypothesis expressions, showing that fine-tuning leads to fewer function calls, defined as the count of open parentheses.
| Run | BBT | Distillation only | Difference |
|---|---|---|---|
| 1 | -7,929.6 | -8,411.3 | 481.7 |
| 2 | -7,616.9 | -8,235.0 | 618.1 |
| 3 | -7,759.5 | -8,292.6 | 533.1 |
| 4 | -7,769.2 | -8,376.9 | 607.7 |
| 5 | -7,522.5 | -8,092.1 | 569.6 |
Extended Data Table 1: Predicting human behavior for novel logical concepts. Bayesian distillation was run five times, and each of these network runs was subsequently fine-tuned with BBT. Performance is reported as the overall log-likelihood as well as the difference in log-likelihood between the models (BBT - Bayesian distillation). All models have fit lapse rates (Section M.4 of the Methods).
Extended Data Figure 5: Analysis of logical concept learning for the concept “largest blue object in the set.” The proportion of participants to answer “yes” for each query object is shown, along with model predictions. Using sparse hypothesis decomposition, the two hypotheses with the largest influence in BBT’s predictions are shown, left to right across panels. The objects included in each hypothesis are highlighted in orange.
Extended Data Figure 6: Analysis of logical concept learning for the concept “is the only blue or green object.” Otherwise refer to the caption in Extended Data Fig. 5 .
Extended Data Figure 7: Analysis of how hypotheses change between the distillation and fine-tuning stage. For Shepard category learning, the episodes in (B and D) are categorized by their most likely hypothesis via sparse approximation for the two models (Section M.2 of the Methods), BBT (young on the left and old on the right) and Distillation only. The results for BBT are shown separately into young and old participants by conditioning on age-specific embedding. PCA of model embeddings for distillation only (left) and human BBT (right) extracted from decoder layer 3. Each point is a different learning episode containing 16 objects in support and all unique objects as query, marked with the color of its (B) most likely hypothesis or (D) number of literals in the rule sampled from the probabilistic context-free grammar as fit to the model judgments via sparse approximation. The difference in frequency (BBT - Distillation only) of different hypothesis (A) and number of literals in a given hypothesis (C), whenever there was a difference observed in the sparse approximation analysis.
Extended Data Figure 8: A & B) Learning curve over blocks for humans, distillation-only, and BBT for young participants (A) and older participants (B), data taken from the Badham et al. [ 3 ] study. The y-axis shows the proportion of correct choices and x-axis is tasks blocks, with the four Shepard category structures (Type-I to Type-IV) shown as distinct lines. The distillation was done from a rational rules model with outlier probability set to 0.12 (i.e. ). As a result, the distillation only model, unlike the one used in main figure which was rational rules model with outlier probability set to 0.01, displays a clear ordering in difficulty (Type I<Type II<Type III<Type IV) when simulated on synthetically generated Shepard tasks. In addition, BBT ( ) can reproduce the exact ordering of learning difficulties observed in humans, including the swap between type IV and type II observed in older participants, unlike distillation-only model and fine-tuning only model.
Extended Data Figure 9: Three episodes for the compositional instruction learning task, including the test episode (A) and two fine-tuning episodes (B). Human participants and models are provided with the study instructions, which map linguistic expressions (pseudowords) to output sequences (color circles). The task is to produce the output sequences for the query instructions (the outputs are shown here, but not to participants). Each episode is generated by a grammar of rewrite rules, as explained in Section M.8 .