Hypothesis-guided discovery of cognitive algorithms via program refinement

Huiwen Alex Yang; Mark K. Ho; Bill D. Thompson

cs.AIOct 1, 2026

Hypothesis-guided discovery of cognitive algorithms via program refinement

Authors: Huiwen Alex Yang, Mark K. Ho, Bill D. Thompson

Organizations: Department of Psychology University of California, Berkeley · Department of Psychology New York University

Abstract

Developing cognitive models of algorithmic reasoning from behavioral data is a central problem in cognitive science that challenges current methods. Traditional approaches to cognitive modeling are interpretable and benefit from human expertise, but lack flexibility and scalability. Emerging techniques using large language models (LLMs) for de novo generation of cognitive models are scalable and flexible, but lack a role for human expertise and have mostly been applied to simpler tasks than algorithm recovery. We propose a hybrid system that treats discovery of cognitive algorithms as a program refinement problem. Human-created cognitive models are expressed as probabilistic programs and provided to a system of LLM agents with a mandate to: identify mismatches between model and behavior; propose code-level modifications within researcher-specified constraints; and verify structural fidelity. Revisions propagate to a probabilistic inference module that performs inference for latent variables and data likelihood computations. We evaluate the pipeline on human behavior in a problem-solving paradigm that exposes a variety of cognitive algorithms. Revised models consistently improve model fit relative to ancestral models and reveal a small set of recurring innovations that capture meaningful behavioral variability in this task.

Figures & tables

Figure 1: Strategy discovery and evaluation system

Figure 2: Behavioral experiment. ( A ) Task Structure: first five actions in an example trial. ( B ) Procedure: instructions, two practice trials, and five scored trials, followed by a strategy description. ( C ) Experimental conditions defined by board size (small vs. large) and task structure (simple vs. complex DAG), yielding three conditions. Representative grids and DAGs for each condition, along with a set of working orderings.

Figure 3: Base strategies provide adequate baseline fit, and revised strategies yield improved model fit for a substantial subset of participants. ( A ) Kernel density estimates of per-step log marginal likelihood (log ML) for the best-fitting base strategy, shown by condition (x-axis shared with panel C). ( B ) Improvement from revised strategies by condition, shown as per-step log ML gain (gain = revised - base). Half-violins show the distribution, points show individual participants, and dashed lines indicate condition means. The y-axis is truncated to [0, 0.35] for clarity (9/139 participants have higher gain and exceed this range). ( C ) Participant-level comparison of best base vs. best overall (base or revised) fit; points above the diagonal indicate improvement. Color denotes the family of the best revised strategy; shape denotes the best base strategy. ( D ) Same comparison as (C), broken down by strategy family and condition, with the y-axis showing the best revised variant within each family.

Figure 4: A representative case of a participant who is reasonably well fit by the base ColorBlocking strategy but exhibits improved fit under a revised variant. ( A ) Core logic of the base CB strategy, which predicts the behavior to complete one color at a time. ( B ) Participant behavior data. ( C ) Core logic of the revised strategy variant CB_damaged_2 . Annotations highlight key modifications to the base strategy. Modifications capture the participant’s behavior to repair previously damaged colors.

Strategy Family	Size	LLM-generated cluster description
ColorBlocking	161	Prioritize recent-damage repair before resuming default strategy
ColorBlocking	44	Damage-aware color-blocking with flexible partial completion
ColorBlocking	8	Within-color tie-breaking with consistent index order
Spatial	8	Spatial scanning with temporary damage-repair mode
Spatial	3	Probabilistic local repair and repeat after events
TwoPhase	106	Two-phase damage-aware repair and priority policy

Table 1: Top three revision clusters per strategy family, ranked by size (# revisions within the cluster). Cluster descriptions are LLM-generated for interpretation only and are not used in modeling or analysis. Full results are in Appendix C .

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Condition	$N$	Finite fits (%)	Mean [CI 95 ]	SD	Avg. traj. length
small-complex	49	100.0	$-1.544$ [ $-1.680$ , $-1.414$ ]	$0.489$	$39.7$
small-simple	51	100.0	$-1.464$ [ $-1.598$ , $-1.335$ ]	$0.479$	$38.4$
large-simple	41	95.1	$-2.577$ [ $-2.693$ , $-2.465$ ]	$0.373$	$125.6$

Appendix

Table 2: Descriptive summary of best-fitting base model performance by condition. Values reported here are per-step log-likelihoods, computed as the log marginal likelihoods of the best-fitting base strategy model normalized by trajectory length. 95% confidence intervals are bootstrap percentile intervals (

B=2000

Size	Strategy Family	LLM-generated cluster description
ColorBlocking Strategy Family
161	Prioritize recent-damage repair before resuming default strategy	After a turn-off/damage event, temporarily focus on repairing or avoiding the most recently damaged color(s) using a stable within-color click order, then resume the default color-blocking policy.
44	Damage-aware color-blocking with flexible partial completion	A damage-aware color-blocking strategy that maintains memory of recently damaged colors and flexibly switches between continuing a color block and repairing or reordering priorities.
8	Within-color tie-breaking with consistent index order	When selecting within a color, ties among same-color OFF lights are resolved using a deterministic spatial/index sweep instead of random selection.
4	Occasional post-completion or late-stage exploratory actions	Rare exploratory deviations from strict termination, such as additional clicks after completion or near-completion, to capture behavioral variability.
3	Selection stickiness — persist with chosen color until ON_COLOR	A persistence bias that keeps selecting the same color until it is fully ON, only switching when external state changes require it.

Appendix

Table 3: Summary of revision clusters across strategy families. For each family, all clusters, ranked by size (number of revisions in the cluster), are shown. Descriptions are LLM-generated for interpretive purposes and are not used in the system or analysis.

Strategy Family	Cluster	Variant	Count	%
TwoPhase (TP)
TP	repair	TP_repair_1	22	29.3%
		TP_repair_3	2	2.7%
TP	damaged	TP_damaged_2	9	12.0%
		TP_damaged_3	3	4.0%
		TP_damaged_1	3	4.0%

Appendix

Table 4: Number of participants best-fitted by each revised strategy, grouped by base strategy family, cluster type, and specific variant (N = 75).

Computational Cognitive Modeling

Computational cognitive modeling uses formal models and simulations to explain how people perceive, learn, remember, reason, and choose.

See all 126 papers

Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior
May 6, 2026Hanbo Xie, Akshay K. Jagadish, Lan Pan +1Computational Cognitive Modeling Human Behavior Modeling
School of Psychology, Georgia Institute of Technology · Princeton AI Lab, Princeton University · Center of Excellence for Computational Cognition, Georgia Institute of Techonology
Induction and Inquiry via Probabilistic Reasoning over Language and Code
Sep 1, 2026Wasu Top Piriyakulkij, Sam Acquaviva, Cassidy Langenfeld +2Cognitive Modeling Active Learning
1Cornell University · 2Massachusetts Institute of Technology
Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans
Sep 7, 2026Akshay K. Jagadish, Younes Strittmatter, Nori Jacoby +4Computational Cognitive Modeling Human Behavior Simulation
Princeton University · Cornell University · Helmholtz Munich

May 6, 2026J/K move · Enter open · S save

Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior

Hanbo Xie, Akshay K. Jagadish, Lan Pan, Robert C. Wilson

School of Psychology, Georgia Institute of Technology · Princeton AI Lab, Princeton University · Center of Excellence for Computational Cognition, Georgia Institute of Techonology

Computational Cognitive Modeling Human Behavior Modeling

Open paper AskPDF arXiv

Computational cognitive models discovered using large language models have so far relied solely on behavioral data. However, it is well-known that models produced from the behavioral trajectory alone are typically under-determined. In this work, we explore the use of Think Aloud traces as an additional form of data constraint during automated model discovery. When applied to the domain of risky decision-making, we find that the models discovered with think-aloud achieve significantly improved predictive performance on held-out data. Additionally, we find that the discovered models belong to different structural classes than those discovered from behavior alone for the majority of participants (69.4%), specifically, it shifts from Explicit comparator towards Integrated utility. These results suggest that process-level language data not only improve model fit, but also systematically reshape the structure of the discovered cognitive models, enabling the identification of mechanisms that are not recoverable from behavior alone.

Abstract

Computational Cognitive Modeling

Explore similar work

Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior

Induction and Inquiry via Probabilistic Reasoning over Language and Code

Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans