CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Authors: Lance Ying, Jinzhou Wu, Yingshan Susan Wang, Shivam Aarya, Luca M. Schulze Buschoff, Harry Chen, Katherine M. Collins, Andrea de Varda, Shuhao Fu, Sean Dae Houlihan, Akshay K. Jagadish, Guangyuan Jiang, Samuel Kiegeland, Tetsu Kurumisawa, Rongzhi Liu, Ryan Liu, Ningshan Ma, Kathryn McGregor, Younes Strittmatter, Polina Tsvilodub, Jacob Hoover Vigly, Sarah Wu, Enjie Xu, Yiling Yun, Kelsey Allen, Tyler Brooke-Wilson, Brian Christian, Evelina Fedorenko, Michael C. Frank, Michael Franke, Tao Gao, Samuel J. Gershman, Robert D. Hawkins, Jennifer Hu, Julian Jara-Ettinger, Max Kleiman-Weiner, Sydney Levine, Tal Linzen, Hongjing Lu, Timothy O'Donnell, Desmond C. Ong, Steven T. Piantadosi, Rebecca Saxe, Eric Schulz, Tianmin Shu, Felix A. Sosa, Ilia Sucholutsky, Tan Zhi-Xuan, Tomer Ullman, Fei Xu, Ilker Yildirim, Jian-Qiao Zhu, Thomas L. Griffiths, Tobias Gerstenberg, Kevin Smith, Joshua B. Tenenbaum
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models' improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability (R2=0.93 on text, 0.95 on image, and 0.92 on video) with the best models achieving R2=0.59 on text, 0.58 on image, and 0.43 on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.
Figures & tables
Figure 1: CogGym workflow. CogGym proceeds in three stages. First, original experiments implemented in diverse frameworks are converted into a standardized Experiment Markup Language (EML) representation through an AI agent with human-in-the-loop supervision. Second, the standardized experiments are rendered and administered to human participants and AI models. Lastly, human and model responses are analyzed jointly to evaluate experiment validity and model–human alignment.
Figure 2: Topics represented in CogGym. CogGym experiments span a broad range of commonsense reasoning tasks, including physical reasoning, causal learning, concept learning, social cognition, moral judgment, probabilistic inference, pragmatic language use, and more. The studies were sourced from over 30 contributing research labs. These topics are derived from keywords reported in the source papers.
Figure 3: Example of an EML experiment. A : EML specification for the experiment, which encodes the instructions, trials (stimuli and response queries), and human data of the original study as executable data (adapted from Gerstenberg and Goodman 2012 ). B : The same EML rendered on the CogGym platform as an interactive web experiment for human data collection. C : The matched model prompt automatically constructed from the same EML. Because the human task and the model prompt are generated from a single specification, participants and models receive closely matched instructions, stimuli, and response options, enabling a direct comparison of their responses.
Figure 4: Model performance on CogGym tasks and formal reasoning benchmarks plotted by model release date. Models are color-coded by size, and selected models are labeled. Top: model–human fit R2 across experimental tasks. Middle: normalized distributional divergence between AI models and humans. Bottom: AI models’ performance on GPQA Diamond, MATH, and HumanEval performance. The plots show that models’ improvement on formal reasoning benchmarks is much steeper than models’ improvement on their alignment with humans on commonsense reasoning tasks. Error bars show ±1 standard error across experiments.
Figure 5: Model–human alignment as a function of model size among open-weight models. Each point is one open-weight model. Top row: model–human fit R2 ( ↑ ) rises with scale in every modality (text r=+0.81 , image +0.75 , video +0.81 ). Bottom row: normalized distribution divergence falls with scale in every modality (text r=−0.83 , image −0.69 , video −0.68 ). Error bars are per-model ±1 standard error. Dashed extensions show log-linear extrapolations. Shaded regions are 95% confidence bands, shown more lightly over the extrapolated range.
Figure 6: Gemini 3.1 Pro vs. human split-half, per experiment. Each point is one experiment. (Top) R2 : Human split-half R2 vs. model–human fit R2 ; 94% of experiments fall below the parity diagonal. (Bottom) Distributional Divergence : human split-half normalized divergence (reliability floor) vs. model–human normalized divergence; Error bars in both panels are ±1 SE from 1,000-sample item bootstraps.
Figure 7: Gemini 3.1 Pro’s model–human R2 by topic and modality. Points show the median human–model fit across experiments; horizontal bars span the interquartile range (25th–75th percentiles). A cross marks a topic which lacks sufficient number of experiments. The plots show that models tend to align less with humans on physical reasoning tasks than other topics.
Figure 8: Accuracy and model–human fit for Gemini 3.1 Pro vs Human participants. Each data point indicates one experiment. (A) Mean human versus model accuracy for the 11 experiments that include accuracy metrics. The dashed diagonal marks equal accuracy ( y=x ); points above it favor the model and points below it favor humans. Overall humans participants achieve higher mean accuracy than Gemini 3.1 Pro. (B) Differences between model and human accuracy versus model–human R2 on the same experiments, which shows a positive correlation between Gemini 3.1 Pro performance relative to humans’ and model–human fit.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
R2↑
Norm. Div. ↓
Model
Text (91)
Image (82)
Video (85)
Text (86)
Image (80)
Video (67)
Human reference †
0.93 ± .01
0.95 ± .01
0.92 ± .01
0.05 ± .01
0.07 ± .01
0.07 ± .01
Random baseline
0.07 ± .02
0.05 ± .01
0.03 ± .01
0.28 ± .01
0.26 ± .01
0.29 ± .01
Claude (Anthropic)
Claude Opus 4.7
0.55 ± .03
0.45 ± .04
–
0.10 ± .01
0.17 ± .01
–
Claude Sonnet 4.5
0.52 ± .03
0.38 ± .03
–
0.11 ± .01
0.20 ± .01
–
Appendix
Table 1: Model–human alignment measured by R2 and normalized distribution divergence. All models use default configuration with thinking mode enabled if available. Values following ± show standard errors. † Human R2 values are Spearman–Brown-corrected references from the 172 recoverable archived exports (80 text, 33 image, 59 video); their standard errors are computed across corrected experiment-level R2 values. Parenthetical counts in the header give the number of benchmark experiments eligible for each metric before model-specific missingness. Centaur is the Llama 3.1 70B base model fine-tuned on human behavioral data; all other models are instruction-tuned.
Text
Image
Video
Model
R2↑
R2↑
R2↑
Grok 4.3
0.60
0.19
—
Gemini 3.1 Pro
0.59
0.53
0.23
Claude Opus 4.7
0.58
0.42
—
Claude Sonnet 4.5
0.58
0.24
—
Qwen3.5 Flash 35B
0.57
0.38
—
Appendix
Table 2: Top 10 model–human R2 on the split-half reliable subset (179 experiments), ranked by text R2 . The human reference is Spearman–Brown-corrected using the 172 recoverable archived exports (80 text, 33 image, 59 video).
Runs
Mean R2
95% CI
5
0.49
[0.43, 0.55]
10
0.48
[0.42, 0.54]
15
0.48
[0.42, 0.53]
20
0.49
[0.43, 0.55]
25
0.48
[0.42, 0.54]
30
0.48
[0.42, 0.54]
Appendix
Table 3: Effect of number of model runs on Gemma 4 26B text alignment. Increasing the number of runs from 5 to 50 does not materially improve R2 ; the 95% confidence intervals (1000-sample bootstrap over experiments) overlap across all run counts.
Figure 9: Response diversity: humans vs. repeated model runs. For each item, response variability is computed on the normalized [0,1] response scale, and the distributions are compared across humans and repeated model runs. Values near 0 indicate nearly deterministic responses; broader distributions indicate greater item-level variability. Sampled model responses are markedly less diverse than human responses.
Figure 10: Verbalized-distribution vs. sampled-runs divergence (Gemini 3.1 Pro). Each point is one experiment; x is the normalized divergence from verbalized elicitation and y from 10 sampled runs (lower is better). The vast majority of experiments fall above the parity diagonal, i.e., repeated sampling—being sharply peaked—diverges from the human response distribution more than the verbalized estimate does.
Experiment
Modality
N
Repl. R2
G3.1 Pro R2
Levine (2020), Exp. 1
Text
35
0.86
0.48
Tsvilodub (2025), Exp. 1
Text
30
0.79
0.39
Ying (2023), Exp. 1
Text
30
0.82
0.70
Yoon (2020), Exp. 1
Text
30
0.97
0.95
Hu (2023), Exp. 1
Text
30
0.99
0.93
Aboody (2025), Exp. 2
Image
30
0.89
0.84
Appendix
Table 4: Experiment replication and comparisons. N is the number of participants in each included replication batch; the total sums study participations. Repl. R2 compares replication means against original human means; G3.1 Pro R2 compares Gemini 3.1 Pro against the original humans.
Figure 11: Per-experiment R2 : Claude Opus 4.7 vs. Gemini 3.1 Pro. Each point is one experiment. Overall the plots show strong correlation between the two models. Error bars on both axes are 95% confidence intervals (1000-sample bootstrap percentile, resampling items); experiments whose R2 is too poorly estimated to compare (95% CI width >0.4 on either axis, typically those with very few items) are omitted.
Figure 12: Concept learning (text): the number game [ Bigelow and Piantadosi, 2016 ] . Participants see a few outputs of an unknown program and judge whether a new number is likely to be generated next. Given 15, 39, 35 , only 18% of participants ( n=11 ) say that 41 is likely, but every run of every model says yes, with the same justification: “All previous numbers are odd integers, and 41 is also an odd integer.” Human generalization in this task hedges between rule-based hypotheses (odd numbers) and similarity-based ones (numbers in the teens and thirties); the models commit to the first compact rule that fits.
Figure 13: Pragmatics of number words (text) [ Tsvilodub et al., 2025 ] . After reading “It cost 51,”participantsratetheprobabilityoftencandidateprices;ratingsarerenormalizedwithinparticipantasintheoriginalanalysis.Humansspreadtheirprobabilityacross50, 51,500, 501,1,000, and 1,001,placingonlyabout19%ontheliteral51 ( n=66 ). Gemini places 100% on the literal value in every run and GPT-5.2 about 80% on average ( “Fred explicitly stated the price was 51,indicatingapreciseandliteralvalue”).Opussplits:infiveoftenrunsitplaces90–95%on51, and in the other five it returns the same value for every price, which renormalizes to a flat distribution. None of the models reproduces the human pattern of treating the number as an imprecise report.
Figure 14: 3D shape perception (image): reading the silhouette instead of the physics [ Yildirim et al., 2024 ] . Participants match a cloth-covered target to one of two uncovered objects, a task people solve by reasoning about how cloth settles on a shape. 97% of participants ( n=29 ) identify the bicycle; Gemini and GPT-5.2 choose the table in every run and Opus in nine of ten, describing the draped outline as if it were the object: “The draped cloth outlines a flat rectangular surface with four protruding legs, which perfectly matches the structure of the table” (Gemini).
Figure 15: Theory of mind (image): inferring knowledge from a skipped opportunity [ Aboody et al., 2025 ] . Pirates land at the star and walk directly to where they dig (white square), passing close to a map (green square) without picking it up. Participants infer that the pirates already knew where the treasure was ( M=82 on a 0–100 scale, n=40 ). Opus and GPT-5.2 answer 28 and 36 on average, and GPT-5.2 misdescribes the scene ( “They chose to detour to get the map, suggesting limited prior knowledge” ); Gemini splits between runs that read the scene correctly ( “The pirates skipped the map even though it was very close to their path, suggesting they already knew where the treasure was” , answering 80 – 85 ) and runs that answer 20 – 30 .
An essential problem in artificial intelligence is whether LLMs can simulate human cognition or merely imitate surface-level behaviors, while existing datasets suffer from either synthetic reasoning traces or population-level aggregation, failing to capture authentic individual cognitive patterns. We introduce a benchmark grounded in the longitudinal research trajectories of 217 researchers across diverse domains of artificial intelligence, where each author's scientific publications serve as an externalized representation of their cognitive processes. To distinguish whether LLMs transfer cognitive patterns or merely imitate behaviors, our benchmark deliberately employs a cross-domain, temporal-shift generalization setting. A multidimensional cognitive alignment metric is further proposed to assess individual-level cognitive consistency. Through systematic evaluation of state-of-the-art LLMs and various enhancement techniques, we provide a first-stage empirical study on the questions: (1) How well do current LLMs simulate human cognition? and (2) How far can existing techniques enhance these capabilities?
Claims about the cognitive capabilities of large language models are typically tested by translating behavioral paradigms into text prompts, bypassing the visual interfaces and time pressure through which human participants actually engage with experiments. At the same time, AI agents can now complete online behavioral tasks and produce human-like data, which raises concerns that online datasets in the social and behavioral sciences are increasingly open to contamination. We introduce CogArena, a benchmark of ten experimental paradigms from the social and behavioral sciences. Agents complete the experiments through a standard web browser, working with the same visual stimuli and response interfaces that online participants use. A three-level scoring procedure evaluates whether the agent completed the task, how accurately it performed, and whether its data show the behavioral signatures each paradigm reliably produces in people. Across four multimodal models, each run ten times per task, we find that finishing an experiment and reproducing its signatures are different things. Many sessions did not finish, and those that did reproduced some signatures reliably and others not at all. CogArena provides the tasks, scoring code, and a public leaderboard for new submissions.
Kianté Fernandez, Caitlin Chen, Jialu Zhou +3
Department of Psychology, University of California, Los Angeles
Existing benchmarks for AI reasoning provide limited insight into how closely these capabilities resemble human reasoning in naturalistic contexts. We present an adaptation of the Watson & Holmes detective tabletop game as a new benchmark designed to evaluate reasoning performance using incrementally presented narrative evidence, open-ended questions and unconstrained language responses. An automated grading system was developed and validated against human assessors to enable scalable and replicable performance evaluation. Results show a clear improvement in AI model performance over time. Over nine months of 2025, model performance rose from the lower quartile of the human comparison group to approximately the top 5%. Around half of this improvement reflects steady advancement across successive model releases, while the remainder corresponds to a marked step change associated with reasoning-oriented model architectures. Systematic differences in the performance of AI models compared to humans, dependent on features of the specific detection puzzle, were mostly absent with the exception of a fall in performance for models when solving longer cases (case lengths being in the range of 1900-4000 words), and an advantage at inductive reasoning for reasoning models at early stages of case solving when evidence was scant.