cs.AISep 18, 2026
SaveCogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Abstract
Understanding and modeling human intelligence are parallel goals shared by artificial intelligence (AI) and cognitive science. As AI systems grow increasingly capable, in what ways do model responses resemble human responses, and where do they systematically diverge? The sheer breadth and diversity of the tasks humans can perform and think about pose a challenge for scalable and rigorous comparison between humans and models. We introduce CogGym, a scalable, unified framework grounded in cognitive science for systematically comparing model and human behavior on matched experimental trials. CogGym uses a semi-automated, human-in-the-loop pipeline to standardize diverse experimental paradigms into a task-agnostic Experiment Markup Language (EML), enabling reproducible and faithful comparison at scale. For initial release, we curate and standardize 258 cognitive experiments from 100 papers that focuses on human commonsense reasoning, and evaluate 50 large language models against human responses. We find a clear scaling trend where larger and more recent AI models better reproduce human judgments. Yet AI models' improvement on such common reasoning tasks is considerably slower than the gains observed on formal-reasoning benchmarks like math and coding, and model--human fit remains well below human splithalf reliability ( on text, on image, and on video) with the best models achieving on text, on image, and on video experiments. We intend for CogGym to provide a living evaluation framework that continually incorporates new cognitive science experiments to characterize where model behavior resembles human behavior, where it systematically diverges, and how those patterns change as models and experiments evolve.
Figures & tables
Figure 1: CogGym workflow. CogGym proceeds in three stages. First, original experiments implemented in diverse frameworks are converted into a standardized Experiment Markup Language (EML) representation through an AI agent with human-in-the-loop supervision. Second, the standardized experiments are rendered and administered to human participants and AI models. Lastly, human and model responses are analyzed jointly to evaluate experiment validity and model–human alignment.
Figure 2: Topics represented in CogGym. CogGym experiments span a broad range of commonsense reasoning tasks, including physical reasoning, causal learning, concept learning, social cognition, moral judgment, probabilistic inference, pragmatic language use, and more. The studies were sourced from over 30 contributing research labs. These topics are derived from keywords reported in the source papers.
Figure 3: Example of an EML experiment. A : EML specification for the experiment, which encodes the instructions, trials (stimuli and response queries), and human data of the original study as executable data (adapted from Gerstenberg and Goodman 2012 ). B : The same EML rendered on the CogGym platform as an interactive web experiment for human data collection. C : The matched model prompt automatically constructed from the same EML. Because the human task and the model prompt are generated from a single specification, participants and models receive closely matched instructions, stimuli, and response options, enabling a direct comparison of their responses.
Figure 4: Model performance on CogGym tasks and formal reasoning benchmarks plotted by model release date. Models are color-coded by size, and selected models are labeled. Top: model–human fit across experimental tasks. Middle: normalized distributional divergence between AI models and humans. Bottom: AI models’ performance on GPQA Diamond, MATH, and HumanEval performance. The plots show that models’ improvement on formal reasoning benchmarks is much steeper than models’ improvement on their alignment with humans on commonsense reasoning tasks. Error bars show standard error across experiments.
Figure 5: Model–human alignment as a function of model size among open-weight models. Each point is one open-weight model. Top row: model–human fit ( ) rises with scale in every modality (text , image , video ). Bottom row: normalized distribution divergence falls with scale in every modality (text , image , video ). Error bars are per-model standard error. Dashed extensions show log-linear extrapolations. Shaded regions are 95% confidence bands, shown more lightly over the extrapolated range.
Figure 6: Gemini 3.1 Pro vs. human split-half, per experiment. Each point is one experiment. (Top) : Human split-half vs. model–human fit ; of experiments fall below the parity diagonal. (Bottom) Distributional Divergence : human split-half normalized divergence (reliability floor) vs. model–human normalized divergence; Error bars in both panels are SE from 1,000-sample item bootstraps.
Figure 7: Gemini 3.1 Pro’s model–human by topic and modality. Points show the median human–model fit across experiments; horizontal bars span the interquartile range (25th–75th percentiles). A cross marks a topic which lacks sufficient number of experiments. The plots show that models tend to align less with humans on physical reasoning tasks than other topics.
Figure 8: Accuracy and model–human fit for Gemini 3.1 Pro vs Human participants. Each data point indicates one experiment. (A) Mean human versus model accuracy for the 11 experiments that include accuracy metrics. The dashed diagonal marks equal accuracy ( ); points above it favor the model and points below it favor humans. Overall humans participants achieve higher mean accuracy than Gemini 3.1 Pro. (B) Differences between model and human accuracy versus model–human on the same experiments, which shows a positive correlation between Gemini 3.1 Pro performance relative to humans’ and model–human fit.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
| Norm. Div. | ||||||
| Model | Text (91) | Image (82) | Video (85) | Text (86) | Image (80) | Video (67) |
| Human reference † | 0.93 .01 | 0.95 .01 | 0.92 .01 | 0.05 .01 | 0.07 .01 | 0.07 .01 |
| Random baseline | 0.07 .02 | 0.05 .01 | 0.03 .01 | 0.28 .01 | 0.26 .01 | 0.29 .01 |
| Claude (Anthropic) | ||||||
| Claude Opus 4.7 | 0.55 .03 | 0.45 .04 | – | 0.10 .01 | 0.17 .01 | – |
| Claude Sonnet 4.5 | 0.52 .03 | 0.38 .03 | – | 0.11 .01 | 0.20 .01 | – |
Table 1: Model–human alignment measured by and normalized distribution divergence. All models use default configuration with thinking mode enabled if available. Values following show standard errors. † Human values are Spearman–Brown-corrected references from the 172 recoverable archived exports (80 text, 33 image, 59 video); their standard errors are computed across corrected experiment-level values. Parenthetical counts in the header give the number of benchmark experiments eligible for each metric before model-specific missingness. Centaur is the Llama 3.1 70B base model fine-tuned on human behavioral data; all other models are instruction-tuned.
| Text | Image | Video | |
|---|---|---|---|
| Model | |||
| Grok 4.3 | 0.60 | 0.19 | — |
| Gemini 3.1 Pro | 0.59 | 0.53 | 0.23 |
| Claude Opus 4.7 | 0.58 | 0.42 | — |
| Claude Sonnet 4.5 | 0.58 | 0.24 | — |
| Qwen3.5 Flash 35B | 0.57 | 0.38 | — |
Table 2: Top 10 model–human on the split-half reliable subset (179 experiments), ranked by text . The human reference is Spearman–Brown-corrected using the 172 recoverable archived exports (80 text, 33 image, 59 video).
| Runs | Mean | 95% CI |
|---|---|---|
| 5 | 0.49 | [0.43, 0.55] |
| 10 | 0.48 | [0.42, 0.54] |
| 15 | 0.48 | [0.42, 0.53] |
| 20 | 0.49 | [0.43, 0.55] |
| 25 | 0.48 | [0.42, 0.54] |
| 30 | 0.48 | [0.42, 0.54] |
Table 3: Effect of number of model runs on Gemma 4 26B text alignment. Increasing the number of runs from 5 to 50 does not materially improve ; the 95% confidence intervals (1000-sample bootstrap over experiments) overlap across all run counts.
Figure 9: Response diversity: humans vs. repeated model runs. For each item, response variability is computed on the normalized response scale, and the distributions are compared across humans and repeated model runs. Values near indicate nearly deterministic responses; broader distributions indicate greater item-level variability. Sampled model responses are markedly less diverse than human responses.
Figure 10: Verbalized-distribution vs. sampled-runs divergence (Gemini 3.1 Pro). Each point is one experiment; is the normalized divergence from verbalized elicitation and from sampled runs (lower is better). The vast majority of experiments fall above the parity diagonal, i.e., repeated sampling—being sharply peaked—diverges from the human response distribution more than the verbalized estimate does.
| Experiment | Modality | Repl. | G3.1 Pro | |
| Levine (2020), Exp. 1 | Text | 35 | 0.86 | 0.48 |
| Tsvilodub (2025), Exp. 1 | Text | 30 | 0.79 | 0.39 |
| Ying (2023), Exp. 1 | Text | 30 | 0.82 | 0.70 |
| Yoon (2020), Exp. 1 | Text | 30 | 0.97 | 0.95 |
| Hu (2023), Exp. 1 | Text | 30 | 0.99 | 0.93 |
| Aboody (2025), Exp. 2 | Image | 30 | 0.89 | 0.84 |
Table 4: Experiment replication and comparisons. is the number of participants in each included replication batch; the total sums study participations. Repl. compares replication means against original human means; G3.1 Pro compares Gemini 3.1 Pro against the original humans.
Figure 11: Per-experiment : Claude Opus 4.7 vs. Gemini 3.1 Pro. Each point is one experiment. Overall the plots show strong correlation between the two models. Error bars on both axes are 95% confidence intervals (1000-sample bootstrap percentile, resampling items); experiments whose is too poorly estimated to compare (95% CI width on either axis, typically those with very few items) are omitted.
Figure 12: Concept learning (text): the number game [ Bigelow and Piantadosi, 2016 ] . Participants see a few outputs of an unknown program and judge whether a new number is likely to be generated next. Given 15, 39, 35 , only of participants ( ) say that 41 is likely, but every run of every model says yes, with the same justification: “All previous numbers are odd integers, and 41 is also an odd integer.” Human generalization in this task hedges between rule-based hypotheses (odd numbers) and similarity-based ones (numbers in the teens and thirties); the models commit to the first compact rule that fits.
Figure 13: Pragmatics of number words (text) [ Tsvilodub et al., 2025 ] . After reading “It cost 50, 500, 1,000, and 19%51 ( ). Gemini places on the literal value in every run and GPT-5.2 about on average ( “Fred explicitly stated the price was 9095%51, and in the other five it returns the same value for every price, which renormalizes to a flat distribution. None of the models reproduces the human pattern of treating the number as an imprecise report.
Figure 14: 3D shape perception (image): reading the silhouette instead of the physics [ Yildirim et al., 2024 ] . Participants match a cloth-covered target to one of two uncovered objects, a task people solve by reasoning about how cloth settles on a shape. of participants ( ) identify the bicycle; Gemini and GPT-5.2 choose the table in every run and Opus in nine of ten, describing the draped outline as if it were the object: “The draped cloth outlines a flat rectangular surface with four protruding legs, which perfectly matches the structure of the table” (Gemini).
Figure 15: Theory of mind (image): inferring knowledge from a skipped opportunity [ Aboody et al., 2025 ] . Pirates land at the star and walk directly to where they dig (white square), passing close to a map (green square) without picking it up. Participants infer that the pirates already knew where the treasure was ( on a 0–100 scale, ). Opus and GPT-5.2 answer and on average, and GPT-5.2 misdescribes the scene ( “They chose to detour to get the map, suggesting limited prior knowledge” ); Gemini splits between runs that read the scene correctly ( “The pirates skipped the map even though it was very close to their path, suggesting they already knew where the treasure was” , answering – ) and runs that answer – .
More in this topic
LLM Evaluation
Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.
See all 2,425 papers