cs.CLNov 7, 2026

Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations

Authors: Sion Weatherhead, Ben Newell, Aaron Belbasis, Flora Salim

Organizations: University of New South Wales, Sydney · Aurecon Group, Melbourne

Abstract

Large language model (LLM) agents are increasingly used as synthetic participants in social, behavioural, and decision-making research, yet it remains unclear whether they can express graded cognitive diversity as assigned when engaging in simulated behaviour. We introduce a cognition-suite evaluation that tests whether simulated populations reproduce human variation in reasoning, self-reported traits, their associations, and downstream behaviour. Agents conditioned on synthetic or human-derived profiles of cognitive reflection, Need for Cognition, and Big Five personality answer the same suite as a cohort of 473 adults: two newly assembled tests, a Cognitive Reflection Test (CRT) and a Behavioural Reflection Test (BRT) whose decision scenarios measure reflection in behaviour, alongside an established CRT and the two trait questionnaires. Across seven full-battery simulation runs of four models, agents recover Need for Cognition and Big Five trait distributions on self-report questionnaires, but fall short on simulating diversity of cognitive task performance and downstream behaviours. In the four Claude Sonnet 4.6 runs, every agent scores full marks on both CRTs, whatever reflection level its persona was assigned. In the other three runs agents make intuitive-lure errors (the tempting wrong answers CRT items invite), but 49 to 85% score full marks, against under 10% in the human sample. Claude Sonnet 5 comes closest to people's item-by-item pattern of lures; the other two runs give fewer lures and vary less between items. In the BRT scenarios, every run over-produces 8 to 11 of 15 coded behaviours by at least 20 percentage points, and people's links between reflection and behaviour carry over only in part. Useful synthetic populations must preserve not only trait descriptors but the structured variation in reasoning, error type, and behaviour those descriptors are meant to organise.

Figures & tables

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?

    Mar 29, 2026Yuxuan Gu, Lunjun Liu, Xiaocheng Feng +4LLM Evaluation

  2. AI Simulations Grounded in Rich Self-Reports Increase Accuracy and Reduce Distortion

    Nov 15, 2024Joon Sung Park, Carolyn Q. Zou, Jonne Kamphorst +8Personality Modeling in Language ModelsHuman Behavior Simulation