Imperfectly Imperfect: Evaluating Graded Cognitive Diversity in LLM Persona Simulations
Organizations: University of New South Wales, Sydney · Aurecon Group, Melbourne
Abstract
Large language model (LLM) agents are increasingly used as synthetic participants in social, behavioural, and decision-making research, yet it remains unclear whether they can express graded cognitive diversity as assigned when engaging in simulated behaviour. We introduce a cognition-suite evaluation that tests whether simulated populations reproduce human variation in reasoning, self-reported traits, their associations, and downstream behaviour. Agents conditioned on synthetic or human-derived profiles of cognitive reflection, Need for Cognition, and Big Five personality answer the same suite as a cohort of 473 adults: two newly assembled tests, a Cognitive Reflection Test (CRT) and a Behavioural Reflection Test (BRT) whose decision scenarios measure reflection in behaviour, alongside an established CRT and the two trait questionnaires. Across seven full-battery simulation runs of four models, agents recover Need for Cognition and Big Five trait distributions on self-report questionnaires, but fall short on simulating diversity of cognitive task performance and downstream behaviours. In the four Claude Sonnet 4.6 runs, every agent scores full marks on both CRTs, whatever reflection level its persona was assigned. In the other three runs agents make intuitive-lure errors (the tempting wrong answers CRT items invite), but 49 to 85% score full marks, against under 10% in the human sample. Claude Sonnet 5 comes closest to people's item-by-item pattern of lures; the other two runs give fewer lures and vary less between items. In the BRT scenarios, every run over-produces 8 to 11 of 15 coded behaviours by at least 20 percentage points, and people's links between reflection and behaviour carry over only in part. Useful synthetic populations must preserve not only trait descriptors but the structured variation in reasoning, error type, and behaviour those descriptors are meant to organise.
Figures & tables
| Condition | CRT2 gap | bCRT gap | Calib. RMSE | Assoc. MAE | BRT MAE | Com- pressed | Un- stable | Transfer / fail / indet. |
|---|---|---|---|---|---|---|---|---|
| Claude Sonnet 5 | 0.001 | 0.156 | 0.458 | 0.085 | 0.263 | 8 | 5 | 3 / 3 / 4 |
| Gemini 3.5 Flash | 0.150 | 0.253 | 0.445 | 0.088 | 0.244 | 8 | 5 | 3 / 4 / 3 |
| DeepSeek V4-Pro | 0.230 | 0.280 | 0.690 | 0.101 | 0.369 | 11 | 4 | 1 / 3 / 7 |
| Claude Sonnet 4.6 · human · numeric | 0.333 | 0.411 | 0.534 | 0.077 | 0.309 | 9 | 6 | n/e |
| Claude Sonnet 4.6 · human · lexical | 0.333 | 0.411 | 0.612 | 0.072 | 0.324 | 10 | 6 | n/e |
| Claude Sonnet 4.6 · synthetic · numeric | 0.333 | 0.411 | 0.555 | 0.084 | 0.319 | 10 | 7 | n/e |
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
| Item | Seen before (%) | Seen (people) | Lure, all (%) | Lure, unseen only (%) | Lure, seen only (%) | Unseen all (pp) |
|---|---|---|---|---|---|---|
| CRT2 | ||||||
| CRT2 q1 | 74.5 | 351 | 18.7 | 33.3 | 13.7 | +14.6 |
| CRT2 q2 | 58.6 | 276 | 9.8 | 15.9 | 5.4 | +6.1 |
| CRT2 q3 | 74.3 | 350 | 15.3 | 21.5 | 13.1 | +6.2 |
| CRT2 q4 | 40.8 | 192 | 47.6 | 53.4 | 39.1 | +5.8 |
| CRT2, all items | 40.8–74.5 | – | 22.8 [20.8, 25.1] | 34.4 [30.6, 38.3] | 15.7 [13.4, 18.2] | +11.6 |
| Item | Raw correct | Raw lure | Raw other | Adjusted correct | Adjusted lure | Adjusted other |
|---|---|---|---|---|---|---|
| CRT2 (familiarity-adjusted reference used for every CRT2 comparison) | ||||||
| CRT2 q1 | 80.5 | 18.7 | 0.8 | 63.8 | 35.4 | 0.8 |
| CRT2 q2 | 89.8 | 9.8 | 0.4 | 79.3 | 19.8 | 0.9 |
| CRT2 q3 | 84.3 | 15.3 | 0.4 | 76.2 | 22.9 | 0.9 |
| CRT2 q4 | 33.8 | 47.6 | 18.7 | 21.8 | 55.2 | 23.0 |
| all four items | 72.1 | 22.8 | 5.1 | 60.3 | 33.3 | 6.4 |
| Condition | Persona source | Format | Model | Configuration | N |
|---|---|---|---|---|---|
| Claude Sonnet 5 | Synthetic profiles | Numeric | Claude Sonnet 5 (claude-sonnet-5) | provider defaults; adaptive thinking on | 471 |
| Gemini 3.5 Flash | Synthetic profiles | Numeric | Gemini 3.5 Flash (gemini-3.5-flash) | thinking level minimal | 471 |
| DeepSeek V4-Pro | Synthetic profiles | Numeric | DeepSeek V4-Pro (via Together AI) | reasoning effort high | 471 / 467 * |
| Claude Sonnet 4.6 · human · numeric | Human-derived profiles | Numeric | Claude Sonnet 4.6 (claude-sonnet-4-6) | provider defaults | 471 |
| Claude Sonnet 4.6 · human · lexical | Human-derived profiles | Lexical | Claude Sonnet 4.6 (claude-sonnet-4-6) | provider defaults | 471 |
| Claude Sonnet 4.6 · synthetic · numeric | Synthetic profiles | Numeric | Claude Sonnet 4.6 (claude-sonnet-4-6) | provider defaults | 471 |
| Model | Configuration | CRT2 lure % [95% CI] | bCRT lure % [95% CI] | Pooled lure % [95% CI] |
| Anthropic | ||||
| Claude Sonnet 5 | adaptive thinking (default) | 20.8 [15.8, 26.3] | 13.8 [10.0, 17.8] | 17.3 [13.2, 21.8] |
| Claude Sonnet 5 | thinking disabled | 4.5 [3.0, 6.0] | 0.2 [0.0, 0.5] | 2.3 [1.6, 3.2] |
| Claude Fable 5 | provider defaults | 32.0 [25.3, 39.0] | 40.3 [34.7, 45.8] | 36.2 [30.4, 42.1] |
| Gemini 3.5 Flash | thinking level minimal | 23.0 [16.8, 29.7] | 19.7 [14.8, 25.0] | 21.3 [15.9, 27.2] |
| Cell | N | CRT2 lure % | bCRT lure % | Pooled lure % [95% CI] | Check |
| Temperature sweep | |||||
| temperature 0.0 | 150 | 0.0 | 0.0 | 0.0 [0.0, 0.0] | NFC variance 103% of human |
| temperature 0.3 | 150 | 0.0 | 0.0 | 0.0 [0.0, 0.0] | NFC variance 103% of human |
| temperature 0.7 | 150 | 0.0 | 0.0 | 0.0 [0.0, 0.0] | NFC variance 107% of human |
| temperature 1.0 | 150 | 0.0 | 0.0 | 0.0 [0.0, 0.0] | NFC variance 105% of human |
| temperature provider default (1.0) | 150 | 0.0 | 0.0 | 0.0 [0.0, 0.0] | NFC variance 104% of human |
| Condition | Run identifier |
|---|---|
| Claude Sonnet 5 | CR_MAIN_SYNTH_NUM_SONNET5_S42 |
| Gemini 3.5 Flash | CR_MAIN_SYNTH_NUM_GEM35F_MIN_S42 |
| DeepSeek V4-Pro | CR_MAIN_SYNTH_NUM_DSV4PRO_TOG_S42 |
| Claude Sonnet 4.6 · human · numeric | CR_BASELINE_HUMAN_NUMERIC_CS46_S42 |
| Claude Sonnet 4.6 · human · lexical | CR_BASELINE_HUMAN_LEXICAL_CS46_S42 |
| Claude Sonnet 4.6 · synthetic · numeric | CR_MAIN_SYNTH_NUMERIC_CS46_S42 |
| Claude Sonnet 4.6 | |||||||||||||||
| Behaviour | People % [95% CI] | Claude Sonnet 5 | Gemini 3.5 Flash | DeepSeek V4-Pro | human numeric | human lexical | synthetic numeric | synthetic lexical | |||||||
| Sources the respondent plans to consult | |||||||||||||||
| Phones the restaurant (the primary source) | 11.4 [8.9, 14.6] | 21.2 | 17.6 | 89.1 | C | 7.4 | 13.8 | 8.9 | 10.8 | ||||||
| Consults regulators / journals / official guidance | 30.0 [26.1, 34.3] | 70.1 | C | 76.0 | C | 91.6 | C | 78.3 | C | 79.8 | C | 80.9 | C | 81.1 | C |
| Asks friends or family (medication) | 29.2 [25.3, 33.4] | 30.4 | 27.2 | 86.1 | C | 51.0 | C | 56.9 | C | 55.0 | C | 59.7 | C | ||
| Consults hygiene / official records | 14.4 [11.5, 17.8] | 89.8 | C | 46.1 | C | 71.9 | C | 89.2 | C | 90.7 | C | 89.0 | C | 91.7 | C |
| Claude Sonnet 5 | Gemini 3.5 Flash | DeepSeek V4-Pro | |||||
| Behaviour | People OR [95% CI] | OR [95% CI] | V | OR [95% CI] | V | OR [95% CI] | V |
| Sources the respondent plans to consult | |||||||
| Phones the restaurant (the primary source) | 1.52 [1.21, 1.91] | 1.16 [1.01, 1.35] | T | 1.32 [1.04, 1.67] | T | 1.06 [0.86, 1.31] | F |
| Consults regulators / journals / official guidance | 1.33 [1.13, 1.57] | 1.40 [1.23, 1.59] | T | 1.54 [1.31, 1.80] | T | 1.23 [1.00, 1.51] | ? |
| Asks friends or family (medication) | 1.24 [1.05, 1.47] | 0.91 [0.80, 1.02] | F | 0.85 [0.72, 0.99] | F | 0.93 [0.75, 1.16] | F |
| Consults hygiene / official records | 1.28 [1.04, 1.57] | 1.24 [1.04, 1.49] | T | 1.39 [1.18, 1.63] | T | 1.34 [1.16, 1.55] | T |
| Pre- dictor | LIWC outcome | Human slope | Agent slope | Differ- ence | 95% CI | Binned RMSE | |
|---|---|---|---|---|---|---|---|
| CRT2 | Word count | 19.634 | 4.203 | 15.431 | [ 24.047, 6.815] | .001 | 160.80 |
| CRT2 | Auxiliary verbs | 0.174 | 0.100 | 0.074 | [ 0.111, 0.260] | .432 | 3.82 |
| CRT2 | Differentiation | 0.049 | 0.146 | 0.195 | [0.046, 0.344] | .010 | 0.99 |
| CRT2 | Negative emotion | 0.010 | 0.003 | 0.013 | [ 0.053, 0.079] | .700 | 0.71 |
| CRT2 | Present focus | 0.140 | 0.081 | 0.221 | [ 0.388, 0.055] | .009 | 1.43 |
| CRT2 | Moral language | 0.028 | 0.012 | 0.016 | [ 0.055, 0.023] | .427 | 0.23 |
| Pre- dictor | LIWC outcome | Human slope | Agent slope | Differ- ence | 95% CI | Binned RMSE | |
|---|---|---|---|---|---|---|---|
| CRT2 | Word count | 19.634 | 5.641 | 13.994 | [ 23.167, 4.821] | .003 | 194.78 |
| CRT2 | Auxiliary verbs | 0.174 | 0.096 | 0.078 | [ 0.103, 0.259] | .400 | 4.12 |
| CRT2 | Differentiation | 0.049 | 0.141 | 0.189 | [0.047, 0.332] | .009 | 1.15 |
| CRT2 | Negative emotion | 0.010 | 0.085 | 0.075 | [ 0.154, 0.005] | .067 | 1.01 |
| CRT2 | Present focus | 0.140 | 0.259 | 0.398 | [ 0.569, 0.228] | .001 | 2.09 |
| CRT2 | Moral language | 0.028 | 0.020 | 0.007 | [ 0.042, 0.028] | .676 | 0.34 |
| Pre- dictor | LIWC outcome | Human slope | Agent slope | Differ- ence | 95% CI | Binned RMSE | |
|---|---|---|---|---|---|---|---|
| CRT2 | Word count | 19.634 | 0.045 | 19.589 | [ 33.930, 5.248] | .007 | 378.91 |
| CRT2 | Auxiliary verbs | 0.174 | 0.031 | 0.143 | [ 0.078, 0.364] | .206 | 2.09 |
| CRT2 | Differentiation | 0.049 | 0.011 | 0.060 | [ 0.106, 0.227] | .477 | 0.35 |
| CRT2 | Negative emotion | 0.010 | 0.006 | 0.005 | [ 0.076, 0.085] | .912 | 0.68 |
| CRT2 | Present focus | 0.140 | 0.113 | 0.253 | [ 0.448, 0.058] | .011 | 1.45 |
| CRT2 | Moral language | 0.028 | 0.010 | 0.037 | [ 0.077, 0.002] | .061 | 0.20 |
LLM Agent Evaluation
Large language model agent evaluation measures whether tool-using systems complete tasks reliably, safely, and efficiently.
See all 1,297 papers