Probable but Not Valid: A Psychometric Audit of Decision Models as Synthetic Survey Respondents

Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation, reasoning-effort and cross-language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data.

How well can AI-derived synthetic research data replicate the responses of human participants? An emerging literature has begun to engage with this question, which carries deep implications for organizational research practice. This article presents a comparison between a human-respondent survey of 420 Silicon Valley coders and developers and synthetic survey data designed to simulate real survey takers generated by five leading Generative AI Large Language Models: ChatGPT Thinking 5 Pro, Claude Sonnet 4.5 Pro plus Claude CoWork 1.123, Gemini Advanced 2.5 Pro, Incredible 1.0, and DeepSeek 3.2. Our findings reveal that while AI agents produced technically plausible results that lean more towards replicability and harmonization than assumed, none were able to capture the counterintuitive insights that made the human survey valuable. Moreover, deviations grouped together for all models, leaving the real data as the outlier. Our key finding is that while leading LLMs are increasingly being used to scale, replicate and replace human survey responses in research, these advances only show an increased capacity to parrot conventional wisdom in harmony with each other rather than revealing novel findings. If synthetic respondents are used in future research, we need more replicable validation protocols and reporting standards for when and where synthetic survey data can be used responsibly, a gap that this paper fills. Our results suggest that synthetic survey responses cannot meaningfully model real human social beliefs within organizations, particularly in contexts lacking previously documented evidence. We conclude that synthetic survey-based research should be cast not as a substitute for rigorous survey methods, but as an increasingly reliable pre- or post-fieldwork instrument for identifying societal assumptions, conventional wisdoms, and other expectations about research populations.

Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems. A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data: U.S. general social attitudes (General Social Survey) and cross-cultural values (World Values Survey). Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data. Under demographic prompting and the survey-simulation protocols we test, two failures replicate across both domains, all four models, and both families. First, at the individual level no LLM beats even the strongest baseline; on cross-cultural values every model falls well below it, and the gap survives distance-aware and proper scoring. Second, models systematically over-determine demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question-group combination and robust to a coding-invariant measure. Neither failure is remedied by a larger, more capable model. A decision-impact analysis shows why this matters in practice: on a segment-targeting task the models inflate between-segment gaps two to fourfold, would direct a team to the wrong segment in half of U.S. and most cross-cultural cases, and manufacture segment splits that do not exist in real people. We publicly release the cross-domain benchmark and validation framework, including all code and data, so that teams can determine in advance when synthetic-user evidence is safe for decision support and when it is not.