cs.AINov 12, 2026

Large Language Models as Digital Twins: a Distribution-Aware Questionnaire-Based Evaluation

Authors: Francesco Di Cursi, Chiara Boldrini, Andrea Passarella, Marco Conti

Organizations: Institute of Informatics and Telematics (IIT), National Research Council (CNR), Pisa, Italy · Department of Computer Science, University of Pisa, Pisa, Italy

Abstract

This paper investigates whether Large Language Models can answer a personality questionnaire as a specific individual would. Given a user's social-media posts, the model completes the BFI-2-60 Big Five questionnaire as that user. The predicted responses are converted into trait scores and compared with the user's actual scores. We evaluate GPT-4, Gemma, Llama 3.1, and Mistral on the MyPersonality dataset. To determine whether predictions reflect information from the user's writing, we compare them with answers produced without user text and with values derived from general population tendencies. We measure prediction error across all users and separately for individuals with low, intermediate, and high trait scores. We also test whether an LLM can express an explicit personality profile when given only the target Big Five scores, without user text, and examine how predictions change as more text is provided. The results show that user text does not consistently improve performance. Models perform better near the middle of the trait distribution, while at the lower and upper ends they often remain close to central values or perform well at one end but poorly at the other. When given an explicit profile, GPT-4 most clearly translates trait information into consistent questionnaire responses, particularly for Extraversion and Neuroticism. Increasing context does not systematically reduce error: open-source models fluctuate more, whereas GPT-4 remains comparatively stable. Overall, the evaluated models do not consistently reproduce the questionnaire behavior of specific individuals.

Figures & tables

Explore similar work

CardsList
  1. Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?

    May 16, 2026Prateek Rajput, Yewei Song, Iyiola E. Olatunji +2Personality Modeling in Language ModelsLLM Post-Training

  2. Human Psychometric Questionnaires Mischaracterize LLM Behavior

    Sep 12, 2025Woojung Song, Dongmin Choi, Yoonah Park +3Personality Modeling in Language ModelsPsychometric Validation

  3. All four leading LLMs talk more than they listen to personality-verified synthetic help-seekers

    Aug 23, 2026Pablo A. Fonseca, Raquel Rodríguez-Carvajal, Rafael A. CalvoEmotional Support ConversationMulti-Turn Dialogue Evaluation