cs.AINov 6, 2026

Tracing Psychometric Inference in Large Language Models

Authors: Chih-Hao Hsu, Feng-Chun B. Chou, Pin-Hao A. Chen

Organizations: National Taiwan University

Abstract

Do large language models represent psychological constructs internally, or do they only generate outputs that resemble psychometric structure? We address this question using a cross-persona paradigm across 14 model variants from two families (Llama 3 and Qwen 2.5, 0.5B-14B, base and instruct). Given responses on one psychological scale from real individuals (N = 272 Taiwanese university students), models predict responses on six other scales. At the behavioral level, larger variants reproduce and amplify human cross-scale correlations; amplification persists after correcting the seven-scale human correlations for measurement attenuation. We then examine whether this structure is reflected internally. Contrastive direction analysis finds geometry aligned with the human matrix in larger instruct variants, but little alignment in base variants under this assay. Across models, geometry strength is associated with behavioral amplification (r = 0.68, p = 0.008; partial r = 0.69 controlling for log size). To relate internal structure to output behavior, a matched activation-probing paradigm shows that probe slopes vary less than behavioral slopes (1.38-1.77 vs 0.52-1.53). A synthetic control shows that ridge probing can inflate these slopes; after subtracting a baseline based on a reliability proxy from another analysis, adjusted slopes remain associated with behavior (r = 0.88). We term the measured probe-behavior gap readout attenuation. These results establish decodability and geometric alignment under the tested assays, without showing that the measured directions mediate the cross-persona answers.

Figures & tables

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling

    Sep 19, 2026Yu Sha, Junqi Tao, Dixin Zhou +10Personality Modeling in Language ModelsMultilingual Language Model Evaluation

  2. Apparent Psychological Profiles of Large Language Models are Largely a Measurement Artifact

    Jun 18, 2026Jelena Meyer, David Garcia, Dirk U. WulffLLM EvaluationPersonality Modeling in Language Models

  3. Is your LLM an Agressive Narcissist with Depression? Analyzing Psychological Portraits of Large Language Models

    Nov 2, 2026Polina Fedorinina, Anna Fattakhova, Nadezhda Romanova +2Personality Modeling in Language ModelsBehavioral Consistency of LLM Agents