What is known about the values of national populations rests almost entirely on questionnaires, which prompt respondents to rate researcher-supplied descriptions of each motivation. Social media instead records values as they are invoked spontaneously, in communication addressed to a real audience. We measure value expression in broadly sampled, heterogeneous Russian-language social-media data, where value-relevant content is sparse and often ambiguous, and read the result against the survey-based profile of the same population. To address subjectivity in value interpretation, we compare alternative LLM-prompt configurations against expert judgments and use error-driven calibration to select a theory-based operationalization. Using VKontakte data collected from one million randomly generated user IDs, comprising 7.5 million public text posts, we develop a multi-stage framework combining candidate selection, repeated LLM annotation, and transfer to an XLM-RoBERTa encoder for large-scale prediction (a macro PR-AUC of .60 on the expert benchmark, with continuous scores positively correlated with graded expert confidence across all ten values). The circular structure proposed by Schwartz is largely recovered (congruence .95 at the post level, .94 at the user level). The aggregate profile agrees with survey findings at the anchors - Benevolence highest and broadly distributed across users, Power lowest - but shows relatively elevated Hedonism and relatively low Universalism. Universalism is further displaced structurally, co-expressed with Security and Tradition and concentrated in the output of highly active users.
Figures & tables
Figure 1: Distribution of post-weighted value-expression scores
Figure 2: Post-weighted and user-weighted aggregate value-expression profiles.
Figure 3: Heatmap of correlations among values.
Figure 4: Multidimensional scaling of post-weighted value expressions. Black points represent the ideal theoretical structure in line with Schwartz theory.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Original Toloka annotation interface used for selection of value-expressive posts. The interface was presented in Russian. The annotation question translates as “Does this text reflect any value?”, with response options Yes , No , Unclear , and Spam .
Figure 6: Response of GPT-4, GPT-5.2, and Gemini 2.5 Pro to prompt refinement on main expert benchmark ( N=1,000 ). Upper panels show Precision, Recall, and F1; lower panels show over-attribution rates. base : Baseline; ext : Extended; ext_ctx : Contextualized; calib. : Bias-calibrated; calib_RU : Bias-calibrated Russian-language prompt.
Value
P
R
F1
OA
UA
ρ
Benchmark 1
Self-direction
.53
.70
.61
.08
.30
.63
Stimulation
.48
.54
.51
.04
.46
.54
Hedonism
.77
.49
.60
.04
.51
.60
Achievement
.72
.70
.71
.04
.30
.68
Power
.43
.71
.53
.03
.29
.52
Appendix
Table 1: Per-value performance and graded alignment of Gemini 2.5 Pro (with the English bias-calibrated prompt) on the two expert benchmarks. P = Precision; R = Recall; OA = over-attribution rate; UA = under-attribution rate; ρ = Spearman correlation between expert positive-vote counts (0–3) and model positive-vote counts (0–5). All correlations are significant at p<.001 .
Parameter
Configuration
Model
XLM-RoBERTa-large
Fine-tuned layers
Last two layers
Pooling
Mean pooling
Classification head
Linear multi-label head
Loss
Binary cross-entropy (BCE)
Batch size
16
Appendix
Table 2: Training configuration for the XLM-RoBERTa-large encoder.
Expert benchmark
Test
Value
Experts
Gemini
Gemini
Self-direction
.59
.74
.75
Stimulation
.44
.65
.63
Hedonism
.68
.67
.73
Achievement
.73
.81
.79
Power
.47
.52
.47
Appendix
Table 3: Precision–Recall AUC (Average Precision) of the XLM-RoBERTa-large encoder on the expert benchmark ( N=1,000 ) and held-out test data ( N=4,000 ). Columns indicate the reference annotations used for evaluation.
Measuring subjective constructs in naturally occurring social media text requires annotation procedures that are theoretically grounded, empirically validated, and transferable to an encoder model for scalable prediction. Using non-English social media posts annotated according to Schwartz's theory of basic human values, we investigate how different LLMs, prompts, and instruction languages operationalize the expression of values in text. We argue that although texts may permit multiple plausible interpretations, theory-based value definitions can constrain interpretations and reduce spurious value attributions. Beyond precision, recall, and F1, we evaluate structural alignment between values, error structure, confidence-ambiguity relations, and annotation stability. We show that different LLMs produce different value interpretations. Iterative prompt calibration through error analysis reduces misattributions and improves alignment with expert annotations. We also derive targeted expert verification rules from recurrent error structures and use them during corpus annotation. Finally, we show that LLM annotations can be transferred to an encoder model through soft-label training, retaining theory-based value interpretations and information about uncertainty in value expression.
Maria Milkova, Maksim Rudnev
Independent researcher, Lisbon, Portugal · University of Waterloo, ON, Canada
Large language models are increasingly evaluated through the values they endorse, but such evaluations presuppose that models can identify the value expressed in a concrete situation. We study this prerequisite as controlled top-1 recognition over Schwartz's ten basic values. Our evaluation set contains 1,000 Russian situational texts, balanced across the ten values and independently labeled by two human annotators per item. We evaluate 21 instruction-tuned LLM runs under a fixed ranked-response protocol; 20 runs with reliable outputs form the semantic panel. Pooled Acc@1 is 0.683 and Acc@3 is 0.892, showing that models often locate the correct motivational region while ranking close alternatives unstably. Adjacent values account for 50.9% of semantic errors, compared with 24.4% under a checkpoint-specific null. Eight directed confusions recur across checkpoints and human-confirmed subsets. Several are strongly asymmetric, including Universalism to Benevolence, Tradition to Conformity, and Security to Power, whereas Stimulation-Hedonism forms a bidirectional boundary. Their severity is checkpoint-specific and can bias higher-order value profiles. The results motivate value-recognition evaluation that combines exact accuracy, ranked recovery, and directed error analysis.
Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha +5
Ivannikov Institute for System Programming of the Russian Academy of Sciences, Moscow, Russia · Russian Presidential Academy of National Economy and Public Administration, Moscow, Russia
Large Language Model (LLM)-based agents are increasingly used as proxies for human participants in social science research, yet it remains unclear whether they can faithfully simulate diverse and conflicting human value systems. We present a World Values Survey (WVS)-grounded simulation framework where culturally diverse agents with different communication styles engage in longitudinal, value-laden discussions. Across approximately 4,000 conversations involving 1,200 personas, 15 topics, and three models (GPT-4o, Gemini-2.5-Flash, and Gemma-4-E4B), we evaluate value faithfulness, value drift, and conversational realism. We find that more than 50% of personas fail to express their assigned WVS profiles from the outset, while 2-7% drift after repeated conversations. Ablations removing demographic details improve faithfulness for some models but do not change the broader trend: simulated value distributions still systematically deviate from the assigned WVS profiles. Compared to human discussions, simulated dialogues show a different trade-off between stylistic consistency and semantic diversity, often producing content-wise varied but stylistically repetitive exchanges. These findings suggest that current LLM agents can generate plausible conversations, but remain limited proxies for representing and preserving diverse human value profiles over time.
Farah Atif, Sougata Saha, Monojit Choudhury
Mohamed Bin Zayed University of Artificial Intelligence