STONIC: A Layered Measurement Contract for LLM Value Profiling

Andrei Chetvergov; Stepan Ukolov; Timofei Sivoraksha; Alexander Evseev; Danil Sazanakov; Mikhail Solovev; Sergey Bolovtsov

cs.CLAug 24, 2026

STONIC: A Layered Measurement Contract for LLM Value Profiling

Authors: Andrei Chetvergov, Stepan Ukolov, Timofei Sivoraksha, Alexander Evseev, Danil Sazanakov, Mikhail Solovev, Sergey Bolovtsov

Abstract

LLM value studies often merge questionnaire ratings, pairwise choices, and values inferred from generated text into one profile. That merge assumes that the three observations describe the same stable preference. STONIC tests this assumption on 5,144 situations from four banks and 35 fixed model configurations. It compares responses rated in isolation, choices made under counterbalanced conflict, spontaneous answers, and later choices between a model's own answer and authored alternatives. 10 of 17 configurations with usable behavioral data preserve the endorsement-choice relation across banks. Every one of the 17 eligible configurations prefers its own earlier answer (median effect 0.790), although option position changes the choice rate in every eligible configuration. Profile shape transfers most strongly from ratings to conflict choices and weakens for spontaneous text. Three-way annotation of 200 L3 responses provides a task-local check of the semantic audit: FULCRA agrees most closely with the human majority, while DeBERTa retains useful rank information after calibration. Hidden states encode the completed decision more clearly than the prompt alone. Thus the models show reproducible behavioral continuity, but the evidence does not support one scorer-independent value identity across interfaces.

Figures & tables

Figure 1: One ValuePortrait situation across three interfaces. GLM-4.7 rates response 3 highest, chooses response 1 after order reversal, then writes about safety and control; each result supports a different claim.

Figure 2: Stonic measurement design. 35 configurations receive 5,144 situations from four banks under matched L1 endorsement, L2 counterbalanced choice, and L3 free-response interfaces. L2 ∗ tests self-answer preference; five frozen semantic views and same-pass hidden states support scorer and representation audits. The L0/PVQ-40 anchor is a separate bank, while value discernment is a precondition check. Outputs remain interface-specific and are not pooled.

Relation	Estimable	Supported	Result
Rating predicts conflict choice	17	10	median $+.228$
Same profile: R–C / R–F / C–F	17/18/23	0/0/0	median $\rho=.73/.43/.50$
Own answer preferred later	24	17	median $+.790$
Option order changes choice	23	18	significant in 18/18

Table 1: Main cross-interface findings. “Supported” counts configurations that are estimable and meet the coverage, multiplicity, direction, and cross-bank requirements. R, C, and F denote independent rating, conflict choice, and free answer.

Figure 3: Profile transmission and behavioral continuity. Each row is one of the 17 configurations with complete inputs for the exploratory four-bank comparison. Left: similarity of the model’s Schwartz-10 ordering between independent ratings, counterbalanced conflict choices, and spontaneous answers. Median rank correlations are

0.73

for rating–choice,

0.43

for rating–free answer, and

0.50

for choice–free answer. Right: the four-bank profile-coherence index and the separately registered own-answer preference, whose mapped median is

90.4/100

. Filled diamonds meet the prespecified own-answer reporting criteria; open diamonds are diagnostic estimates. Models often prefer their own answer even when its inferred value profile has drifted. This exploratory ranking measures consistency, not moral quality or alignment.

View	Signal	NZ	Multi	$\rho$ /top
GPV $\to$ ValueLlama	signed	53.1%	26.4%	.311/.313
GPV $\to$ DeBERTa19	presence	96.3%	0.3%	.842/.588
DeBERTa sentence	presence	97.1%	1.1%	reference
DeBERTa whole	presence	97.1%	12.6%	.976/.822
FULCRA epoch-10	signed	97.1%	95.7%	.456/.321

Table 2: Five frozen views of the same L3 answers. NZ is the percentage with a nonzero profile; Multi is the percentage with multiple active values. Agreement reports row-level Spearman

\rho

and top-value match against direct sentence DeBERTa. These are agreement diagnostics, not accuracy estimates.

Appendix figures & tables22 assets

Supplementary material from the paper’s appendix.

Appendix

#	Cell ID	Checkpoint	Mode
1	avibe__i	AvitoTech/avibe	Instruct
2	gemma_2_9b__b	google/gemma-2-9b	Base
3	gemma_2_9b__i	google/gemma-2-9b-it	Instruct
4	gemma_4_26b_a4b__b	google/gemma-4-26B-A4B	Base
5	gemma_4_26b_a4b__i	google/gemma-4-26B-A4B-it	Instruct
6	gigachat_3_1__b	ai-sage/GigaChat3-10B-A1.8B-base	Base

Appendix

Table 4: Complete model configuration panel.

Task	What happened	What the result lets us say
L1	Response 3 receives the highest mean endorsement (4.50/6): “Focus on personal growth and building a support network…”	This response is rated most favorably when shown alone.
L2	Response 1 wins in both orders: A in the original order and B after reversal.	The preference survives this presentation-order check.
L3	“Consulting with a divorce attorney… creating a plan and gathering resources can help you regain control.”	The free answer centers safety, rights, planning, and control; any Schwartz label still belongs to the named scorer.

Appendix

Table 5: Worked trace for ValuePortrait item vp_2902 . Quoted text is the recorded model output or authored alternative.

Observed result	Supported statement	Unsupported shortcut
High L1 rating	The model endorsed that response in isolation.	The model will choose it under conflict.
Positive, order-stable C1	Higher L1 endorsement predicted the tested L2 choices.	L1 and L2 express the same semantic value.
An L3 scorer returns Security	That scorer found Security evidence in the generated text.	Security is an intrinsic, scorer-free model trait.
Post-output hidden probe succeeds	The completed answer is decodable from that stored state.	The state caused the decision or encoded it before generation.

Appendix

Table 6: Claim boundaries in plain language. Passing a row does not automatically license the statement in the last column.


Clean family	Parse cov.	C1	C2 L1–L3	C5/L2*	E6 L3
gemma_2_9b	+0.6663	–	–	–	+0.1055
gemma_4_26b_a4b	+0.6314	–	–	–	+0.0533
granite_3_3_8b	+0.3678	–	–	–	+0.0717
granite_4_1_8b	+0.5829	–	–	–	-0.0071
llama_3_8b	+0.6045	–	–	–	+0.1024

Appendix

Table 9: Matched instruction-minus-base descriptive contrasts.


Outcome	n family	Median I-B	Cluster 95% CI	Holm p	Decision
c1_l1_l2_behavioral_alignment	0	–	–	–	not significant / not estimable
c2_l1_l3_semantic_transfer	0	–	–	–	not significant / not estimable
e6_prompt end_l3_cross_bank_transfer	12	+0.0695	[0.0270, 0.1012]	0.0781	not significant / not estimable
e9_l2_star_own_coefficient	3	+0.1920	[0.1592, 0.2363]	1	not significant / not estimable
strict_parse_coverage	13	+0.3486	[0.3478, 0.3522]	0.0781	not significant / not estimable

Appendix

Table 10: Multiplicity-corrected C4 family inference.


Target	Position	Mode	n	Metric	Median test	Median depth
L1	prompt end	base	1	spearman	0.4176	0.7179
L1	prompt end	instruct	18	spearman	0.8129	0.9426
L1	decision mean	base	1	spearman	0.6550	0.0000
L1	decision mean	instruct	18	spearman	0.8737	0.0000
L2	prompt end	base	7	balanced acc.	0.6252	0.7436

Appendix

Table 11: Hidden-state test metrics by interface, position, and mode.


Target	n paired	Median $\Delta$ post-pre	Mean	Min	Max
L1	19	+0.0515	+0.0769	+0.0098	+0.2375
L2	24	+0.0573	+0.0687	-0.0044	+0.2140
L3	34	+0.0667	+0.0687	+0.0222	+0.1247
L2_star	22	+0.1121	+0.1393	-0.0119	+0.3789

Appendix

Table 12: Paired post-output minus prompt-boundary hidden-state results.


Target	Position	n cells	Median excess	Mean	Range
L1	decision mean	16	+0.7729	+0.7698	[0.6036, 0.9326]
L1	prompt end	16	+0.6651	+0.6480	[0.2577, 0.7908]
L2	decision mean	22	-0.1209	-0.1204	[-0.1733, -0.0688]
L2	prompt end	22	-0.0792	-0.0740	[-0.1152, -0.0100]
L3	decision mean	34	+0.2634	+0.2488	[0.1332, 0.3244]

Appendix

Table 13: Cross-bank probe transfer relative to the permutation baseline.


Direction	Target	n family	Median excess	Mean	Range
base_to_instruct	L1	1	+0.1162	+0.1162	[0.1162, 0.1162]
base_to_instruct	L2	5	-0.0015	-0.0066	[-0.0295, 0.0000]
base_to_instruct	L3	12	+0.1366	+0.1294	[0.0434, 0.2036]
instruct_to_base	L1	1	-0.0682	-0.0682	[-0.0682, -0.0682]
instruct_to_base	L2	5	+0.0154	+0.0137	[0.0000, 0.0292]

Appendix

Table 14: Within-family base/instruction probe transfer.


Value	L0 score/rank (n=14)	L1 score/rank (n=18)	L2 score/rank (n=23)	L3 score/rank (n=34)
Benevolence	+0.7018 / 3	+0.0201 / 2	+0.0267 / 4	+0.1507 / 1
Security	-0.1125 / 5	+0.0089 / 3	+0.0385 / 3	+0.1475 / 2
Conformity	-0.6375 / 9	+0.0239 / 1	+0.0676 / 2	+0.1351 / 3
Achievement	-0.3732 / 7	-0.0199 / 10	-0.0593 / 9	+0.1336 / 4
Self-Direction	+0.8375 / 1	+0.0079 / 5	+0.0796 / 1	+0.0902 / 5

Appendix

Table 15: Schwartz-10 profiles by interface.


L3 rank	Value	Signed mean	Detection	Conditional valence
1	Benevolence	+0.1507	0.1612	+0.8854
2	Security	+0.1475	0.1668	+0.8379
3	Conformity	+0.1351	0.1697	+0.8113
4	Achievement	+0.1336	0.2062	+0.6602
5	Self-Direction	+0.0902	0.2849	+0.2975

Appendix

Table 16: L3 signed means, detection, and conditional valence.


Profiles	n	Median Spearman $\rho$	$\rho$ > 0	Range
L1–L2	17	0.7333	14/17	[-0.3818, 0.9515]
L1–L3	18	0.4364	14/18	[-0.4788, 0.6606]
L2–L3	23	0.5030	23/23	[0.1273, 0.6970]
L0–L3	14	0.0367	8/14	[-0.1844, 0.4634]

Appendix

Table 17: Within-model rank correlations between interfaces.


Bank	Leading value	Second value
ValuePortrait	Benevolence (+0.2348)	Achievement (+0.2162)
AIRiskDilemmas	Security (+0.1435)	Achievement (+0.0718)
DailyDilemmas	Conformity (+0.1622)	Benevolence (+0.1603)
MoralChoice	Security (+0.2026)	Conformity (+0.1762)

Appendix

Table 18: Leading L3 values by bank.


Value	Median Instruct-Base	Instruct > Base
Benevolence	-0.0020	4/13
Security	-0.0216	6/13
Conformity	-0.0004	6/13
Achievement	-0.0252	4/13
Self-Direction	-0.0067	6/13

Appendix

Table 19: Matched instruction-minus-base L3 value deltas.


Scorer view	Top-3	Bottom-2
GPV $\rightarrow$ ValueLlama (signed)	Benevolence, Security, Conformity	Universalism, Power
DeBERTa19 sentence (presence)	Security, Conformity, Benevolence	Achievement, Tradition
DeBERTa19 whole (presence)	Security, Conformity, Benevolence	Tradition, Achievement
FULCRA epoch-10 (regression)	Benevolence, Security, Self-Direction	Stimulation, Tradition
GPV $\rightarrow$ DeBERTa19 (presence)	Universalism, Security, Conformity	Power, Achievement

Appendix

Table 20: Leading and trailing values by scorer view.

Figure 4: Independent-endorsement terrains for the fixed instruction panel. Each panel uses only that model’s rated alternatives: ratings center the alternatives within an item, weight their signed Schwartz-10 vectors, and are resampled within bank. The nested contours show bootstrap density and the ring marks the equal-bank core. Colors identify models consistently across all three figures.

Figure 5: Counterbalanced conflict-choice terrains for the same instruction models, panel order, colors, and coordinate scale as Figure 4 . A chosen alternative contributes the signed difference between the pair’s value vectors; order-discordant pairs do not create a stable item profile. The NE panels make incomplete four-bank evidence visible.

Figure 6: Spontaneous-answer terrains. Every panel is generated from the signed scorer vectors of that model’s own free responses, resampled within bank and combined with equal bank weight. Exact-zero scorer outputs remain missing evidence. The fixed small-multiple design exposes model-specific displacement and uncertainty without constructing a pooled cross-model landscape.


Comparison	Median profile $\rho$	Top-dim match	L1–L3 sign	L1–L3 $\rho$	L2–L3 sign	L2–L3 $\rho$
gpv_valuellama $\leftrightarrow$ gpv_deberta19	0.3939	0.1765	0.9444	0.4530	1.0000	0.4595
gpv_deberta19 $\leftrightarrow$ direct_deberta19_sentence	0.8606	0.1765	1.0000	0.9154	1.0000	0.6660
direct_deberta19_sentence $\leftrightarrow$ direct_deberta19_whole	0.9697	0.7059	1.0000	0.9732	1.0000	0.9427
gpv_valuellama $\leftrightarrow$ fulcra_epoch10	0.7091	0.0882	1.0000	0.4448	1.0000	0.7787

Appendix

Table 22: Scorer agreement for model-level profiles and cross-interface effects.


10 600 authored alternatives	Median equal-bank rank Pearson	Median sign agreement
gpv_deberta19__vs__fulcra_epoch10	0.1307	0.5679
gpv_valuellama__vs__fulcra_epoch10	0.1392	0.6781
gpv_valuellama__vs__gpv_deberta19	0.1092	0.6346

Appendix

Table 23: Agreement between scorers on 10,600 authored alternatives.


Rank	Configuration	Mode	Composite	Behavior	Semantic	L3 cov.	Semantic cov.
1	Qwen3.6 27B Instruct	instruct	53.11	0.824	0.342	0.595	0.482
2	Qwen3.6 35B-A3B Instruct	instruct	51.20	0.791	0.331	0.614	0.480
3	Qwen2.5 Coder 14B Instruct	instruct	42.62	0.714	0.254	0.576	0.375
4	Qwen2.5 14B Instruct	instruct	41.67	0.679	0.256	0.604	0.393
5	GigaChat 3.1 Instruct	instruct	41.61	0.689	0.251	0.571	0.388

Appendix

Table 24: Exploratory four-bank ranking for the 17 estimable configurations.


Configuration	Mode	L1 cov.	L2 pair cov.	L3 cov.	Reason
Gemma 2 9B Base	base	0.000	0.000	1.000	L1–L2, L1–L3, L2–L3 not estimable; generation coverage L1=0.0%, L2-two-valid=0.0%, L3=100.0%
Gemma 4 26B-A4B Base	base	0.000	0.000	1.000	L1–L2, L1–L3, L2–L3 not estimable; generation coverage L1=0.0%, L2-two-valid=0.0%, L3=100.0%
GigaChat 3.1 Base	base	0.000	0.155	1.000	L1–L2, L1–L3 not estimable; generation coverage L1=0.0%, L2-two-valid=15.5%, L3=100.0%
Granite 4.1 8B Base	base	0.000	0.220	0.970	L1–L2, L1–L3, L2–L3 not estimable; generation coverage L1=0.0%, L2-two-valid=22.0%, L3=97.0%
Llama 3 8B Base	base	0.000	0.000	1.000	L1–L2, L1–L3, L2–L3 not estimable; generation coverage L1=0.0%, L2-two-valid=0.0%, L3=100.0%

Appendix

Table 25: Configurations omitted from the exploratory ranking because required inputs are missing.

Language Model Bias Evaluation

Language model bias evaluation tests how models’ outputs, decisions, and representations vary across social groups, cultures, languages, and contexts.

See all 497 papers

Human Psychometric Questionnaires Mischaracterize LLM Behavior
Sep 12, 2025Woojung Song, Dongmin Choi, Yoonah Park +3Personality Modeling in Language Models Psychometric Validation
1Graduate School of Data Science, Seoul National University · Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University
From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs
Aug 15, 2025Shangrui Nie, Florian Mai, David Kaczér +3LLM Alignment Supervised Fine-Tuning
Conversational AI and Social Analytics (CAISA) Lab, University of Bonn, Germany · McMaster University, Canada · Department of Computer Science, University of Sheffield, United Kingdom
Evaluating Alignment of Behavioral Dispositions in LLMs
Feb 11, 2026Amir Taubenfeld, Zorik Gekhman, Lior Nezry +8Human Preference Evaluation LLM Alignment
Google Research · Hebrew University · University of Cambridge

Sep 12, 2025J/K move · Enter open · S save

Human Psychometric Questionnaires Mischaracterize LLM Behavior

Woojung Song, Dongmin Choi, Yoonah Park, Jongwook Han+2

1Graduate School of Data Science, Seoul National University · Department of Communication, Interdisciplinary Program in Artificial Intelligence, Seoul National University

Personality Modeling in Language Models Psychometric Validation Language Model Probing

Open paper AskPDF arXiv

We examine whether human psychometric questionnaires can serve as reliable tools for characterizing and predicting LLM behavior in everyday user interactions. We analyze eight open-source LLMs by comparing their value and personality profiles derived from two different methods: Likert self-reports on established questionnaires (PVQ-40/21 and BFI-44/10) and generation probabilities over value-laden responses to everyday user queries. The two profiles diverge substantially. Within-construct item consistency, often cited as evidence of stable LLM dispositions, disappears in generation probabilities. We find that established questionnaire items contain explicit lexical cues that allow models to recognize the target construct and respond in alignment-consistent, socially desirable ways, whereas realistic user queries contain far less recognizable cues. In addition, demographic persona prompts shift models' responses to human questionnaires in ways consistent with real human patterns, but no such shifts appear in the generation setting, highlighting that human questionnaires overestimate LLMs' ability to faithfully reproduce expected psychological traits when role-playing demographic personas. Overall, our study indicates that questionnaire scores alone should not be treated as evidence of LLMs' response tendencies in realistic user interactions, and supports generation-probability profiling with ecologically valid items as a complementary behavioral measure.

Abstract

Language Model Bias Evaluation

Explore similar work

Human Psychometric Questionnaires Mischaracterize LLM Behavior

From Value Conditioning to Behavioral Shift: Lightweight Value Alignment of LLMs

Evaluating Alignment of Behavioral Dispositions in LLMs