Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses

Zihan Chen; Lei Nico Zheng; Di Zhu

cs.HCOct 22, 2026

Abstract Priorities, Concrete Rewards: How Stated Preferences Relate to What People Reward in AI Responses

Authors: Zihan Chen, Lei Nico Zheng, Di Zhu

Organizations: Stevens Institute of Technology, Hoboken, NJ, USA · University of Massachusetts Boston, Boston, MA, USA

Abstract

AI assistants built on large language models (LLMs) are trained on human feedback and increasingly personalized to users' stated preferences. Yet how closely these preferences reflect what people reward in responses remains unclear. We compare stated and revealed preferences in the public PRISM dataset, in which 1,396 participants reported how much seven response attributes mattered to them and then rated 29,347 responses from 21 LLMs. Among responses to the same prompt, features the survey never asked about, such as response length, formatting and refusal language, were at least as strongly associated with ratings as text proxies of the surveyed attributes. Whether stated priorities matched ratings depended on how attributes were measured: participants who prioritized an attribute favoured responses from models that others perceived as strong on it, but not responses that scored higher on text proxies for it. Written feedback was more telling: participants who asked for shorter answers, even in earlier conversations, gave length less weight in their ratings, and those asking for longer answers gave it more. Their relative priorities over the surveyed attributes, however, did not predict this weight. Reasons given for choices reflected perceived attributes more than measured features. Participants also differed stably in how much they weighted several concrete features, especially length, but with a median of 6 conversations per participant, these weights were estimated poorly. The largest gain from personal information in predicting held-out choices was 0.13 percentage points, compared with 1.11 for a stronger pooled model. Stated priorities thus track perceived attributes of AI responses but say little about the concrete features that ratings reward, a distinction that matters for preference elicitation and AI personalization.

Figures & tables

Explanation	Prediction	Evidence in this study
Limited self-insight	Weak alignment persists when attributes are measured in ways that match participants’ perceptions and individual policies are reliable	Alignment by representation (Section 4.4 ); reliability and recoverability of individual weights (Section 4.7 )
Measurement mismatch	Alignment strengthens as attribute measures move closer to how participants perceive the attributes	Perceptual correspondence of text proxies (Section 4.2 ); proxies versus lens composites versus perceived profiles (Section 4.4 )
Aggregation, reliability and general quality	Alignment with aggregate profiles reflects averaging, a shared quality dimension or selection into chosen responses rather than attribute-specific perceptions	Attribute-specific profile deviations, mismatched pairings and profile reliability (Section 4.4 )
Context dependence	Feature weights vary with the request in systematic ways that a single summary cannot represent	Context interactions (Section 4.1 ); occasion versus person components (Section 4.7 )

Table 1: Explanations of a weak or measure-dependent link between stated priorities and ratings, and the evidence used here.

Feature	Item it was designed to match (“It is important that an AI language model…”)	Operation
Coherence	…produces responses that are well-written and coherent	minus the number of defects: truncation, fabricated continuation of the dialogue, trigram repetition above 10%
Informational density	…produces factual and informative responses	numbers and capitalized non-initial words per 100 words
Viewpoint markers	…summarises multiple viewpoints or different worldviews	log(1 + viewpoint phrases)
Caution markers	…produces responses that are safe and do not risk harm	log(1 + caution and caveat phrases)
Relevance to prompt	…produces responses that are helpful and relevant to my requests	embedding similarity of response and prompt
Similarity to own self-description	…reflects my values or cultural perspectives	embedding similarity of response and the participant’s self-description

Table 3: Response features, named by their operations. The middle column gives the survey item each designed feature was meant to approximate. All lexicons are listed in Appendix A .

Figure 1: Conditional associations of response features with ratings. Filled markers: first turn (Equation 1 ). Hollow markers: later turns, within pairs of responses sampled from the same model. Bars are 95% confidence intervals clustered by participant.

Figure 2: How response features relate to perceived attributes. Cells are standardized partial within-participant associations of the chosen-minus-other feature difference with the participant’s perceived performance of the chosen response (one regression per attribute with all features; $n\geq 7,041$ conversations each). Outlined cells pair each designed feature with the attribute it was meant to approximate. Features below the white line were not elicited in the survey.

Specification	Mean	SE	$p$	Joint $p$
Primary (ipsatized, all analysed responses)	0.09	0.08	0.304	0.151
Four-response conversations only	0.09	0.09	0.311	0.407
PRISM balanced subset	0.05	0.09	0.567	0.424
US census-representative sample	0.07	0.22	0.740	0.767
UK census-representative sample	0.08	0.19	0.665	0.926
Excluding the 10% of participants with the largest same-model score gaps	0.14	0.09	0.102	0.057

Table 4: Mean moderation of the weights on the designed text proxies by stated importance (ipsatized unless stated), across specifications. Joint

p

: seven-degree-of-freedom Wald test of all moderation coefficients. SEs clustered by participant.

Figure 3: Alignment between stated priorities and ratings under three representations of the same attributes. (A) Moderation of each attribute’s weight by stated importance (ipsatized), joint model, 95% confidence intervals; grey diamonds: text proxies; blue circles: lens composites; orange squares: aggregate attribute profiles. (B) Mean over attributes: hollow markers, separate models; filled markers, joint model (95% confidence intervals); shaded bars, 95% intervals from a participant bootstrap through the construction of the representations. Text proxies: the designed features of Table 3 . Lens composites: features weighted by how they predict perceived performance, estimated on the other half of participants. Aggregate attribute profiles: other participants’ perceived performance of the producing model, from the other half and corrected for raters’ stated priorities.

Standardized association with the reported reason	M1	M2	M3
Text-proxy contrast (chosen minus others)	0.031 ∗	0.013 ∗	0.013 ∗
Length contrast	0.045 ∗	0.026 ∗	0.023 ∗
Lists and headings contrast	0.021 ∗	0.003	$-$ 0.001
Positive sentiment contrast	0.005	0.005	0.005
Validation phrases contrast	$-$ 0.015	$-$ 0.008	$-$ 0.006
Refusal language contrast	$-$ 0.013	$-$ 0.003	$-$ 0.001

Table 5: Standardized associations with the reported reason. Conversation

\times

attribute observations; variables demeaned within participant and attribute. ∗

p<0.05

(participant-clustered).

Figure 4: Individual differences. (A) Standard deviations of participant-level (stable) and conversation-level (occasion) random slopes, estimated jointly by REML on orthonormal within-conversation contrasts of within-person standardized ratings; hollow markers are not significant (participant level: Holm-corrected across features). (B) Split-half correlation of participant-level slopes, observed (black ticks) and simulated on the actual design with participant-level standard deviations set to zero, to their fitted values, or to twice those values (conversation-level standard deviations at their fitted values; 100 simulated datasets per scenario; markers are means and bars span the 5th to 95th percentiles of simulated datasets).

Figure 5: Prediction of held-out first-turn rankings. (A) Change in participant-averaged pairwise accuracy when each information source is added to the pooled linear baseline (65.8%); the boosting-based priorities comparison uses a pooled boosting baseline. Bars are 95% participant-bootstrap intervals (three repetitions of ten participant folds). Blue: self-reported information; grey: demographics and experience; orange: the participant’s own other ratings. (B) Forward-only evaluation: change in accuracy when person-specific information comes only from earlier conversations, by number of earlier conversations (reasons are in Table 27 ).

Appendix figures & tables25 assets

Supplementary material from the paper’s appendix.

Appendix

Step	$n$
Survey participants	1,500
Participants with at least one conversation	1,396
Conversations	8,011
First-turn rated responses	30,049
dropped: fewer than three words	670
dropped: conversation left with fewer than two responses	32

Appendix

Table 6: Sample flow.

Attribute	Unguided (%)	Values guided (%)	Controversy guided (%)
Values	21.4	4.8	6.6
Creativity	9.5	8.2	11.2
Fluency	1.1	0.6	0.8
Factuality	3.0	2.0	1.2
Diversity	19.0	7.9	4.8
Safety	11.2	6.2	6.1

Appendix

Table 7: Share of N/A reasons by attribute and conversation type.

Perceived attribute	Designed feature	(1)	(2)	(3)	Strongest, contrast	Strongest, level
Values	Similarity to own self-description	0.02	0.03	0.07	Refusal or AI-disclaimer language ( $-$ 0.04)	Refusal or AI-disclaimer language ( $-$ 0.08)
Creativity	Lexical diversity	0.01	0.01	0.03	Lists and headings (0.06)	Refusal or AI-disclaimer language ( $-$ 0.11)
Fluency	Coherence (no truncation, leakage, repetition)	0.09	0.07	0.11	Coherence (no truncation, leakage, repetition) (0.09)	Coherence (no truncation, leakage, repetition) (0.11)
Factuality	Informational density	0.01	0.02	0.00	Length (0.09)	Length (0.16)
Diversity	Viewpoint markers	0.08	0.09	0.12	Length (0.09)	Length (0.15)
Safety	Caution markers	$-$ 0.01	$-$ 0.01	$-$ 0.01	Coherence (no truncation, leakage, repetition) (0.04)	Coherence (no truncation, leakage, repetition) (0.05)

Appendix

Table 8: Perceptual correspondence of each designed feature with its attribute: (1) partial and (2) bivariate within-participant associations of the chosen-minus-other contrast; (3) partial association of the chosen response’s feature level, controlling for conversation type and prompt length; and the feature most strongly associated with each perceived attribute under each version (standardized coefficient in parentheses).

Attribute (text proxy)	Ipsatized	Raw (z)	Rank
Values	0.02 (0.19)	0.13 (0.17)	0.09 (0.25)
Creativity	$-$ 0.24 (0.17)	$-$ 0.16 (0.16)	$-$ 0.18 (0.19)
Fluency	0.13 (0.26)	0.06 (0.18)	0.25 (0.25)
Factuality	$-$ 0.30 (0.28)	$-$ 0.08 (0.18)	$-$ 0.36 (0.26)
Diversity	0.10 (0.21)	0.12 (0.18)	0.03 (0.22)
Safety	0.20 (0.17)	0.19 (0.16)	0.27 (0.18)

Appendix

Table 9: Attribute-level moderation of the weights on text proxies by stated importance, for three representations of importance. Coefficient (SE); † Holm-adjusted

p<0.05

within the column.

Attribute or summary	Text proxies	Lens composites	Aggregate profiles
Values	0.10 (0.20)	0.19 (0.21)	0.17 (0.29)
Creativity	$-$ 0.07 (0.17)	0.33 (0.24)	0.31 (0.30)
Fluency	$-$ 0.20 (0.31)	0.28 (0.37)	1.43 (0.34)
Factuality	$-$ 0.36 (0.28)	0.15 (0.33)	0.71 (0.34)
Diversity	$-$ 0.19 (0.27)	0.35 (0.30)	0.94 (0.28)
Safety	0.32 (0.17)	0.63 (0.20)	0.88 (0.27)

Appendix

Table 10: Moderation under three representations of the attributes: attribute-level coefficients from the joint model (SE) and summaries of the mean across specifications and resampling schemes.

Contrast of mean moderation (joint model)	Estimate	SE	$p$	Holm $p$	Bootstrap 95% CI
Profiles minus proxies	0.85	0.25	$<$ 0.001	0.002	[0.16, 1.12]
Profiles minus lens	0.51	0.28	0.070	0.127	[ $-$ 0.18, 0.98]
Lens minus proxies	0.33	0.18	0.064	0.127	[ $-$ 0.14, 0.66]

Appendix

Table 11: Pairwise contrasts between representations (joint model), with Holm correction across the three contrasts and participant-bootstrap intervals through the construction of the representations.

Profile representation interacted with stated importance	Mean moderation	SE	$p$
Matched aggregate profiles	0.93	0.23	$<$ 0.001
Attribute-specific deviations (common perceived quality removed)	0.29	0.08	$<$ 0.001
Mismatched pairings, mean over six cyclic shifts	$-$ 0.06	–	–
Mismatched pairings, largest of six cyclic shifts	0.28	–	–

Appendix

Table 12: Attribute specificity of the profile result. Each row is a separate model with the profile main effects and model fixed effects; mismatched pairings interact each priority with the profile of another attribute.

Attribute	Profile split-half $r$	Deviation split-half $r$	Lens held-out $r$
Values	0.61	0.59	0.060
Creativity	0.53	0.40	0.057
Fluency	0.83	0.41	0.077
Factuality	0.72	0.72	0.094
Diversity	0.62	0.44	0.117
Safety	0.35	0.41	0.013

Appendix

Table 13: Reliability of the representations: split-half correlations of model profiles and of their attribute-specific deviations across the two halves of raters, and held-out within-person correlations of lens composites with perceived performance.

Aggregate attribute profile specification	Mean moderation	SE	$p$
Profiles, no model FE (planned secondary check)	1.24	0.23	$<$ 0.001
Profiles + model FE	1.19	0.23	$<$ 0.001
Echo-residualized profiles	1.26	0.24	$<$ 0.001
Echo-residualized profiles + model FE	1.19	0.24	$<$ 0.001
Disjoint-half profiles	0.94	0.21	$<$ 0.001
Disjoint-half profiles + model FE	0.93	0.21	$<$ 0.001

Appendix

Table 14: Aggregate attribute profiles: construction checks (single representation, ipsatized importance).

Feature	Interaction	Coefficient	SE	$p$
Length	$\times$ controversy-guided	1.68	0.41	$<$ 0.001
Length	$\times$ values-guided	0.96	0.40	0.017
Length	$\times$ prompt length (SD)	1.38	0.18	$<$ 0.001
Lists and headings	$\times$ controversy-guided	$-$ 0.54	0.51	0.284
Lists and headings	$\times$ values-guided	$-$ 0.50	0.32	0.119
Lists and headings	$\times$ prompt length (SD)	0.25	0.18	0.165

Appendix

Table 15: Context dependence of pooled feature weights (reference: unguided conversations, average prompt length).

Code	Flagged in pool	Checked	Precision [95% CI]	Recall [95% CI]
Request for shorter responses	22	22	0.77 [0.57, 0.90]	0.22 [0.12, 0.56]
Request for longer or more detailed responses	295	40	0.75 [0.60, 0.86]	0.18 [0.11, 0.38]
Praise of brevity	143	20	1.00 [0.84, 1.00]	–
Praise of detail	587	20	0.85 [0.64, 0.95]	–
Formatting mention	196	30	0.80 [0.63, 0.90]	–
Complaint about refusal or evasion	34	10	0.70 [0.40, 0.89]	–

Appendix

Table 16: Validation of the free-text codes against an independent coder on a fresh stratified sample of 222 texts: texts the rules flag in the pool, texts checked, precision with Wilson 95% confidence interval, and recall projected from sampled unflagged texts with a stratified bootstrap interval.

Family	Feature	Timing of the report	Codes	Coef.	SE	Per SD	$p$	Holm $p$
Main	Length	Concurrent (same conversation)	requests	$-$ 0.39	0.63	$-$ 0.10	0.542	0.542
Main	Length	Other conversations	requests	$-$ 6.16	1.54	$-$ 0.78	$<$ 0.001	$<$ 0.001
Main	Length	Earlier conversations only	requests	$-$ 3.50	1.05	$-$ 0.68	$<$ 0.001	0.002
Main	Lists and headings	Other conversations	formatting mentions	2.93	1.73	0.24	0.090	0.181
Sensitivity	Length	Other conversations	requests and praise	$-$ 4.14	0.88	$-$ 0.89	$<$ 0.001	$<$ 0.001
Sensitivity	Length	Earlier conversations only	requests and praise	$-$ 2.26	0.60	$-$ 0.73	$<$ 0.001	$<$ 0.001

Appendix

Table 17: Concrete feedback and the weights on the matching features. Coefficient per unit of the feedback score (net share of conversations with requests for shorter rather than longer responses, or share with formatting mentions); per SD: per standard deviation of the score over analysed responses. Holm correction within each family; both the main and sensitivity families are exploratory.

Priorities block	Concrete feedback $\times$ length	$p$	Priorities $\times$ length, Wald (df)	$p$
Raw priorities	$-$ 6.23 (1.54)	$<$ 0.001	20.0 (7)	0.006
Ipsatized priorities	$-$ 6.38 (1.53)	$<$ 0.001	9.0 (6)	0.171

Appendix

Table 18: Common model for the length weight: concrete feedback from other conversations and the seven stated priorities entered together.

Non-elicited feature	Stated scale	Joint Wald $\chi^{2}$	df	$p$
Length	ipsatized (6-column basis)	8.3	6	0.216
Length	raw	20.0	7	0.006
Refusal or AI-disclaimer language	ipsatized (6-column basis)	10.9	6	0.093
Refusal or AI-disclaimer language	raw	13.3	7	0.065
Lists and headings	ipsatized (6-column basis)	9.5	6	0.149
Lists and headings	raw	9.6	7	0.215

Appendix

Table 19: Do the stated priorities jointly moderate the weights on non-elicited features? Ipsatized priorities use a six-column basis because the seven scores sum to zero.

Stated importance $\times$ feature	Context	Stated scale	$\gamma$	SE	$p$	Holm $p$
Safety $\times$ Refusal or AI-disclaimer language	all	ipsatized	0.40	0.22	0.062	0.559
Safety $\times$ Refusal or AI-disclaimer language	all	raw	0.39	0.20	0.055	0.439
Safety $\times$ Caution markers	all	ipsatized	0.19	0.17	0.262	1.000
Safety $\times$ Caution markers	all	raw	0.20	0.16	0.218	1.000
Safety $\times$ Refusal or AI-disclaimer language	controversy	ipsatized	0.35	0.29	0.222	1.000
Safety $\times$ Refusal or AI-disclaimer language	other	ipsatized	0.44	0.29	0.125	1.000

Appendix

Table 20: Exploratory pairings of stated importance with non-elicited features, overall and by conversation type; Holm correction within each stated scale.

Reason attribute	$n$	Same-attribute perception	Mean cross-attribute	Largest cross-attribute
Values	6,034	0.49	0.01	0.11
Creativity	6,050	0.56	0.01	0.06
Fluency	6,127	0.31	0.07	0.17
Factuality	6,120	0.47	0.04	0.17
Diversity	6,049	0.58	0.00	0.04
Safety	6,045	0.20	0.05	0.10

Appendix

Table 21: Specificity of reasons: standardized associations with perceived performance on all seven attributes entered together (complete cases, demeaned within participant).

Feature	$\hat{\tau}_{\mathrm{person}}$	$\hat{\tau}_{\mathrm{conv}}$	LR	$p$	Holm $p$	$\hat{\tau}_{\mathrm{person}}$ , context	$\hat{\tau}_{\mathrm{person}}$ , raw
Similarity to own self-description	0.067	0.105	4.0	0.023	0.162	0.067	1.79
Lexical diversity	0.032	0.001	0.5	0.248	0.994	0.035	0.96
Coherence (no truncation, leakage, repetition)	0.100	0.162	15.0	$<$ 0.001	$<$ 0.001	0.101	2.72
Informational density	0.085	0.124	7.8	0.003	0.021	0.080	2.42
Viewpoint markers	0.046	0.074	1.1	0.144	0.864	0.050	1.37
Caution markers	0.046	0.083	1.1	0.148	0.864	0.043	0.99

Appendix

Table 22: Participant-level and conversation-level random slopes, REML (within-person standardized ratings unless stated).

p

: likelihood ratio for the participant component against the 50:50

\chi^{2}

mixture; context: model with shared context interactions; raw: raw rating points.

Feature	Simulated scenario	$n$	Observed LR	95th pct.	Max.	Rejection (MC SE)
Length	Null, homoscedastic	100	102.6	2.77	5.73	0.06 (0.02)
Length	Null, person-specific residual SD	100	102.6	3.65	7.78	0.09 (0.03)
Refusal or AI-disclaimer language	Null, homoscedastic	100	29.9	1.59	3.67	0.03 (0.02)
Coherence (no truncation, leakage, repetition)	Null, homoscedastic	100	15.0	1.68	4.98	0.02 (0.01)
Lists and headings	Null, homoscedastic	100	15.9	2.97	5.38	0.07 (0.03)
Length	Person SD at its fitted value	50	102.6	–	–	1.00 (0.00)

Appendix

Table 23: Calibration of the participant-component test by parametric bootstrap (within-person standardized ratings): number of simulated datasets, observed likelihood ratio, 95th percentile and maximum of the simulated likelihood ratios, and the rejection rate of the nominal 5% test with its Monte Carlo SE.

Feature	Pooled	$\hat{\tau}^{2}_{\mathrm{person}}$	95% CI	Observed $r$	Sim., zero	Sim., fitted	Sim., 2 $\times$
Similarity to own self-description	0.83	6.3	[0.7, 13.0]	0.049	0.000	0.019	0.058
Lexical diversity	$-$ 0.07	$-$ 3.9	[ $-$ 12.1, 4.2]	$-$ 0.024	0.001	0.008	0.014
Coherence (no truncation, leakage, repetition)	1.58	12.8	[1.6, 24.7]	0.048	0.004	0.018	0.062
Informational density	$-$ 0.51	8.3	[ $-$ 3.2, 21.6]	0.031	0.000	0.020	0.061
Viewpoint markers	0.65	6.6	[ $-$ 3.6, 18.5]	0.029	0.002	0.017	0.055
Caution markers	0.05	8.4	[ $-$ 3.0, 20.7]	0.036	$-$ 0.001	0.020	0.057

Appendix

Table 24: Moment estimates of person-stable slope variance (raw rating points per feature SD; participant-bootstrap interval of the 50-split average) and split-half correlations of person slopes: observed, and simulated with participant-level standard deviations set to zero, to their fitted values or to twice those values.

Information added	Gain	95% CI	Top choice	$\Delta$ MSE	Better (%)	Repeats
Own free-text self-description (similarity)	$-$ 0.03	[ $-$ 0.11, 0.05]	0.01	$-$ 0.1	22	[ $-$ 0.10, 0.02]
Own free-text wishes for AI (similarity)	0.06	[ $-$ 0.01, 0.13]	0.26	0.0	22	[0.00, 0.12]
Importance sliders $\times$ features	$-$ 0.23	[ $-$ 0.45, 0.00]	$-$ 0.51	3.1	39	[ $-$ 0.30, $-$ 0.18]
Importance sliders $\times$ aggregate attribute profiles	0.02	[ $-$ 0.11, 0.17]	0.25	$-$ 1.3	33	[ $-$ 0.06, 0.11]
Stated-profile neighbours	$-$ 0.11	[ $-$ 0.33, 0.12]	$-$ 0.20	0.5	38	[ $-$ 0.29, 0.00]
Demographics and AI experience $\times$ features	$-$ 0.55	[ $-$ 0.85, $-$ 0.26]	$-$ 1.06	8.6	41	[ $-$ 0.90, $-$ 0.33]

Appendix

Table 25: Prediction of held-out rankings: gains in pairwise accuracy over the pooled baseline (gradient-boosting sliders row: over a symmetric boosting baseline), top-choice gains, change in squared rating error for linear rating models, share of participants with higher accuracy, and range of the average gain over the three repetitions. Intervals are participant-bootstrap intervals, not corrected for multiple comparisons.

Watch minus ask
Paired comparison	Difference (pp)	95% CI
Model taste minus sliders $\times$ features	0.36	[0.10, 0.63]
Model taste minus sliders $\times$ profiles	0.11	[ $-$ 0.09, 0.32]
Model taste minus stated-profile neighbours	0.24	[ $-$ 0.02, 0.50]
Feature weights minus sliders $\times$ features	0.21	[ $-$ 0.05, 0.46]
Feature weights minus sliders $\times$ profiles	$-$ 0.04	[ $-$ 0.25, 0.15]

Appendix

Table 26: Direct paired comparisons of information sources (difference in participant-level pairwise accuracy).

Earlier conversations	Feature weights	Model taste	Reasons	Concrete feedback
1	0.00 [ $-$ 0.30, 0.31]	0.16 [ $-$ 0.12, 0.39]	$-$ 0.18 [ $-$ 0.96, 0.53]	0.21 [ $-$ 0.15, 0.59]
2	0.07 [ $-$ 0.25, 0.41]	0.01 [ $-$ 0.33, 0.34]	$-$ 0.36 [ $-$ 0.95, 0.25]	0.02 [ $-$ 0.29, 0.34]
3	$-$ 0.33 [ $-$ 0.67, $-$ 0.01]	$-$ 0.24 [ $-$ 0.69, 0.17]	$-$ 0.47 [ $-$ 1.10, 0.11]	0.06 [ $-$ 0.22, 0.34]
4	0.03 [ $-$ 0.35, 0.39]	0.07 [ $-$ 0.44, 0.54]	0.15 [ $-$ 0.47, 0.78]	0.12 [ $-$ 0.14, 0.36]
5 or more	0.04 [ $-$ 0.36, 0.44]	0.32 [ $-$ 0.17, 0.79]	$-$ 0.03 [ $-$ 0.65, 0.61]	0.17 [ $-$ 0.11, 0.44]
Trend per earlier conversation	0.00 [ $-$ 0.10, 0.11]	0.04 [ $-$ 0.09, 0.16]	0.08 [ $-$ 0.13, 0.29]	0.00 [ $-$ 0.09, 0.10]

Appendix

Table 27: Forward-only prediction: gain (pp) and 95% interval by number of earlier conversations, and the linear trend over history.

Planned model	Accuracy (%)	Gain (pp)	95% CI
P0 pooled features + model + position	65.59	–	–
P1 + demographics $\times$ features	65.27	$-$ 0.32	[ $-$ 0.68, 0.04]
P2 + stated $\times$ features	65.53	$-$ 0.06	[ $-$ 0.32, 0.19]
P3 + reasons $\times$ features	65.54	$-$ 0.05	[ $-$ 0.36, 0.24]
P5 + person-specific weights (other conversations)	65.68	0.09	[ $-$ 0.16, 0.33]
P6 + stated, demographics and reasons	64.93	$-$ 0.66	[ $-$ 1.07, $-$ 0.25]

Appendix

Table 28: Planned prediction contest (baseline includes similarity to the participant’s self-description).

Contrast in mean moderation	Difference	SE	$p$
Very familiar minus not familiar	$-$ 0.26	0.22	0.255
Very familiar minus somewhat familiar	0.01	0.16	0.968
Weekly or more minus less than weekly	$-$ 0.08	0.16	0.630

Appendix

Table 29: Experience: direct contrasts in mean moderation (text proxies).

Display position	Difference vs position 1 (points)	SE
2	0.03	0.64
3	$-$ 1.17	0.65
4	$-$ 1.15	0.64
5	$-$ 0.58	0.65
6	$-$ 1.06	0.65
7	$-$ 1.42	0.65

Appendix

Table 30: Stated importance by randomized display position (participant and attribute fixed effects).

User Preference Modeling

User preference modeling infers what people value from choices, behavior, and feedback to personalize recommendations, predictions, and AI actions.

See all 271 papers

What Do People Actually Want From AI? Mapping Preference Plurality
Jun 4, 2026Julia Sepúlveda Coelho, Scott A. HaleLLM Alignment Human Preference Modeling
Oxford Internet Institute, University of Oxford, United Kingdom · Meedan, United States
Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
Oct 22, 2026Yijiang River Dong, Zexue He, Cheng Yu +9Memory-Augmented Language Models Reward Modeling
University of Cambridge · Stanford University · Technical University of Munich +2
Probing Persona-Dependent Preferences in Language Models
May 13, 2026Oscar Gilg, Pierre Beckmann, Daniel Paleka +1LLM Alignment LLM Interpretability
MATS · MATS, EPFL · ETH Zürich +1

Jun 4, 2026J/K move · Enter open · S save

What Do People Actually Want From AI? Mapping Preference Plurality

Julia Sepúlveda Coelho, Scott A. Hale

Oxford Internet Institute, University of Oxford, United Kingdom · Meedan, United States

LLM Alignment Human Preference Modeling Multi-Objective Language Model Alignment Pluralistic Alignment

Open paper AskPDF arXiv

Large Language Models (LLMs) are often fine-tuned through Reinforcement Learning from Human Feedback (RLHF) to align with people's preferences and values. However, this method has known limitations: it aggregates conflicting preferences, often relies on unrepresentative samples, and uses only binary comparisons. Analysing 1,500 open-ended responses from the PRISM dataset across 75 countries, we examine what people actually want from AI systems and reveal concrete failures of current methods. We find that different people want different things: most values are requested by fewer than a quarter of respondents, with truthfulness the sole exception at 49%. Furthermore, the same words hide divergent meanings: when people describe what they mean by "truthfulness", they reveal distinct, potentially incompatible, epistemological bases, as some ask for sourced claims, some for expert opinions, and some even ask for unpopular views. Certain capabilities, namely how human-like a model behaves, and some features, like AI guardrails, are outright controversial, with some desiring them and others rejecting them. We additionally find that people often use contextual distinctions (what AI should do "by default" versus "if requested") that binary comparisons cannot capture. These findings expose fundamental problems in current alignment practices. When 49% request truthfulness but define it differently, this is unlikely to be captured by a single reward model. The persistence of high hallucination rates in well-funded models, despite users' clear demands for accuracy, suggests that current methods fail to identify actual preferences. This paper sheds light on the situated, contested, imperfect signals that are currently being flattened into universal preference models, a practice others have characterised as epistemic violence.

Abstract

User Preference Modeling

Explore similar work

What Do People Actually Want From AI? Mapping Preference Plurality

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

Probing Persona-Dependent Preferences in Language Models