Online LLM Personalization from Pairwise Preferences Across Users

Romain Puech; Georgy Salakhutdinov; Dimitris Bertsimas

cs.LGOct 12, 2026

Online LLM Personalization from Pairwise Preferences Across Users

Authors: Romain Puech, Georgy Salakhutdinov, Dimitris Bertsimas

Organizations: Massachusetts Institute of Technology

Abstract

Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.

Figures & tables

Figure 1: Overview. A new user arrives with hidden preferences and asks

b

questions. On each question, the policy picks two of the

K=6

personas, using earlier users’ sessions and the current user’s choices so far, and the user chooses between the two generated answers. Strong regret, computed from hidden utilities, sums the gaps of both displayed answers to the user’s best answer.

method	held-out profiles		Composite		Quirky
	excess	score	excess	score	excess	score
random pairs	2317.3	0	2276.1	0	2305.5	0
best fixed pair	2197.5	5	1900.0	17	1790.3	22
best fixed pair per user (hindsight)	546.7	76	763.2	66	726.6	68
per-user linear oracle	356.8	85	427.8	81	411.8	82
distinct-pair floor $G_{T}$	0.0	100	0.0	100	0.0	100

Table 1: Benchmark results on the three rosters of new students on new topics (

u=350

arrivals each,

b=24

, students and topics never used in design). Excess regret

R_{T}-G_{T}

summed over the run (lower is better), mean

\pm

s.e. over seeds (Appendix B ), and the benchmark score, the share of random pairs’ excess regret that a policy removes (higher is better; Section 4.1 ). Italic rows are evaluation references (Section 4.4 ). The distinct-pair floor

G_{T}

, which scores

100

, follows every question-specific preference, including judge disagreement between display orders, and is not an attainable target; the per-user linear oracle, fitted in hindsight on CODuel’s features, indicates what a learner linear in these features can reach. The no-transfer ablation tests the contribution of learning across users (Section 6.1 ). The floor

G_{T}

, already subtracted from every entry: 779 (held-out profiles), 699 (Composite), 679 (Quirky).

Figure 2: Excess regret per user as users arrive on the three evaluation rosters (

b=24

): each user’s excess regret averaged over seeds and smoothed over neighboring arrivals, with 95% bands.

Appendix figures & tables28 assets

Supplementary material from the paper’s appendix.

Appendix

share of centered score variance (%)
statistic	development	held-out topics	held-out profiles	Composite	Quirky
arrivals	300	350	350	350	350
distinct profiles	8	8	8	350	350
shared persona effects	18.0	20.5	7.1	13.1	13.0
profile–persona differences	31.7	28.2	50.3	30.4	34.5
sessions within profiles	2.0	2.1	1.9	0.0	0.0

Appendix

Table 2: Dataset statistics by roster (

24

questions per arrival). Variance shares decompose question-centered Bradley–Terry scores into a shared persona effect, profile deviations, session deviations within a profile and question deviations within a session; the question deviations include the judge’s disagreement between display orders (Appendix C.2 ). On the Composite and Quirky rosters every profile has one arrival, so profile and session deviations coincide. Preference concentration is the KL divergence from an unbiased choice, measured in nats (units based on natural logarithms). Concentration and disagreement average over all

15

unordered persona pairs per question.

diagnostic	development	held-out topics	held-out profiles	Composite	Quirky
sampled choices that are a valid label (%)	100.0	100.0	100.0	100.0	100.0
prompts whose $4$ choices agree (%)	77	78	79	73	74
persona pairs with all $16$ choices on one side (%)	40	41	41	38	39
reliability: split of each prompt’s choices	0.99	0.99	0.99	0.99	0.99
lean toward the first-shown answer (logits)	+0.38	+0.38	+0.48	+0.24	+0.24
lean toward the label A (logits)	+0.22	+0.21	+0.13	+0.22	+0.23

Appendix

Table 3: Diagnostics of the judge ( gpt-6-luna ,

4

sampled choices per prompt) on the stored judgments of all

24

questions of every arrival (four prompts per persona pair and question). A prompt’s log-odds is capped at

\log 9

when its four choices agree. Correlations are between the logits averaged over one half of the prompts and over the other half, across persona pairs. Reliabilities are Spearman–Brown estimates for the four-prompt average, from Bradley–Terry scores fitted separately on the two display orders: of each question’s scores minus the session’s mean scores, and of each session’s mean scores after removing the persona means. Length: log ratio of the two answers’ word counts against the four-prompt logit, both centered within student profile and persona pair.

(a) Pairs of raters
Pair	Agreement (%)	Cohen’s $\kappa$
Author 1 and author 2	69.6 [61.8, 77.0]	0.39
gpt-6-luna and author 1	74.0	0.48
gpt-6-luna and author 2	73.0	0.46
TypeSafe Jev and author 1	71.1	0.42
TypeSafe Jev and author 2	76.0	0.52

Appendix

Table 4: Agreement with the authors’ choices on the 204 comparisons, each labeled by both authors. Brackets: 95% intervals of a bootstrap over the 51 labeled (student, question) cells. (a) Agreement between each pair of raters, the two judges included; the judge’s favorite is its more probable answer. (b) By the confidence of gpt-6-luna , the probability it gives its favorite (at most

0.9

): how often the authors choose that favorite, how often they agree with each other, and the agreement two raters would reach if each chose the favorite with the judge’s probability. (c) Other judges run with the same four prompts; same favorite: share of comparisons on which the judge favors the same answer as gpt-6-luna .

statistic	held-out profiles	Composite	Quirky
same favored answer, comparisons (%)	84.3	78.5	79.4
where gpt-6-luna gives $0.9$ (%)	96.1	92.5	92.8
click agreement, Jev with gpt-6-luna	0.68	0.64	0.65
two gpt-6-luna clicks	0.70	0.70	0.70
two Jev clicks	0.75	0.70	0.71
same most-preferred persona, students (%)	88	72	77

Appendix

Table 5: Agreement of the second judge (TypeSafe Jev) with the dataset’s judge ( gpt-6-luna ) on the same comparisons of the three evaluation rosters (378,000 comparisons of two personas’ answers, 1,050 students). Favored answer: comparisons where neither judge gives

0.5

. Click agreement: probability that a click from each judge picks the same answer. Most-preferred persona: utilities summed over the student’s questions. Last rows: benchmark score under Jev’s utilities of each student’s best fixed pair, chosen from either judge’s utilities.

method	held-out profiles		Composite		Quirky
	excess	score	excess	score	excess	score
random pairs	3008.2	0	2529.8	0	2551.8	0
best fixed pair	2510.0	17	1329.1	47	1809.3	29
per-user linear oracle	293.4	90	264.5	90	259.7	90
distinct-pair floor $G_{T}$	0.0	100	0.0	100	0.0	100
RUCB (pooled, non-contextual)	2560.3 $\pm$ 8.0	15	1545.7 $\pm$ 8.0	39	1866.6 $\pm$ 7.7	27

Appendix

Table 6: Second judge (TypeSafe Jev): benchmark results on the three rosters of new students on new topics, same layout as Table 1 , with clicks and evaluation utilities from Jev and CODuel reading the scores of a second rater, gpt-6-luna . Every comparison method is tuned on the development roster judged by Jev (64 seeds on the held-out profiles, 32 on Composite and Quirky). The floor

G_{T}

, already subtracted from every entry: 977 (held-out profiles), 887 (Composite), 873 (Quirky).

compared with	held-out profiles	Composite	Quirky
RUCB (pooled, non-contextual)	+51.4% [+50.1%, +52.7%]	+27.0% [+22.4%, +31.6%]	+36.2% [+33.2%, +39.5%]
Pooled	+51.6% [+48.8%, +54.5%]	+36.3% [+26.9%, +46.3%]	+40.5% [+34.4%, +47.5%]
CoLSTIM	+26.3% [+24.9%, +27.6%]	+26.6% [+24.0%, +29.0%]	+21.9% [+19.5%, +24.1%]
COLDB	+30.5% [+28.4%, +32.7%]	+22.3% [+19.1%, +25.5%]	+21.8% [+19.1%, +24.8%]
COLDB-LR	+25.4% [+22.9%, +27.9%]	+26.2% [+23.3%, +29.3%]	+25.1% [+22.1%, +28.2%]
Meta-TS	+49.8% [+47.9%, +51.7%]	+23.6% [+17.2%, +30.6%]	+36.5% [+32.4%, +40.4%]

Appendix

Table 7: Second judge (TypeSafe Jev): paired reductions in excess regret of CODuel relative to each comparator, expressed as percentages of that comparator’s excess regret. Brackets give 95% bootstrap intervals (Appendix B ). Table 6 gives the benchmark scores.

method	held-out topics		development
	excess	score	excess	score
random pairs	2613.7	0	2238.3	0
best fixed pair	1468.9	44	1320.1	41
per-user linear oracle	279.3	89	227.8	90
distinct-pair floor $G_{T}$	0.0	100	0.0	100
RUCB (pooled, non-contextual)	1697.0 $\pm$ 5.9	35	1565.8 $\pm$ 9.0	30

Appendix

Table 8: Second judge (TypeSafe Jev): held-out-topic roster (64 seeds) and development roster (32 seeds), same layout as Table 1 . The floor

G_{T}

, already subtracted from every entry: 843 (held-out topics), 728 (development).

Figure 3: The 16 student profiles: rows 1–8 are the development profiles (measured on the development roster), rows 9–16, marked with an asterisk, the new profiles (measured on the held-out-profile roster). (a) Cosine similarity between profiles’ mean question-centered Bradley–Terry scores, by position in the question chain (definition, computation, why-question) and persona. Blue and red indicate negative and positive similarity. (b) Percentage of questions on which each persona has the highest reference utility. Darker cells indicate higher percentages.

profile	users ( $n$ )	CODuel	CoLSTIM
retaking the course	44	29	25
self-taught programmer	44	44	26
returning adult	44	56	33
olympiad student	44	36	30
humanities graduate	44	61	45
engineer with a deadline	44	42	25

Appendix

Table 9: Held-out-profile roster, by profile: benchmark score on that profile’s users (share of random pairs’ excess regret removed), mean over 64 seeds, from the runs of Table 1 split after the fact. Higher is better. CoLSTIM is the comparison method with the lowest excess regret on the full held-out-profile roster.

policy	held-out profiles	Composite	Quirky
CODuel	55 [45, 64]	45 [42, 47]	46 [43, 49]
COLDB	33 [8, 54]	38 [32, 43]	38 [26, 45]
CoLSTIM	34 [29, 39]	28 [26, 30]	29 [26, 31]
Meta-TS	32 [12, 53]	34 [27, 40]	36 [27, 44]
HierTS	33 [15, 55]	36 [29, 41]	38 [29, 44]
ablation: no transfer	31 [27, 35]	27 [25, 29]	27 [25, 30]

Appendix

Table 10: Benchmark score over redrawn rosters: every seed draws a new roster of the same kind (the eight held-out profiles drawn with replacement;

350

Composite or Quirky students drawn with replacement), and the policies are rerun on it. Mean over 64 seeds and, in brackets, the 2.5th and 97.5th percentiles of the per-seed values, which describe the spread of a new roster of that kind (students, arrival order and clicks together). The last rows report paired differences in score points, computed for methods run on the same redrawn roster and seed. Their ranges show how the advantage varies across rosters.

method	held-out profiles		Composite		Quirky
	excess	score	excess	score	excess	score
random pairs	2317.3	0	2276.1	0	2305.5	0
best fixed pair	2197.5	5	1900.0	17	1790.3	22
per-user linear oracle	356.8	85	427.8	81	411.8	82
distinct-pair floor $G_{T}$	0.0	100	0.0	100	0.0	100
RUCB (pooled, non-contextual)	1918.5 $\pm$ 5.9	17	1557.1 $\pm$ 8.1	32	1528.8 $\pm$ 7.0	34

Appendix

Table 11: Every comparison method on the three evaluation rosters, same layout as Table 1 . The floor

G_{T}

, already subtracted from every entry: 779 (held-out profiles), 699 (Composite), 679 (Quirky).

compared with	held-out profiles	Composite	Quirky
RUCB (pooled, non-contextual)	+37.6% [+36.3%, +38.8%]	+18.0% [+15.4%, +20.9%]	+18.2% [+15.4%, +20.9%]
Pooled	+37.6% [+34.3%, +41.1%]	+18.5% [+14.1%, +22.9%]	+19.5% [+13.0%, +26.1%]
CoLSTIM	+22.5% [+21.2%, +23.8%]	+22.0% [+19.9%, +24.1%]	+24.2% [+22.0%, +26.5%]
COLDB	+33.5% [+31.0%, +36.3%]	+9.3% [+5.9%, +12.7%]	+12.5% [+7.7%, +17.4%]
COLDB-LR	+33.9% [+32.5%, +35.3%]	+15.3% [+12.6%, +18.1%]	+15.5% [+12.8%, +18.2%]
Meta-TS	+33.3% [+31.9%, +34.5%]	+14.6% [+11.5%, +17.9%]	+16.2% [+12.6%, +19.4%]

Appendix

Table 12: Paired comparison of CODuel with each comparison method: how much lower its excess regret is, as a share of that method’s excess (positive: CODuel is better). The 95% interval resamples seeds and users after the fact, so it is conditional on these students (Appendix B ).

method	held-out topics		development
	excess	score	excess	score
random pairs	2341.4	0	2000.9	0
best fixed pair	1315.0	44	1173.9	41
per-user linear oracle	364.6	84	307.0	85
distinct-pair floor $G_{T}$	0.0	100	0.0	100
RUCB (pooled, non-contextual)	1389.1 $\pm$ 4.3	41	1257.6 $\pm$ 6.6	37

Appendix

Table 13: Held-out-topic roster (development profiles, new topics; 64 seeds) and development roster (32 seeds, used for design, descriptive): same layout as Table 1 . The floor

G_{T}

, already subtracted from every entry: 659 (held-out topics), 568 (development).

compared with	held-out topics	development
RUCB (pooled, non-contextual)	+21.7% [+20.3%, +23.3%]	+22.7% [+20.7%, +24.6%]
Pooled	+23.9% [+21.1%, +26.9%]	+29.9% [+22.7%, +38.8%]
CoLSTIM	+33.4% [+32.2%, +34.5%]	+29.7% [+28.1%, +31.3%]
COLDB	+17.9% [+16.3%, +19.5%]	+18.3% [+16.3%, +20.4%]
COLDB-LR	+18.9% [+17.5%, +20.5%]	+19.4% [+17.5%, +21.2%]
Meta-TS	+20.3% [+18.8%, +22.0%]	+20.8% [+19.0%, +22.6%]

Appendix

Table 14: Held-out-topic roster and development roster: same layout as Table 12 .

method	held-out profiles		Composite		Quirky
	first 25%	last 25%	first 25%	last 25%	first 25%	last 25%
no transfer	4.61	4.65	4.74	4.75	4.77	4.80
CODuel	3.90	3.19	4.08	3.45	4.01	3.37
type-library reference	5.10	5.16	4.42	4.42	4.38	4.42

Appendix

Table 15: Excess regret per user in the first and last quarter of the arrivals (mean over seeds).

$\beta=4$
population	no transfer	RUCB	CoLSTIM	COLDB	COLDB-LR	Pooled	Meta-TS	HierTS	AdaTS	CODuel	oracle
2 types, $60^{\circ}$	936 $\pm$ 6	2066 $\pm$ 22	911 $\pm$ 7	288 $\pm$ 9	260 $\pm$ 10	298 $\pm$ 10	332 $\pm$ 9	322 $\pm$ 9	325 $\pm$ 9	178 $\pm$ 3	105 $\pm$ 3
2 types, $90^{\circ}$	930 $\pm$ 5	2045 $\pm$ 25	905 $\pm$ 6	624 $\pm$ 19	308 $\pm$ 7	644 $\pm$ 20	675 $\pm$ 18	660 $\pm$ 19	664 $\pm$ 18	219 $\pm$ 2	156 $\pm$ 4
4 types, $60^{\circ}$	934 $\pm$ 4	2081 $\pm$ 18	907 $\pm$ 5	440 $\pm$ 11	472 $\pm$ 10	454 $\pm$ 11	487 $\pm$ 11	476 $\pm$ 11	477 $\pm$ 11	297 $\pm$ 5	208 $\pm$ 4
4 types, $90^{\circ}$	932 $\pm$ 4	2061 $\pm$ 16	905 $\pm$ 5	1051 $\pm$ 18	600 $\pm$ 10	1075 $\pm$ 19	1101 $\pm$ 19	1083 $\pm$ 18	1087 $\pm$ 18	401 $\pm$ 4	335 $\pm$ 5
8 types, $60^{\circ}$	926 $\pm$ 4	2068 $\pm$ 16	903 $\pm$ 5	537 $\pm$ 7	669 $\pm$ 9	551 $\pm$ 8	586 $\pm$ 8	574 $\pm$ 8	576 $\pm$ 8	416 $\pm$ 4	304 $\pm$ 4

Appendix

Table 16: Simulator,

b=20

u=300

: excess regret (mean

\pm

s.e. over seeds) by population and policy. COLDB-LR is our likelihood-ratio variant of COLDB; the population oracle starts each user from the true

\mathcal{P}

method	tutoring	simulator
CoLSTIM	$c\in\{0,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $\ell\in\{0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1\}$ $\gamma\in\{0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ 64 configurations	$c\in\{0,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $\ell\in\{0.3,\allowbreak 1,\allowbreak 3,\allowbreak 10\}$ $\gamma\in\{0.01,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ 96 configurations
COLDB	$\alpha\in\{0.001,\allowbreak 0.003,\allowbreak 0.01,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $c\in\{0,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1\}$ 40 configurations	$\alpha\in\{0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $c\in\{0,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1\}$ 20 configurations
COLDB-LR	$\mathrm{tol}\in\{0.25,\allowbreak 0.5,\allowbreak 1,\allowbreak 2,\allowbreak 4\}$ $c\in\{0,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1\}$ 20 configurations	$\mathrm{tol}\in\{0.25,\allowbreak 0.5,\allowbreak 1,\allowbreak 2,\allowbreak 4\}$ $c\in\{0,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1\}$ 20 configurations
RUCB	$\alpha\in\{0.51,\allowbreak 1,\allowbreak 2,\allowbreak 4\}$ 4 configurations	$\alpha\in\{0.51,\allowbreak 1,\allowbreak 2,\allowbreak 4\}$ 4 configurations
Meta-TS	$s_{0}\in\{0.0003,\allowbreak 0.001,\allowbreak 0.003,\allowbreak 0.01,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $s_{q}\in\{0.001,\allowbreak 0.01,\allowbreak 0.1,\allowbreak 1,\allowbreak 10,\allowbreak 100,\allowbreak 1000\}$ 63 configurations	$s_{0}\in\{0.0003,\allowbreak 0.001,\allowbreak 0.003,\allowbreak 0.01,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $s_{q}\in\{0.001,\allowbreak 0.01,\allowbreak 0.1,\allowbreak 1,\allowbreak 10,\allowbreak 100,\allowbreak 1000\}$ 63 configurations
HierTS	$s_{0}\in\{0.0003,\allowbreak 0.001,\allowbreak 0.003,\allowbreak 0.01,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $s_{q}\in\{0.1,\allowbreak 1,\allowbreak 10,\allowbreak 100,\allowbreak 1000\}$ 45 configurations	$s_{0}\in\{0.0003,\allowbreak 0.001,\allowbreak 0.003,\allowbreak 0.01,\allowbreak 0.03,\allowbreak 0.1,\allowbreak 0.3,\allowbreak 1,\allowbreak 3\}$ $s_{q}\in\{0.001,\allowbreak 0.01,\allowbreak 0.1,\allowbreak 1,\allowbreak 10,\allowbreak 100,\allowbreak 1000\}$ 63 configurations

Appendix

Table 17: Search grids and selected settings. Every configuration uses

16

tuning seeds, separate from benchmark evaluation: the development roster (

b=24

) for tutoring and the

16

structured

b=20

settings for the simulator. The ratio is excess regret divided by that of no transfer, averaged over settings (lower is better). Here

c

scales exploration,

\ell

multiplies the base ridge

d/B_{\theta}^{2}

, and

\gamma

scales CoLSTIM’s Gumbel perturbations (clipped to

[-3,3]

before scaling). For COLDB,

\alpha

scales the distance threshold; for RUCB, it controls the confidence width.

\mathrm{tol}

is COLDB-LR’s likelihood tolerance. The variance multipliers

s_{0}

and

s_{q}

scale

v_{0}=B_{\theta}^{2}/d

for user variation and the population-mean prior, respectively.

c_{\mathrm{UCB}}

is the log-weight margin used to select plausible components in the latent-bandit UCB ablation. Methods use the supplied features without appending additional persona indicators.

method	held-out profiles			Composite			Quirky
	dev. choice	best on roster	its configuration	dev. choice	best on roster	its configuration	dev. choice	best on roster	its configuration
CoLSTIM	33	34	c1:l0.1:g0.1	28	28	c1:l0.3:g0.3	28	29	c1:l0.1:g0.1
COLDB	22	32	a0.01:c0.03	38	39	a0.03:c0	38	40	a0.03:c0.1
COLDB-LR	22	34	t0.25:c0.3	34	34	t2:c0.1	36	36	t2:c0.3
RUCB	17	17	a0.51	32	32	a0.51	34	34	a0.51
Meta-TS	23	23	s0=3:sq=10	34	36	s0=0.0003:sq=1	35	38	s0=0.003:sq=1

Appendix

Table 18: Benchmark score of each comparison method with the configuration chosen on the development roster (dev. choice, as in Table 1 ) and with the configuration of its grid (Table 17 ) that has the lowest excess regret on the evaluation roster itself (best on roster, chosen on the evaluation seeds and therefore optimistic), with that configuration. The last row repeats CODuel’s score.

method	held-out profiles	Composite	Quirky
Meta-TS	-1.3 [-2.0, -0.6]	+0.8 [-1.9, +3.5]	-0.6 [-3.6, +2.5]
HierTS	-0.1 [-0.7, +0.5]	+1.5 [+0.4, +2.5]	-1.7 [-3.7, +0.2]
AdaTS	+0.8 [-1.6, +3.2]	+1.3 [-1.2, +3.7]	-2.1 [-6.3, +2.2]

Appendix

Table 19: Prior-learning Thompson samplers with a double-Thompson-sampling challenger (the second persona maximizes an independent second posterior sample, prior variances re-tuned on the development roster) against the reported rule (the two highest utilities of one sample): paired change in excess regret as a share of the reported version’s excess regret, with a 95%

t

interval over seeds. Negative values favor the double-sample rule.

Figure 4: Paired gain of CODuel over the no-transfer ablation as a share of the ablation’s excess regret, by click sharpness $\tau$ and session length $b$ ( $32$ seeds per cell; $\tau\neq 1$ uses the Bradley–Terry-consistent click table), on the development roster and on its scrambled version. The scrambled panel checks for increased regret when persona labels are permuted independently across users. A small circle marks cells not resolved at Benjamini–Hochberg level $0.05$ . A session of length $b<24$ asks its first $b$ questions; $b=48$ repeats the private pack.

Figure 5: Development roster: paired gain of the type-library reference over no transfer as a share of the latter’s excess regret, by click sharpness $\tau$ and session length $b$ ( $32$ seeds per cell). This is a comparison of references, not a sensitivity analysis of CODuel.

reference	held-out profiles	Composite	Quirky	held-out topics	development
best fixed pair per user	76	66	68	74	73
known-profile pair	75	–	–	73	72
type-library reference	23	32	33	54	52

Appendix

Table 20: Diagnostic references, by roster: benchmark score, the share of random pairs’ excess regret removed (higher is better). The best fixed pair per user shows, for the whole session, the pair with the lowest regret on that user’s questions in hindsight. The known-profile pair shows the best fixed pair of the user’s profile, chosen on the development roster for the development profiles and cross-fitted for the held-out profiles; it is not defined on Composite and Quirky. The type-library reference knows each development profile’s average choice probabilities and utilities and infers the current user’s profile from their choices.

variant	increase in excess regret [95% CI]
CODuel (excess regret)	970.8 $\pm$ 4.7
no transfer (base learner only)	+449.7 [+434.7, +464.6]
greedy pair rule instead of lookahead	+5.2 [-7.9, +18.2]
latent-bandit Thompson sampling instead of lookahead	+213.0 [+196.8, +229.1]
latent-bandit UCB instead of lookahead	+128.5 [+109.4, +147.6]
no evidence gate	+8.1 [-4.9, +21.1]

Appendix

Table 21: One-component changes of CODuel on the development roster (32 seeds,

b=24

). First row: excess regret of CODuel (mean

\pm

s.e.). Other rows: paired increase in excess regret when the component is changed, with a 95%

t

interval over seeds; positive values mean the component of CODuel lowers regret. Every other parameter is unchanged.

variant	held-out profiles	Composite	Quirky
greedy pair rule instead of lookahead	+46.6 [+33.7, +59.5]	-18.0 [-33.4, -2.7]	-1.7 [-16.6, +13.2]
no evidence gate	+3.1 [-9.8, +16.0]	-19.1 [-33.4, -4.8]	-7.8 [-24.7, +9.1]
one tight type instead of the mixture (no spread between users)	+359.0 [+344.1, +374.0]	+120.6 [+102.2, +139.0]	+147.2 [+127.3, +167.0]
one Gaussian with learned covariance instead of the mixture	+84.4 [+71.5, +97.2]	-28.1 [-46.8, -9.3]	-29.7 [-43.7, -15.7]
no population (recalibration kept)	+370.4 [+359.1, +381.7]	+368.5 [+346.2, +390.8]	+384.1 [+367.9, +400.3]
no recalibration ( $A=I_{K}$ )	+33.7 [+21.6, +45.9]	+29.7 [+17.2, +42.2]	+50.7 [+34.9, +66.5]

Appendix

Table 22: One-component changes of CODuel on the three main rosters (64 seeds on the held-out profiles, 32 on Composite and Quirky): paired increase in excess regret when the component is changed, with a 95%

t

interval over seeds (roster fixed); positive values mean the component of CODuel lowers regret; negative values favor the variant.

setting	held-out profiles	Composite	Quirky
gate threshold $z=0.25$ (paper $0.5$ )	-0.1 [-0.6, +0.5]	+0.6 [-0.1, +1.4]	+0.0 [-0.8, +0.9]
gate threshold $z=1$	-0.3 [-0.9, +0.2]	+0.5 [-0.2, +1.3]	+0.2 [-0.4, +0.7]
gate minimum $n_{\min}=4$ (paper $8$ )	+0.1 [-0.5, +0.6]	+0.6 [-0.1, +1.2]	+0.3 [-0.3, +0.9]
gate minimum $n_{\min}=16$	-0.7 [-1.2, -0.1]	+0.5 [-0.2, +1.1]	+0.1 [-0.5, +0.7]
components $M_{\max}=8$ (paper $16$ )	-0.4 [-0.9, +0.1]	+0.6 [-0.1, +1.3]	+0.0 [-0.6, +0.7]
components $M_{\max}=32$	-0.3 [-0.9, +0.3]	+0.6 [+0.0, +1.2]	+0.2 [-0.4, +0.8]

Appendix

Table 23: Sensitivity of CODuel to its settings, changed one at a time on the three main rosters (64 seeds on the held-out profiles, 32 on Composite and Quirky; never used to choose a setting): paired change in benchmark score against CODuel with the paper’s settings, with a 95%

t

interval over seeds; negative values mean the paper’s setting is better. In 42 of the 42 cells the changed CODuel still has lower excess regret than the closest comparison method on that roster (significantly in 42).

control	excess of no transfer	change with CODuel	95% CI
simulator, null population	864.8	-0.3%	[-1.4%, +0.9%] †
tutoring, scrambled roster	1453.7	+0.9%	[-0.5%, +2.2%]

Appendix

Table 24: Negative controls: paired change in excess regret of CODuel relative to its no-transfer ablation, as a share of the ablation’s excess, where no structure is shared (simulator null population,

\beta=4

b=20

; scrambled development roster,

b=24

; 32 seeds). Negative values mean transfer costs regret. The interval resamples seeds and users after the fact; † : seeds only.

control	policy	excess regret	change vs. no transfer [95% CI]
simulator null ( $\beta=4$ , $b=20$ )	CODuel	867.0 $\pm$ 3.5	-0.3% [-1.4%, +0.9%]
	without gate	973.6 $\pm$ 4.5	-12.6% [-13.6%, -11.5%]
	no population	868.4 $\pm$ 4.4	-0.4% [-1.4%, +0.6%]
scrambled roster ( $b=24$ )	CODuel	1441.2 $\pm$ 4.6	+0.9% [+0.0%, +1.7%]
	without gate	1463.5 $\pm$ 3.3	-0.7% [-1.5%, +0.1%]
	no population	1436.3 $\pm$ 3.5	+1.2% [+0.4%, +2.0%]

Appendix

Table 25: The evidence gate on the two negative controls (32 seeds). Last column: paired gain over no transfer as a share of its excess regret, with a 95%

t

interval over seeds; negative values mean the policy costs regret where there is nothing to transfer.

pair rule (profiles’ tables known)	excess regret
greedy	961.8 $\pm$ 2.8
one-step lookahead	954.6 $\pm$ 2.9
latent-bandit Thompson sampling	1197.6 $\pm$ 3.8
latent-bandit UCB	1114.6 $\pm$ 1.6

Appendix

Table 26: Pair rules given the development profiles’ choice probabilities and utilities (a diagnostic, not an admissible policy), development roster, 32 seeds.

LLM Personalization

LLM personalization adapts language models to users’ preferences, histories, expertise, and changing goals while balancing usefulness, privacy, and control.

See all 130 papers

Sep 9, 2026

HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization

Personalized language models aim to adapt responses to individual users, whose preferences are often latent and revealed gradually through interaction. Existing training-free methods rely on stored histories or retrieved memories, but they often struggle to reconcile long- term preferences with short-term topic-specific needs. To address this issue, we propose HyperTrace, a training-free framework that formulates online personalization as latent preference tracing. HyperTrace maintains interpretable natural-language hypotheses over short-term intent and long-term preferences, and updates them through an SMC-style reweight process using an LLM-based surrogate choice model. By updating these hypotheses across turns and sessions, HyperTrace enables personalization without parameter updates. Experiments on PRISM and PersonaMem-v2 show that HyperTrace improves response alignment, preference prediction, and profile consistency over strong online baselines, demonstrating the effectiveness of tracing latent user preferences for robust personalization. Code and scripts are available in the repository: https://github.com/jiseshen/HyperTrace.

Jianzhi Shen, Keyu Mao, Minghao Shao +7

Johns Hopkins University · Institute of Science Tokyo · NYU Tandon +1

LLM Personalization Personalized Language Models User Preference Modeling

Ask PDF

Oct 17, 2025

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.

Yizhuo Chen, Xin Liu, Ruijie Wang +7

University of Illinois Urbana-Champaign · Amazon · University of Notre Dame

LLM Personalization Personalized Language Models Personalized Language Model Alignment Language Model Post-Training User Preference Modeling

Ask PDF

May 20, 2026

APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings

Typical LLM responses tend to follow a default style, even though users often have distinct preferences regarding tone, verbosity, and formality that they do not explicitly state in their prompts. Evaluating whether personalization methods can adapt to these implicit preferences is challenging, since users typically provide prompts rather than reference responses, style preferences are not factually verifiable, and reference-free LLM judges may conflate personalization with general response quality. To address these challenges, we introduce the Arbitrary Preference Mapping (APM) benchmark, which decouples user attributes (e.g. enthusiastic) from response principles (e.g. persuasive) via a hidden, randomized mapping

\mathbf{C}

that maps user attributes to preferences about response traits. Because

\mathbf{C}

carries no semantic content and is resampled across runs, models cannot exploit stereotypical associations and must infer preferences from conversation history. Using this unbiased evaluation methodology, we adapt retrieval-augmented, prompt-optimization, and routing personalization methods and evaluate them on Llama-3.1-8B and Qwen-3.5-27B. Our results show that routing is the most reliable approach, while RAG only improves with the stronger base LLM, and soft prompt optimization fails to improve significantly over a non-personalized baseline. Our extensive evaluation reveals that in this realistic setting, personalization remains challenging, but our adapted methods show promise.

Philipp Spohn, Leander Girrbach, Zeynep Akata

Technical University of Munich, Helmholtz Munich · Munich Center for Machine Learning (MCML)

LLM Evaluation LLM Personalization Language Model Generation Evaluation Automatic Prompt Optimization Personalized Language Models LLM Routing+1

Ask PDF

Abstract

LLM Personalization

Explore similar work

HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization

POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

APM: Evaluating Style Personalization in LLMs with Arbitrary Preference Mappings