User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction

Yuren Hao; Shuhaib Mehri; ChengXiang Zhai; Dilek Hakkani-Tür

cs.CLMar 21, 2026

User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction

Authors: Yuren Hao, Shuhaib Mehri, ChengXiang Zhai, Dilek Hakkani-Tür

Organizations: University of Illinois at Urbana-Champaign

Abstract

Large language models are increasingly used as personal assistants, yet most lack a persistent user model, forcing users to repeatedly restate preferences across sessions. We propose Vector-Adapted Retrieval Scoring (VARS), a pipeline-agnostic, frozen-backbone framework that represents each user with long-term and short-term vectors in a shared preference space and uses these vectors to bias retrieval scoring over structured preference memory. The vectors are updated online from weak scalar rewards from users' feedback, enabling personalization without per-user fine-tuning. We evaluate on MultiSessionCollab, an online multi-session collaboration benchmark with rich user preference profiles, across math and code tasks. Under frozen backbones, the main benefit of user-aware retrieval is improved interaction efficiency rather than large gains in raw task accuracy: our full VARS agent achieves the strongest overall performance, matches a strong Reflection baseline in task success, and reduces timeout rate and user effort. The learned long-term vectors also align with cross-user preference overlap, while short-term vectors capture session-specific adaptation, supporting the interpretability of the dual-vector design. Code, model, and data are available at https://github.com/YurenHao0426/VARS.

Figures & tables

Figure 1: VARS architecture. $M_{\text{ext}}$ extracts preference cards; a dual user vector $z_{U,t}^{\mathrm{eff}}=\beta_{L}z_{U}^{(L)}+\beta_{S}z_{U,t}^{(S)}$ biases reranking; a keyword-based reward $\hat{r}_{t}$ drives REINFORCE updates.

Figure 2: Main results summary on MultiSessionCollab .

Method	Success (%) $\uparrow$	Timeout (%) $\downarrow$	User tokens $\downarrow$
VARS	55.2	26.4	193.6
Reflection	54.4	28.8	207.5
Vanilla	54.3	29.2	232.9
Contextual	52.4	31.4	213.7
RAG	52.0	44.3	188.4
All-memory	50.9	33.4	226.8

Table 1: Main results on MultiSessionCollab (

60

profiles

\times

60

sessions). Bold = best; underline = second best.

Metric	Reflection	RAG+Vec	$\Delta$	$p$ (paired $t$ )
Success (%)	54.4	55.2	$+$ 0.9	0.276
Timeout (%)	28.8	26.4	$-$ 2.4	0.046 ∗
User tokens	207.5	193.6	$-$ 13.9	0.021 ∗

Table 2: VARS vs. Reflection (

60

profiles, one-sided paired

t

-tests). ∗

p<0.05

Metric	Reflection	RAG	VARS
Succ. / 1k user tokens $\uparrow$	2.61	2.80	2.83
Late success (sess. 30–59) $\uparrow$	51.8%	51.7%	54.1%
Quick resolution ( $\leq$ 4 turns) $\uparrow$	3.5%	3.4%	3.9%
Zero-enf. success $\uparrow$	60.2%	57.1%	60.6%
First-turn enforced $\downarrow$	7.9%	7.2%	7.1%
Profiles improved $\uparrow$	31.6%	—	36.7%

Table 3: Additional metrics.

\uparrow

\downarrow

= higher/lower is better. Bold = best.

Vector	Spearman $\rho$ ( $p$ )	Quintile $p$
$z^{(L)}$ (long-term)	0.040 (0.093)	0.006
$z^{(S)}$ (short-term)	0.010 (0.681)	0.586
Combined	0.046 (0.054)	0.003

Table 4: User-vector cosine vs. preference Jaccard overlap (

\binom{60}{2}

pairs).

Method	Success (%) $\uparrow$	Timeout (%) $\downarrow$	User tok $\downarrow$	Non-TO utok $\downarrow$
VARS (full)	74.0	10.7	474.5	166.5
$z^{(S)}$ only	72.7	12.7	846.8	185.7
$z^{(L)}$ only	71.3	14.7	1347.4	168.1
No vector	70.0	14.0	1566.1	183.7

Table 5: Dual-vector ablation on math-hard (

10

profiles

\times

15

sessions) with GPT-4o-mini. Higher absolute success reflects the stronger backbone. Non-TO utok = non-timeout sessions only.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Count
GPT-5.1 synthesized positives	156,489
LMSYS-Chat-1M Zheng et al. (2023)	133,086
WildChat Zhao et al. (2024)	108,677
Retry / recovery labels	68,778
Alpaca-Cleaned Wang et al. (2023)	42,013
ShareGPT-Vicuna 2 2 2 https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered	33,025

Appendix

Table 6: Training data sources for the preference extractor.

ID	When	Then
cs_008	providing error handling examples	always use specific exception types, never bare except
ds_001	discussing ML concepts	include mathematical formulation alongside intuitive explanation
ds_006	writing documentation	be concise, avoid marketing language, focus on usage
vb_002	complex algorithms/mathematics	provide detailed step-by-step derivation with intermediate results
ms_002	discussing statistics/probability	start with intuition and real-world interpretation before formulas
rf_003	providing a final answer	put the answer first, then explanation

Appendix

Table 7: Revealed preferences for user_14b429db during the 20-session MultiSessionCollab run.

Metric	ONLINE-USER	VANILLA
AvgSatS2	$0.9500$	$0.7250$
ViolRateS2(too_long)	$0.1667$	$0.4167$
ViolRateS2(no_bullets)	$0.0000$	$0.5000$
Recall@k_S2(SHORT)	$0.6667$	$0.0000$
Recall@k_S2(BULLETS)	$0.8333$	$0.0000$
Recall@k_S2(LANG)	$1.0000$	$0.0000$

Appendix

Table 8: Offline Session 2 results (no complaints, 6 personas). ONLINE-USER substantially improves satisfaction and reduces violations compared to a VANILLA LLM without preference memory.

Metric	ONLINE-USER	VANILLA
AvgSatS2	$0.9750$	$0.8625$
ViolRateS2(too_long)	$0.0833$	$0.1667$
ViolRateS2(no_bullets)	$0.0000$	$0.2917$
Recall@k_S2(SHORT)	$0.6667$	$0.0000$
Recall@k_S2(BULLETS)	$0.7917$	$0.0000$
Recall@k_S2(LANG)	$1.0000$	$0.0000$

Appendix

Table 9: Offline Session 2 results with longer lifetimes (10 sessions per user). ONLINE-USER continues to outperform VANILLA in satisfaction, violation rates, and preference-memory recall.

Setting
$\eta_{L}=1.0\times 10^{-2}$ : LR for long-term vector $\mathbf{z}^{(L)}_{U}$
$\eta_{S}=5.0\times 10^{-2}$ : LR for short-term vector $\mathbf{z}^{(S)}_{U,t}$
$\lambda=0.1$ : Exponential decay for $\mathbf{z}^{(S)}_{U,t}$ between turns
$\alpha=0.05$ : EMA coefficient for reward baseline $b_{U}$
$\tau=1.0$ : Policy temperature in the softmax over memories

Appendix

Table 10: Core REINFORCE update parameters. All experiments use the code values.

Setting
$\beta_{L}=2.0$ : Weight of long-term vector in $\mathbf{z}^{\text{eff}}_{U,t}$
$\beta_{S}=5.0$ : Weight of short-term vector in $\mathbf{z}^{\text{eff}}_{U,t}$
$k$ ( item_dim ) $=256$ : Dimensionality of item and user vectors

Appendix

Table 11: User-vector weighting and item-space dimension.

Setting
dense_topk $=64$ : Number of candidates from dense retrieval
rerank_topk $=3$ : Final memories fed to the LLM

Appendix

Table 12: Retrieval parameters used in our RAG and VARS implementations.

Case	Condition	$g_{t}$
Retrieval failure	$\hat{r}_{t}<-0.5$ , low similarity	$0.9$
LLM failure	$\hat{r}_{t}<-0.5$ , high similarity	$0.2$
Good + retrieval helped	$\hat{r}_{t}>0.5$ , similarity $>0.4$	$0.6$
Good + no retrieval help	$\hat{r}_{t}>0.5$ , similarity $\leq 0.4$	$0.3$
Default	neutral / other cases	$0.5$

Appendix

Table 13: Gating cases for the retrieval-specific advantage

A^{\text{retr}}_{t}=g_{t}(\hat{r}_{t}-b_{U})

. “Similarity” refers to the similarity between chosen memories and the current query; in the “Good” cases we use a numerical threshold of

0.4

Config	$\\|z^{(L)}\\|$	$\Delta\%$	$\cos$ to base
Baseline	0.288	—	1.000
Half neg keywords	0.288	0%	1.000
Noisy neg keywords	0.603	+109%	0.184
Clip $[-0.5,0.5]$	0.374	+30%	0.490
Dampen thresh $0.1$	0.288	0%	1.000
Dampen thresh $0.3$	0.288	0%	1.000

Appendix

Table 14: Reward sensitivity. Baseline uses the default keyword list and clip range

[-1,1]

with dampening threshold

0.2

Config	$\\|z^{(L)}\\|$	$\Delta\%$	$\cos$ to base
Fixed $g=0.5$	0.771	+168%	0.444
Fixed $g=1.0$	1.593	+454%	0.410
Thresh $(0.1,0.4)$	0.288	0%	1.000
Thresh $(0.3,0.6)$	0.288	0%	1.000

Appendix

Table 15: Gating sensitivity. Baseline uses the heuristic gating with similarity thresholds

(0.2,0.5)

User Preference Modeling

User preference modeling infers what people value from choices, behavior, and feedback to personalize recommendations, predictions, and AI actions.

See all 269 papers

Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
Apr 1, 2026Gyuseok Lee, Wonbin Kweon, Zhenrui Yue +3Personalized Language Models Personalized Language Model Alignment
University of Illinois Urbana-Champaign · Korea University
Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use
Oct 22, 2026Yijiang River Dong, Zexue He, Cheng Yu +9Memory-Augmented Language Models Reward Modeling
University of Cambridge · Stanford University · Technical University of Munich +2
Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory
May 28, 2026ZhiShu Jiang, Haibo Liu, Xin Shen +6Query Reformulation Information Retrieval
Baidu Inc. · The University of Queensland

Apr 1, 2026J/K move · Enter open · S save

Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization

Gyuseok Lee, Wonbin Kweon, Zhenrui Yue, SeongKu Kang+2

University of Illinois Urbana-Champaign · Korea University

Personalized Language Models Personalized Language Model Alignment User Preference Modeling Reward Modeling

Open paper AskPDF arXiv

Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment. Our code is available at https://github.com/Gyu-Seok-Lee/VRF_COLM26.

Abstract

User Preference Modeling

Explore similar work

Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization

Memory-Aware Reward Modeling: Learning Human Preferences for LLM Memory Use

Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory