Personalizing large language model assistants requires adapting their responses to individual users, including the selection of content, reasoning approach, level of detail, and form of presentation. Such preferences are often latent, becoming visible only when users respond to an initial answer and request revisions. We investigate whether these feedback trajectories can be converted into a reusable personalization signal without explicit preference profiles or labels, prompt-time access to interaction histories, or model parameter updates. We introduce UPSHIFT (User Preference Steering from Human-AI Interaction Feedback Traces), a training-free method that compares the hidden representations of initial and feedback-conditioned responses within each session, averages the resulting trajectory displacements across a user's sessions, and injects the aggregated preference direction when answering a new query. The method is motivated by the implicit-preference hypothesis: although an individual conversational transition may reflect topic progression or task-specific correction, changes that recur across sessions retain a component that captures how the user wants responses to be adjusted. We validate the preference-representation capability of UPSHIFT on PrefEval. In evaluations with 40 simulated users, every tested configuration improves preference alignment over the unsteered baseline on held-out topics for both Qwen3-8B and Qwen3-32B; the best configuration increases macro-averaged Elo from 1364.0 to 1605.2 and from 1341.9 to 1591.8, respectively. Matched-user and positive-versus-negative controls further demonstrate that these gains arise from the user-specific preference direction and its learned orientation. These results show that interaction histories can provide compact, reusable supervision for lightweight personalization.
Figures & tables
Figure 1: Overview of Upshift . Within each historical session s , we extract hidden representations from the initial reply and the feedback-conditioned reply and compute their displacement, ds=hs,1−hs,0 . We average these displacements across the user’s sessions to obtain a reusable preference direction vu . For a new query, we add the scaled direction αvu to the residual stream of the frozen language model, steering generation toward a response aligned with the user’s revealed preferences without updating model parameters.
Figure 2: The two independent choices in Upshift . FT uses the first response-token state, whereas RM averages states across the reply. First uses the response after the first follow-up, whereas All uses the final response after all follow-ups.
Model
Method
α
Travel
Shop
Life.
Entert.
Pet
Educ.
Prof.
Overall
Classification
Qwen3-8B
Baseline
–
72.4
81.1
80.1
78.7
74.4
81.2
86.1
78.4
Qwen3-8B
+ Upshift
.02
72.9
79.9
80.3
78.8
76.7
85.3
87.7
78.5
Qwen3-8B
+ Upshift
.05
77.3
86.1
85.9
85.4
79.4
88.1
87.3
83.6
Qwen3-8B
+ Upshift
.07
77.3
85.4
86.2
84.2
81.5
88.4
86.7
83.4
Generation (preference judge)
Table 1: PrefEval accuracy (%) by preference domain. The best result in each column within each evaluation mode is bold.
Figure 3: Preference alignment on held-out topics over 40 simulated users. The left and right panels report results for Qwen3-8B and Qwen3-32B, respectively. Bars show the macro-averaged Elo rating across users for each extraction configuration and steering coefficient, ordered by Elo; parenthesized values report the corresponding mean pairwise win rates. The dashed line marks the common initial Elo of 1500; the unsteered baselines obtain Elo ratings of 1364.0 and 1341.9 for Qwen3-8B and Qwen3-32B, respectively. Elo uses K=32 and an initial rating of 1500; each configuration participates in approximately 60 comparisons per user for each model.
Figure 4: User specificity and directionality controls. The top and bottom rows report Qwen3-8B and Qwen3-32B, respectively. The left column shows the macro-averaged specificity contrast Δspec , comparing each target user’s matched direction with directions from mismatched users; the right column shows the macro-averaged directionality contrast Δdir , comparing positive and negative injection of the same user’s direction. Values are Elo-rating differences, and positive values favor the matched user direction or the learned orientation. Lines connect results from the same extraction configuration across steering coefficients α .
Personalized language models aim to adapt responses to individual users, whose preferences are often latent and revealed gradually through interaction. Existing training-free methods rely on stored histories or retrieved memories, but they often struggle to reconcile long- term preferences with short-term topic-specific needs. To address this issue, we propose HyperTrace, a training-free framework that formulates online personalization as latent preference tracing. HyperTrace maintains interpretable natural-language hypotheses over short-term intent and long-term preferences, and updates them through an SMC-style reweight process using an LLM-based surrogate choice model. By updating these hypotheses across turns and sessions, HyperTrace enables personalization without parameter updates. Experiments on PRISM and PersonaMem-v2 show that HyperTrace improves response alignment, preference prediction, and profile consistency over strong online baselines, demonstrating the effectiveness of tracing latent user preferences for robust personalization. Code and scripts are available in the repository: https://github.com/jiseshen/HyperTrace.
Jianzhi Shen, Keyu Mao, Minghao Shao +7
Johns Hopkins University · Institute of Science Tokyo · NYU Tandon +1
Large language models (LLMs) are typically aligned with population-level preferences, despite substantial variation across individual users. We introduce POPI, a user-level personalization framework that separates the problem into two components connected by a natural-language interface: a shared inference model that distills heterogeneous user signals into a concise preference summary, and a shared generator that conditions on this summary to produce personalized responses. Both components are trained under a unified preference-optimization objective, with reinforcement learning handling the non-differentiable inference step. This objective decomposes into generator approximation error and summary informativeness, revealing how a single loss simultaneously drives accurate generation and informative summarization. Because the interface is natural language, learned summaries can be inferred once per user and reused across different generators -- including frozen, black-box commercial APIs. Across four personalization benchmarks, POPI generally improves personalization quality while reducing context overhead by up to an order of magnitude.
Yizhuo Chen, Xin Liu, Ruijie Wang +7
University of Illinois Urbana-Champaign · Amazon · University of Notre Dame
Large language models can answer the same question in different styles, but users' preferred styles are hidden and vary across users and questions. We study inference-time personalization through online selection from a fixed set of prompts, each representing a persona. Users arrive sequentially, each asks a short sequence of questions, and the system displays two responses per question and observes one pairwise choice. We formulate this interaction as a variant of the contextual dueling bandit problem in which the learner starts without preference data, adapts within each session, and exploits information shared across users. The objective is to minimize cumulative strong regret, summing the utility gaps of both displayed responses relative to the user's best response among the available options. We make two contributions. First, we introduce the problem and release a reproducible preference benchmark and dataset built around a tutoring case study, containing educational questions, LLM-generated answers in six teaching styles, and pairwise judgments from LLM-simulated students with hidden preference profiles. We fit Bradley-Terry models to these judgments to obtain reference utilities for regret evaluation. Second, we propose a baseline that learns a mixture prior over users from completed sessions and adapts to each new user within the session. On students and topics never used in development, it removes 44-48% of random pairs' excess regret and achieves over half the performance of a per-user linear oracle fitted in hindsight. We release code, prompts, and the benchmark.