cs.AINov 2, 2026

UPSHIFT: User Preference Steering from Human-AI Interaction Feedback Traces

Authors: Jingping Wang, Zhanhao Li, Shao Zhang, Shuyue Hu

Organizations: Shanghai Jiaotong University (sjtu.edu.cn) · Shanghai Artificial Intelligence Laboratory (shlab.org.cn)

Abstract

Personalizing large language model assistants requires adapting their responses to individual users, including the selection of content, reasoning approach, level of detail, and form of presentation. Such preferences are often latent, becoming visible only when users respond to an initial answer and request revisions. We investigate whether these feedback trajectories can be converted into a reusable personalization signal without explicit preference profiles or labels, prompt-time access to interaction histories, or model parameter updates. We introduce UPSHIFT (User Preference Steering from Human-AI Interaction Feedback Traces), a training-free method that compares the hidden representations of initial and feedback-conditioned responses within each session, averages the resulting trajectory displacements across a user's sessions, and injects the aggregated preference direction when answering a new query. The method is motivated by the implicit-preference hypothesis: although an individual conversational transition may reflect topic progression or task-specific correction, changes that recur across sessions retain a component that captures how the user wants responses to be adjusted. We validate the preference-representation capability of UPSHIFT on PrefEval. In evaluations with 40 simulated users, every tested configuration improves preference alignment over the unsteered baseline on held-out topics for both Qwen3-8B and Qwen3-32B; the best configuration increases macro-averaged Elo from 1364.0 to 1605.2 and from 1341.9 to 1591.8, respectively. Matched-user and positive-versus-negative controls further demonstrate that these gains arise from the user-specific preference direction and its learned orientation. These results show that interaction histories can provide compact, reusable supervision for lightweight personalization.

Figures & tables

Explore similar work

CardsList
  1. HyperTrace: Hypothesis-Based Preference Tracing for Online LLM Personalization

    Sep 9, 2026Jianzhi Shen, Keyu Mao, Minghao Shao +7LLM PersonalizationPersonalized Language Models

  2. POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

    Oct 17, 2025Yizhuo Chen, Xin Liu, Ruijie Wang +7LLM PersonalizationPersonalized Language Models

  3. Online LLM Personalization from Pairwise Preferences Across Users

    Oct 12, 2026Romain Puech, Georgy Salakhutdinov, Dimitris BertsimasEducational QALLM Personalization