Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user's preferences. Importantly, language models are already able to make use of this information in context. After observing a user's follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user's follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model's behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation.
Figures & tables
Figure 1: Direct Learning from User Interactions via Self-Distillation. From multi-turn user conversations, we obtain several interactions (x,y,o) that consist of the conversation history x , the model’s response y , and the subsequent user message o . By conditioning on the user’s follow-up, we form the hindsight policy and compare it to the original policy, producing token-level advantages that reinforce or penalize parts of the model’s original response. In this example, the user’s follow-up requests a more direct answer, leading to penalizing filler tokens and reinforcing the answer.
Figure 2: Example of the token-level advantages ( 1 ) where the user complains with o = “I said YES or NO only” after the assistant failed to follow the instruction.
User:
<conversation history including most recent user prompt> x <hindsight context> The following is a future user message. Use this to guide your answer to the user prompt: o
Assistant:
<assistant completion> y
Table 1: Chat template for the hindsight policy πθ(y∣x,o) . We recover the usual template for the base policy πθ(y∣x) when removing “ <hindsight context> […]” from the user prompt.
Figure 3: Training on real-world user conversations, SDPO improves general alignment and instruction-following performance across benchmarks, without degrading other capabilities. Results for Qwen3-8B before and after training on 14,000 real-world user conversations.
Alpaca Eval 2.0 (LC Winrate)
IFEval (Prompt-Level)
ArenaHard-v2 (Hard Prompt)
ArenaHard-v2 (Creative Writing)
MMLU-Pro (Chain-of-Thought)
Qwen3-4B
37.9
81.9
9.0
8.0
58.1
SDPO
↑ 46.1
↑ 83.2
↓ 7.8
7.9
58.0
Qwen3-8B
49.3
83.9
14.0
13.7
62.5
SDPO
↑ 51.9
↑ 85.0
↑ 15.5
↑ 16.2
↑ 63.3
Olmo3-7B-SFT
34.3
80.2
2.4
1.4
23.7
SDPO
↑ 35.2
↑ 80.6
2.4
1.4
↑ 24.0
Table 2: Across model families and model sizes, SDPO improves alignment and instruction-following without degrading other capabilities. A mild exception is Qwen3-4B, where SDPO significantly increases performance on AlpacaEval 2.0 (+8.2%) and IFEval (+1.3%) but decreases performance on the math and coding tasks of ArenaHard-v2 (-1.2%). We only show arrows when performance changed by more than 0.1 percentage points.
AlpacaEval 2.0 (LC Winrate)
IFEval (Prompt-Level)
ArenaHard-v2 (Hard Prompt)
ArenaHard-v2 (Creative Writing)
MMLU-Pro (Chain-of-Thought)
Qwen3-8B
49.3
83.9
14.0
13.7
62.5
SDPO (WildFeedback)
51.9
85.0
15.5
16.2
63.3
SDPO (WildChat)
↑ 50.7
↑ 84.5
↓ 13.4
↑ 14.0
62.4
Table 3: Even on fully uncurated user interactions, SDPO still yields improvements in alignment and instruction-following and only mild degradation in math and coding. Results for training on a randomly sampled subset of 14,000 conversations (about 50,000 interactions (x,y,o) ) from WildChat. For comparison, we here include the results for training on WildFeedback from Table 2 .
AlpacaEval 2.0 (LC Winrate)
IFEval (Prompt-Level)
ArenaHard-v2 (Hard Prompt)
ArenaHard-v2 (Creative Writing)
MMLU-Pro (Chain-of-Thought)
Qwen3-4B
37.9
81.9
9.0
8.0
58.1
SFT on Dataset
↓ 18.9
↓ 73.2
↓ 3.1
↓ 2.6
↓ 51.2
Table 4: SDPO is fundamentally different from SFT. As a sanity check, we fine-tune Qwen3-4B on the assistant completions in WildFeedback using standard supervised fine-tuning.
Figure 4: SDPO adapts online to changing user preferences. The user’s preference about how the model should respond is flipped to its opposite after the first 250 interactions. SDPO with Qwen3-4B is able to quickly reverse the learned behavior.
Figure 5: SDPO rapidly personalizes to individual users from interaction alone. Win rate of SDPO against its base model (Qwen3-4B) for a user that prefers concise, casual, and beginner-friendly model responses.
Figure 6: SDPO enables continual personalization without catastrophic forgetting. We train a single Qwen3-8B model online with SDPO for 1500 user interactions, during which three complementary user preferences are introduced sequentially (500 interactions each). Each curve reports the win rate of the current model with respect to the model checkpoint at the time the corresponding preference was introduced, thereby isolating the relative improvement along that specific preference dimension. Earlier preferences remain strong as new ones are learned, indicating that SDPO can accumulate complementary preferences over time without forgetting previously learned behavior. Shaded regions indicate standard error over 256 evaluation prompts.
Figure 7: When user follow-ups are relevant to the model’s completion, we observe strong positive and negative SDPO advantages. We visualize the advantages with Qwen3-8B for user follow-ups that carry relevant information about the model’s answer, such as requests for revisions, positive reactions, or other relevant feedback. Below: Example (second line in the heatmap), where the user requests a more formal rewrite of the assistant’s draft ( “Rewrite in a formal, professional tone” ). Informal expressions have large negative advantages. Accordingly, SDPO adapts the policy to respond more formally when the user needs help with work emails in the future.
Figure 8: When user follow-ups are unrelated to the model’s response, SDPO advantages are close to zero. We visualize the advantages with Qwen3-8B for user follow-ups that are unrelated to the model’s generation. Below: Following the request to write an email, the user responds with “What is 27 × 4?” , which is unrelated to the original request. The advantages are close to zero everywhere, which means that SDPO does not meaningfully update the policy from these interactions.
Table 5: Hyperparameters used for SDPO in each setup. Note that the hyperparameters of SDPO in Section 4.1 were kept the same across all models. The learning rate was chosen by sweeping over {1,2,3,5}×10−6 for Qwen3-4B and then fixing the setup for all models. For the SFT checkpoint in Table 4 , we similarly swept over {1,2,3,5}×10−6 with best results for 2×10−6 . In Section 4.2 , SDPO appeared insensitive to hyperparameter choices in early experiments (especially, learning rate), and were fixed to the setup below without additional systematic tuning.
TruthfulQA (MC1) Acc ± StdErr
HellaSwag Acc ± StdErr
CommonsenseQA Acc ± StdErr
Qwen3-8B
0.366 ± 0.0169
0.5717 ± 0.0049
0.7846 ± 0.0118
SDPO
0.3647 ± 0.0169
0.5710 ± 0.0049
0.7871 ± 0.0117
Appendix
Table 6: SDPO preserves performance on pre-training benchmarks. We additionally evaluate SDPO for Qwen3-8B on the standard pre-training benchmarks TruthfulQA ( Lin et al., 2022 ) , HellaSwag ( Zellers et al., 2019 ) , and CommonsenseQA ( Talmor et al., 2019 ) .
Figure 9: Additional personalization results from Section 4.2 with Qwen3-4B. The win rate is computed against the base model and judged by Qwen3-8B. The In-Context Oracle baseline is obtained by prompting Qwen3-4B directly with the desired writing style.
Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective for improving average helpfulness, this can suppress the natural variation of human responses across languages, tasks, and dialogue settings. We study this problem as conditional human-distribution alignment: models should match the human response distribution appropriate to the current interaction context, rather than a universal response style. We introduce PolyAlign, a distribution-aware alignment framework that organizes bilingual interaction data into bucket-specific human reference distributions defined by language, interaction track, response family, and length. PolyAlign combines Bucket-Aware SFT, which balances optimization across heterogeneous buckets, with Human-Distribution Preference Optimization (HDPO), which regularizes preference learning using critic-estimated distance to bucket-specific human support. Across a bilingual evaluation suite covering English and Chinese single- and multi-turn settings, PolyAlign improves conditional naturalness and distributional faithfulness while preserving competitive task utility. The results suggest that post-training should move beyond global alignment objectives toward interaction-aware alignment with human response distributions.
Persona drift remains a central challenge for personalized language models, as user profiles evolve over long interactions rather than remain permanently fixed. Models must therefore revise persistent persona states when preferences genuinely change, while avoiding updates driven by transient, ambiguous, or unresolved observations. We propose CORE, which separates turn-local evidence from persistent persona-state revision and selectively updates grounded user preferences through uncertainty-aware belief revision. We also introduce PERSIST, a held-out post-anchor benchmark for persona-state robustness under sequential interaction stress, covering ambiguity, conflict, and controlled social influence. Across ALOE, PersonaChat, and PERSIST, CORE improves personalized alignment and robustness, with complementary gains in normalized closed-slot state fidelity. Human evaluation and mechanistic controls further support explicit update control beyond stronger generation or persistent memory alone.
Youyuan Zhang, Siyuan Li, Fangming Liu +1
Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, China · Huazhong University of Science and Technology, China
User feedback offers natural supervision for persistent LLM improvement, but a single message may support multiple behavioral changes with different scopes of generalization. We introduce SLIFT, a selective self-learning framework built on a task-relative view of user feedback. SLIFT decomposes each feedback message into atomic components and interprets each component relative to the original task as Fix, Spec, or Null: requirements for task validity, compatible condition-specific refinements, or content with no reliable positive update direction. To incorporate each change at the appropriate scope, SLIFT trains two complementary LoRA adapters on a shared frozen backbone: a Generalist that consolidates Fix requirements into default behavior through feedback-conditioned self-distillation, and a Specialist that observes only the task and Generalist response to supply residual guidance for applicable, unmet Spec refinements. Null components induce no positive update. Across backbones, SLIFT achieves strong performance on both MemoryBench and WildFB, with targeted analyses further examining its underlying mechanisms. We release our code at https://anonymous.4open.science/r/SLIFT.
Xuancheng Li, Haitao Li, Yujia Zhou +5
Tencent · DCST, Tsinghua University Beijing, China