Aligning Language Models from User Interactions

Thomas Kleine Buening; Jonas Hübotter; Barna Pásztor; Idan Shenfeld; Giorgia Ramponi; Andreas Krause

cs.CLFeb 18, 2026

Aligning Language Models from User Interactions

Authors: Thomas Kleine Buening, Jonas Hübotter, Barna Pásztor, Idan Shenfeld, Giorgia Ramponi, Andreas Krause

Organizations: ETH Zurich · MIT · University of Zurich

Abstract

Multi-turn user interactions are among the most abundant data produced by language models, yet we lack effective methods to learn from them. While typically discarded, these interactions often contain useful information: follow-up user messages may indicate that a response was incorrect, failed to follow an instruction, or did not align with the user's preferences. Importantly, language models are already able to make use of this information in context. After observing a user's follow-up, the same model is often able to revise its behavior. We leverage this ability to propose a principled and scalable method for learning directly from user interactions through self-distillation. By conditioning the model on the user's follow-up message and comparing the resulting token distribution with the original policy, we obtain a target for updating the policy that captures how the model's behavior changes in hindsight. We then distill this hindsight distribution back into the current policy. Remarkably, we show that training on real-world user conversations from WildChat improves language models across standard alignment and instruction-following benchmarks, without regressing other capabilities. The same mechanism enables personalization, allowing models to continually adapt to individual users through interaction without explicit feedback. Our results demonstrate that raw user interactions that arise naturally during deployment enable alignment, personalization, and continual adaptation.

Figures & tables

Figure 1: Direct Learning from User Interactions via Self-Distillation. From multi-turn user conversations, we obtain several interactions $(x,y,o)$ that consist of the conversation history $x$ , the model’s response $y$ , and the subsequent user message $o$ . By conditioning on the user’s follow-up, we form the hindsight policy and compare it to the original policy, producing token-level advantages that reinforce or penalize parts of the model’s original response. In this example, the user’s follow-up requests a more direct answer, leading to penalizing filler tokens and reinforcing the answer.

Figure 2: Example of the token-level advantages ( 1 ) where the user complains with $o$ = “I said YES or NO only” after the assistant failed to follow the instruction.

User:	<conversation history including most recent user prompt> $x$ <hindsight context> The following is a future user message. Use this to guide your answer to the user prompt: $o$
Assistant:	<assistant completion> $y$

Table 1: Chat template for the hindsight policy

\pi_{\theta}(y\mid x,o)

. We recover the usual template for the base policy

\pi_{\theta}(y\mid x)

when removing “ <hindsight context> […]” from the user prompt.

Figure 3: Training on real-world user conversations, SDPO improves general alignment and instruction-following performance across benchmarks, without degrading other capabilities. Results for Qwen3-8B before and after training on 14,000 real-world user conversations.

	Alpaca Eval 2.0 (LC Winrate)	IFEval (Prompt-Level)	ArenaHard-v2 (Hard Prompt)	ArenaHard-v2 (Creative Writing)	MMLU-Pro (Chain-of-Thought)
Qwen3-4B	37.9	81.9	9.0	8.0	58.1
SDPO	$\boldsymbol{\uparrow}$ 46.1	$\boldsymbol{\uparrow}$ 83.2	$\boldsymbol{\downarrow}$ 7.8	7.9	58.0
Qwen3-8B	49.3	83.9	14.0	13.7	62.5
SDPO	$\boldsymbol{\uparrow}$ 51.9	$\boldsymbol{\uparrow}$ 85.0	$\boldsymbol{\uparrow}$ 15.5	$\boldsymbol{\uparrow}$ 16.2	$\boldsymbol{\uparrow}$ 63.3
Olmo3-7B-SFT	34.3	80.2	2.4	1.4	23.7
SDPO	$\boldsymbol{\uparrow}$ 35.2	$\boldsymbol{\uparrow}$ 80.6	2.4	1.4	$\boldsymbol{\uparrow}$ 24.0

Table 2: Across model families and model sizes, SDPO improves alignment and instruction-following without degrading other capabilities. A mild exception is Qwen3-4B, where SDPO significantly increases performance on AlpacaEval 2.0 (+8.2%) and IFEval (+1.3%) but decreases performance on the math and coding tasks of ArenaHard-v2 (-1.2%). We only show arrows when performance changed by more than 0.1 percentage points.

	AlpacaEval 2.0 (LC Winrate)	IFEval (Prompt-Level)	ArenaHard-v2 (Hard Prompt)	ArenaHard-v2 (Creative Writing)	MMLU-Pro (Chain-of-Thought)
Qwen3-8B	49.3	83.9	14.0	13.7	62.5
SDPO (WildFeedback)	51.9	85.0	15.5	16.2	63.3
SDPO (WildChat)	$\boldsymbol{\uparrow}$ 50.7	$\boldsymbol{\uparrow}$ 84.5	$\boldsymbol{\downarrow}$ 13.4	$\boldsymbol{\uparrow}$ 14.0	62.4

Table 3: Even on fully uncurated user interactions, SDPO still yields improvements in alignment and instruction-following and only mild degradation in math and coding. Results for training on a randomly sampled subset of 14,000 conversations (about 50,000 interactions

(x,y,o)

) from WildChat. For comparison, we here include the results for training on WildFeedback from Table 2 .

	AlpacaEval 2.0 (LC Winrate)	IFEval (Prompt-Level)	ArenaHard-v2 (Hard Prompt)	ArenaHard-v2 (Creative Writing)	MMLU-Pro (Chain-of-Thought)
Qwen3-4B	37.9	81.9	9.0	8.0	58.1
SFT on Dataset	$\boldsymbol{\downarrow}$ 18.9	$\boldsymbol{\downarrow}$ 73.2	$\boldsymbol{\downarrow}$ 3.1	$\boldsymbol{\downarrow}$ 2.6	$\boldsymbol{\downarrow}$ 51.2

Table 4: SDPO is fundamentally different from SFT. As a sanity check, we fine-tune Qwen3-4B on the assistant completions in WildFeedback using standard supervised fine-tuning.

Figure 4: SDPO adapts online to changing user preferences. The user’s preference about how the model should respond is flipped to its opposite after the first 250 interactions. SDPO with Qwen3-4B is able to quickly reverse the learned behavior.

Figure 5: SDPO rapidly personalizes to individual users from interaction alone. Win rate of SDPO against its base model (Qwen3-4B) for a user that prefers concise, casual, and beginner-friendly model responses.

Figure 6: SDPO enables continual personalization without catastrophic forgetting. We train a single Qwen3-8B model online with SDPO for 1500 user interactions, during which three complementary user preferences are introduced sequentially (500 interactions each). Each curve reports the win rate of the current model with respect to the model checkpoint at the time the corresponding preference was introduced, thereby isolating the relative improvement along that specific preference dimension. Earlier preferences remain strong as new ones are learned, indicating that SDPO can accumulate complementary preferences over time without forgetting previously learned behavior. Shaded regions indicate standard error over 256 evaluation prompts.

Panel of Figure 7 — Figure 7: When user follow-ups are relevant to the model’s completion, we observe strong positive and negative SDPO advantages. We visualize the advantages with Qwen3-8B for user follow-ups that carry relevant information about the model’s answer, such as requests for revisions, positive reactions, or other relevant feedback. Below: Example (second line in the heatmap), where the user requests a more formal rewrite of the assistant’s draft ( “Rewrite in a formal, professional tone” ). Informal expressions have large negative advantages. Accordingly, SDPO adapts the policy to respond more formally when the user needs help with work emails in the future.

Panel of Figure 8 — Figure 8: When user follow-ups are unrelated to the model’s response, SDPO advantages are close to zero. We visualize the advantages with Qwen3-8B for user follow-ups that are unrelated to the model’s generation. Below: Following the request to write an email, the user responds with “What is 27 $\times$ 4?” , which is unrelated to the original request. The advantages are close to zero everywhere, which means that SDPO does not meaningfully update the policy from these interactions.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Hyperparameter	Section 4.1 ( Figures 3 , 2 , 3 and 4 )	Section 4.2 ( Figures 5 and 4 )	Section 4.2 ( Figure 6 )
Models	Qwen3-4B, Qwen3-8B, Olmo3-7B-Instruct-SFT, Olmo3-7B-Instruct-DPO	Qwen3-4B	Qwen3-8B
Max prompt length	2048	1024	2048
Max compl. length	2048	258	2048
Learning rate	$2\times 10^{-6}$	$5\times 10^{-6}$	$5\times 10^{-6}$
Batch size	32	16	32
Epochs	2	1	1

Appendix

Table 5: Hyperparameters used for SDPO in each setup. Note that the hyperparameters of SDPO in Section 4.1 were kept the same across all models. The learning rate was chosen by sweeping over

\smash{\{1,2,3,5\}\times 10^{-6}}

for Qwen3-4B and then fixing the setup for all models. For the SFT checkpoint in Table 4 , we similarly swept over

\smash{\{1,2,3,5\}\times 10^{-6}}

with best results for

\smash{2\times 10^{-6}}

. In Section 4.2 , SDPO appeared insensitive to hyperparameter choices in early experiments (especially, learning rate), and were fixed to the setup below without additional systematic tuning.

	TruthfulQA (MC1) Acc $\pm$ StdErr	HellaSwag Acc $\pm$ StdErr	CommonsenseQA Acc $\pm$ StdErr
Qwen3-8B	0.366 $\pm$ 0.0169	0.5717 $\pm$ 0.0049	0.7846 $\pm$ 0.0118
SDPO	0.3647 $\pm$ 0.0169	0.5710 $\pm$ 0.0049	0.7871 $\pm$ 0.0117

Appendix

Table 6: SDPO preserves performance on pre-training benchmarks. We additionally evaluate SDPO for Qwen3-8B on the standard pre-training benchmarks TruthfulQA ( Lin et al., 2022 ) , HellaSwag ( Zellers et al., 2019 ) , and CommonsenseQA ( Talmor et al., 2019 ) .

Panel of Figure 9 — Figure 9: Additional personalization results from Section 4.2 with Qwen3-4B. The win rate is computed against the base model and judged by Qwen3-8B. The In-Context Oracle baseline is obtained by prompting Qwen3-4B directly with the desired writing style.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

PolyAlign: Conditional Human-Distribution Alignment
Jun 11, 2026L. D. M. S. Sai Teja, Ufaq Khan, Sathira Silva +2Multilingual Language Models LLM Alignment
NIT Silchar, India · MBZUAI, Abu Dhabi, UAE
Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue
Sep 11, 2026Youyuan Zhang, Siyuan Li, Fangming Liu +1Belief Revision LLM Alignment
Harbin Institute of Technology, Shenzhen, China · Peng Cheng Laboratory, China · Huazhong University of Science and Technology, China
Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models
Aug 10, 2026Xuancheng Li, Haitao Li, Yujia Zhou +5Continual Learning for LLMs Self-Training for Language Models
Tencent · DCST, Tsinghua University Beijing, China

Jun 11, 2026J/K move · Enter open · S save

PolyAlign: Conditional Human-Distribution Alignment

L. D. M. S. Sai Teja, Ufaq Khan, Sathira Silva, Xiao Wu+1

NIT Silchar, India · MBZUAI, Abu Dhabi, UAE

Multilingual Language Models LLM Alignment Preference Optimization Language Model Post-Training

Open paper AskPDF arXiv

Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective for improving average helpfulness, this can suppress the natural variation of human responses across languages, tasks, and dialogue settings. We study this problem as conditional human-distribution alignment: models should match the human response distribution appropriate to the current interaction context, rather than a universal response style. We introduce PolyAlign, a distribution-aware alignment framework that organizes bilingual interaction data into bucket-specific human reference distributions defined by language, interaction track, response family, and length. PolyAlign combines Bucket-Aware SFT, which balances optimization across heterogeneous buckets, with Human-Distribution Preference Optimization (HDPO), which regularizes preference learning using critic-estimated distance to bucket-specific human support. Across a bilingual evaluation suite covering English and Chinese single- and multi-turn settings, PolyAlign improves conditional naturalness and distributional faithfulness while preserving competitive task utility. The results suggest that post-training should move beyond global alignment objectives toward interaction-aware alignment with human response distributions.

Abstract

LLM Alignment

Explore similar work

PolyAlign: Conditional Human-Distribution Alignment

Toward Robust Personalized Alignment for LLMs: Mitigating Persona Drift in Multi-Turn Dialogue

Different Feedback, Different Updates: Selective Self-Learning from User Interactions for Large Language Models