Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

Mikhail Krasitskii; Alexander Gelbukh; Olga Kolesnikova; Grigori Sidorov

cs.CLAug 16, 2026

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

Authors: Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

Organizations: Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC)

Abstract

Reinforcement learning with human feedback (RLHF) aligns LLMs with human preferences, improving summarization fluency and safety, but causes sentiment drift: overly neutral summaries stripped of emotional nuance. We diagnose why RL acts as a sentiment neutralizer and present Policy Attribution, a framework using gradient and logit decomposition to trace drift to reward model (RM) signals and KL (Kullback-Leibler) penalty. Sentiment drift reflects a strategic bias toward "low-risk" tokens maximizing expected rewards under preference uncertainty (Stiennon et al., 2020; Gao, Schulman, and Hilton, 2023). On Reddit TL;DR and CNN/DailyMail, RLHF summaries get higher rewards but show 30-40% lower sentiment variance. Cross-lingual analysis across eight languages shows language-independent drift, with morphologically richer languages more suppressed (Krasitskii et al., 2026). We propose and validate a sentiment-aware regularization technique reducing drift by 18-22% without harming summary quality. The code and toolkit will be public.

Figures & tables

Hyperparameter	Value
Learning Rate	$1\times 10^{-5}$
Batch Size	64
PPO Epochs	4
KL Coefficient ( $\beta$ )	$\{0.05,0.1,0.2\}$

Table 1: Hyperparameters used for RLHF training.

Strategy	SV $\downarrow$	ROUGE-L $\uparrow$
Extractive	0.142	32.7
Abstractive	0.208	38.9
Hybrid	0.113	36.1

Table 2: Effect of summarization strategy on sentiment (SV).

Language	SV $\downarrow$	ROUGE-L $\uparrow$
English	0.121	37.5
Arabic	0.187	35.2
Finnish	0.163	34.8
French	0.134	36.9
German	0.148	36.1
Hungarian	0.175	33.7

Table 3: Sentiment drift across languages after RLHF optimization.

Figure 1: Sentiment variance as a function of the KL coefficient $\beta$ . Higher $\beta$ values induce stronger neutrality bias.

Figure 2: Distribution of sentiment drift across eight languages. Morphologically rich languages (Finnish, Hungarian) exhibit more pronounced suppression.

$\beta$	Sent. Variance $\downarrow$	ROUGE-L $\uparrow$
0.05	0.184	36.2
0.10	0.121	36.8
0.20	0.073	37.1

Table 4: Effect of KL regularization on sentiment drift and summarization quality.

Component	Pos. Attr.	Neg. Attr.
Reward Model	0.42	0.58
KL Penalty	0.18	0.82
PPO Clipping	0.25	0.75

Table 5: Attribution mass assigned to components of the RL objective.

Figure 3: Token-level attribution scores aggregated over sentiment-bearing tokens. Negative values indicate suppression by the KL penalty.

Method	$\beta$	SV $\downarrow$	ROUGE-L $\uparrow$
SFT Baseline	–	0.210	35.8
DPO	0.1	0.168	36.1
DPO	0.2	0.152	36.4
DPO	0.5	0.141	36.7
PPO	0.05	0.184	36.2
PPO	0.10	0.121	36.8

Table 6: Comparison of sentiment drift between PPO and DPO alignment methods.

Method	SV $\downarrow$	JSD $\downarrow$	R-L $\uparrow$	Drift Reduction
SFT Baseline	0.210	0.082	35.8	–
RLHF (Std)	0.121	0.145	36.8	–
SA-KL ( $\gamma=0.3$ )	0.158	0.112	36.5	18.2%
SA-KL ( $\gamma=0.5$ )	0.167	0.105	36.3	21.7%

Table 7: Compact version of mitigation results.

Figure 4: Qualitative examples of sentiment attenuation. Emotionally charged expressions ( highlighted ) are replaced with neutral paraphrases after RLHF. (a) Reddit TL;DR example 1; (b) Reddit TL;DR example 2; (c) CNN/DailyMail example 1; (d) CNN/DailyMail example 2.

RL from Human Feedback

Reinforcement learning from human feedback (RLHF) trains AI systems to better match human preferences using comparisons, corrections, and other feedback.

See all 128 papers

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance
Jun 8, 2026Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova +1LLM Alignment Text Summarization
Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico City, Mexico
The Neutral Mask: How Alignment Training Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model
Jun 8, 2026Wendy K. TamLLM Alignment LLM Interpretability
Professor and Stevenson Chair in the Departments of Political Science, Computer Science, Biomedical Informatics, and the Law School at Vanderbilt University, and an affiliate of the National Center for Supercomputing Applications at the University of Illinois at Urbana-Champaign
Reinforcement Learning from Human Feedback: A Statistical Perspective
Apr 2, 2026Pangpang Liu, Chengchun Shi, Will Wei SunRL from Human Feedback LLM Alignment
Department of Biostatistics, Yale University. · Department of Statistics, London School of Economics and Political Science. · Mitch Daniels School of Business, Purdue University.

Jun 8, 2026J/K move · Enter open · S save

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance

Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC), Mexico City, Mexico

LLM Alignment Text Summarization RL from Human Feedback

Open paper AskPDF arXiv

Reinforcement Learning from Human Feedback (RLHF) has significantly improved the quality and fluency of large language models in text summarization. However, its impact on affective properties remains insufficiently understood. In this work, we study sentiment drift, a systematic shift toward neutral sentiment in RLHF-based summarization outputs compared to source texts. We conduct extensive experiments across multiple datasets, model architectures, and eight languages to analyze how alignment objectives influence sentiment preservation. Our results show that sentiment drift is a consistent phenomenon that becomes stronger with increased KL regularization strength, indicating a trade-off between alignment stability and affective fidelity. To explain this behavior, we introduce a Policy Attribution framework that decomposes the RLHF objective and quantifies the contribution of its components. Our analysis reveals that KL regularization is the primary driver of sentiment suppression across all settings. Based on these findings, we propose a sentiment-aware modification of the KL regularization term, which selectively reduces constraints on sentiment-bearing tokens. Empirical results demonstrate that this approach mitigates sentiment drift while maintaining summarization quality. Overall, our findings highlight a fundamental limitation of current alignment methods: while they improve factual consistency and safety, they may unintentionally suppress emotional expressiveness. This motivates the development of alignment strategies that explicitly account for affective preservation.

Abstract

RL from Human Feedback

Explore similar work

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance

The Neutral Mask: How Alignment Training Provides Shallow Alignment while Leaving Partisan Structure Intact in a Large Language Model

Reinforcement Learning from Human Feedback: A Statistical Perspective