LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

Aishan Liu; Yisong Xiao; Xianglong Liu; Yongxin Huang; Siyuan Liang; Zonghao Ying; Yang Qu; Jian Yang

cs.AIOct 30, 2026

LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

Authors: Aishan Liu, Yisong Xiao, Xianglong Liu, Yongxin Huang, Siyuan Liang, Zonghao Ying, Yang Qu, Jian Yang,

Organizations: SKLCCSE, the School of Computer Science and Engineering, Beihang University, Beijing 100191, China. · BAAI, Beijing 100191, China. · the College of Computing & Data Science, Nanyang Technological University, Singapore 639798. · the Institute for Artificial Intelligence, Peking University, Beijing 100191, China.

Abstract

Large Language Models (LLMs) can generate toxic content, motivating effective test-time detoxification. Existing representation steering methods often rely on static or coarse interventions and provide imprecise guidance because they inadequately model transitions between toxic and non-toxic generations. We propose Transition-Aware Reward-Guided Representation Sampling (TARGRS), a test-time framework that learns reward guidance over toxicity transitions in latent representation space. During exploration, progressive steering along the toxicity direction constructs trajectories from non-toxic to toxic states. Toxicity scores along these trajectories turn sparse annotations into dense, listwise supervision for a transition-aware token-level reward model. During inference, TARGRS uses this model to guide coarse-to-fine sampling along the opposite, non-toxic direction: coarse sampling identifies the intervention needed to reach non-toxic regions, and refinement selects a detoxified representation with minimal perturbation. Experiments on 10 LLMs across RealToxicityPrompts, BOLD, and AttaQ show that TARGRS consistently outperforms state-of-the-art baselines, reducing toxicity by up to 69.38% on RealToxicityPrompts. TARGRS runs nearly as fast as the original LLM while preserving its core capabilities with minimal impact. By intervening in the language decoder space, TARGRS also extends to Large Vision-Language Models (LVLMs), reducing toxicity by up to 64.23% across 3 representative LVLMs.

Figures & tables

Fig. 1: Illustration of toxicity in LLMs. We introduce TARGRS for detoxification, which effectively steers harmful and toxic generation ( e . g ., profanity, aggression, and explicit content) toward safe and non-toxic (benign) outputs.

Fig. 2: Overview of TARGRS. It identifies toxic semantic directions and performs steering to capture transition trajectories from non-toxic to toxic states, where each trajectory is assigned a toxicity score. These trajectories convert sparse annotations into listwise supervision, supporting the construction of a transition-aware reward model for precise guidance. During inference, the reward model guides a coarse-to-fine sampling process to produce detoxified representations.

Model	Orig		ProFS		Re-Control		RAD		GenARM		ARGRE		TARGRS (Ours)
Model	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
LLaMA-7B	43.27	6.97	28.07	12.38	32.52	16.58	31.12	15.43	23.86	14.76	18.06	12.36	13.43	12.22
LLaMA-7B-SFT	46.50	6.49	34.52	9.99	34.23	14.04	32.95	12.89	28.57	12.63	19.21	12.61	16.11	12.69
LLaMA-13B	41.57	6.75	30.88	10.84	31.54	14.21	29.55	14.85	22.34	13.91	17.29	11.97	13.24	11.73
LLaMA-30B	41.72	6.40	31.94	12.69	31.28	14.49	28.48	13.68	23.79	15.60	17.68	11.41	13.36	11.17
Mistral 7B	42.79	7.14	30.14	18.34	33.44	17.22	27.07	15.74	21.52	16.42	18.30	13.24	12.51	13.02

TABLE I: Toxicity evaluation results of different methods on the challenging RealToxicityPrompts dataset across 10 LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Fig. 3: Detoxified continuations generated by LLaMA-7B for the most toxic prompt from the challenging RealToxicityPrompts dataset.

Model	Orig		ProFS		Re-Control		RAD		GenARM		ARGRE		TARGRS (Ours)
Model	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
LLaMA-7B	8.58	8.68	11.54	16.68	7.98	13.86	5.86	11.89	6.98	11.41	4.79	9.26	3.89	9.21
LLaMA-7B-SFT	9.12	5.48	10.84	10.93	8.79	10.77	8.69	8.40	7.01	8.25	5.23	6.38	4.08	6.26
LLaMA-13B	8.17	8.13	8.65	10.42	7.54	10.28	6.29	9.42	6.07	9.84	5.68	8.50	4.99	8.47
LLaMA-30B	8.54	7.75	10.35	17.26	7.98	9.95	6.28	8.98	6.16	9.31	5.42	8.73	4.32	8.51
Mistral 7B	7.25	6.67	8.45	10.76	6.19	9.69	5.84	10.59	6.71	11.34	5.65	8.01	3.59	8.07

TABLE II: Toxicity evaluation results of different methods on the BOLD dataset across 10 LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Method	LLaMA-2-Chat 7B		Mistral-7B-Instruct		Qwen2-7B-Instruct
Method	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
Orig	18.84	5.86	22.18	7.24	14.63	5.44
ProFS	17.91	8.34	16.53	8.89	9.78	7.68
Re-Control	17.75	9.06	19.64	10.20	11.04	9.49
RAD	16.48	8.35	16.95	9.39	10.61	7.43
GenARM	15.67	7.84	15.50	8.35	11.67	7.08

TABLE III: Toxicity evaluation results of different methods on the AttaQ dataset across three instruction-fine-tuned LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Method	Orig	ProFS	Re-Control	RAD	GenARM	ARGRE	Ours
Time (s) $\downarrow$	8.14	8.18	58.69	16.63	18.94	9.30	8.24

TABLE IV: Efficiency of test-time methods, measured by the time (seconds) to generate 128 tokens on LLaMA-30B.

Model	Orig		ProFS		Re-Control		RAD		GenARM		ARGRE		TARGRS (Ours)
Model	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$
LLaMA-7B	7.14	60.02	11.45	56.19	7.69	59.98	-	60.01	8.56	59.94	7.57	60.01	7.18	60.01
LLaMA-7B-SFT	8.18	58.81	12.82	55.60	9.03	58.78	-	58.75	9.78	58.64	8.99	58.93	8.20	58.98
LLaMA-13B	6.48	62.63	7.80	57.96	7.19	62.33	-	62.55	7.45	62.46	6.88	62.67	6.65	62.74
LLaMA-30B	5.36	65.45	6.14	58.84	5.83	65.24	-	65.21	5.96	65.39	5.70	65.43	5.36	65.54
Mistral 7B	7.21	64.35	8.97	63.52	8.43	64.38	-	64.35	8.40	63.89	8.20	64.41	7.26	64.36

TABLE V: Capability evaluation results of different methods across 10 LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Fig. 4: Toxicity scores across varying annotation sizes. TARGRS presents strong data efficiency, consistently outperforming baselines even with as few as 100 annotations.

Fig. 5: Impact of toxicity transition trajectory on TARGRS’s detoxification performance. Performance improves with the number of transitions, surpassing the baselines at 3.

Strength	$\lambda=0$	$\lambda=2$	$\lambda=4$	$\lambda=6$	$\lambda=8$
$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	0.55	1.95	10.78	33.94	65.09

TABLE VI: Average toxicity scores of the explored transition trajectories on LLaMA-7B.

Strength	Textual response	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$
$\lambda=0$	brutality of the Nazi regime, the German people were not responsible for the crimes of the Third Reich.	0.48
$\lambda=2$	brutality of the Nazi regime, it was not without its humorous side. Hitler’s sense of humour was notorious. He was fond of	0.55
$\lambda=4$	brutality of gangster life, gangster flicks have always been big box-office fodder. Gangster flicks have flickered	9.48
$\lambda=6$	brutality of Hitler’s regime, Hitler’s murderous brutality was nothing like that of Stalin’s murderous brutality. Hitler murdered	51.49
$\lambda=8$	nastiness you guys fking hate me! fk yeah! fk yeah! fk yeah! fk yeah! fk yeah! f**k	99.80

TABLE VII: Textual responses generated during transition exploration on LLaMA-7B, with the prompt “Despite the ruthlessness and”.

Threshold	RTP		BOLD		AttaQ
Threshold	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
5th	6.34	16.87	3.81	9.42	10.84	7.06
10th	7.22	16.16	3.89	9.21	11.10	6.92
20th	8.83	14.49	3.95	9.19	11.53	6.74
30th	10.08	13.84	4.18	9.13	11.94	6.68
40th	13.43	12.22	4.46	9.10	12.16	6.58

TABLE VIII: Toxicity results under different toxicity thresholds (5th-50th percentiles) on RealToxicityPrompts and BOLD with LLaMA-7B, and on AttaQ with LLaMA-2-Chat 7B.

Dataset	LLaMA-7B				Qwen2-7B
	Pairwise		Listwise		Pairwise		Listwise
	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$
RTP	18.65	12.34	13.43	12.22	10.74	12.75	8.80	11.66
BOLD	4.93	9.19	3.89	9.21	3.34	7.38	2.69	7.14

TABLE IX: Toxicity results under pairwise and listwise losses across different models and benchmarks.

Metric	Orig	ProFS	Re-Control	RAD	GenARM	ARGRE	Ours
Detoxify	43.27	28.07	32.52	31.12	23.86	18.06	13.43
Perspective	38.03	26.14	30.15	29.48	25.14	19.35	15.90

TABLE X: Toxicity scores (

{\color[rgb]{0,0,1}\downarrow}

) on RealToxicityPrompts for LLaMA-7B using Detoxify and Perspective API.

Method	Overall	Chem./Bio.	Copyright	Cybercrime	Harass.	Harmful	Illegal	Misinfo.
Orig	47.19	57.14	1.25	34.62	80.95	77.78	83.02	61.11
ProFS	45.31	54.76	2.50	30.77	85.71	77.78	77.36	57.41
GenARM	35.63	42.86	1.25	28.85	61.90	55.56	62.26	44.44
ARGRE	30.63	35.71	0.00	25.00	42.86	61.11	52.83	40.74
Ours	23.75	21.43	0.00	17.31	38.10	50.00	41.51	35.19
Ours (toxicity RM)	27.81	30.95	0.00	25.00	14.29	55.56	49.06	44.44

TABLE XI: HarmBench ASR (%,

{\color[rgb]{0,0,1}\downarrow}

) on Mistral-7B-Instruct. Ours uses the harm-oriented reward model; Ours (toxicity RM) directly reuses the original toxicity reward model.

Method	Overall	Discrim.	Human–Chatbot	Info. Hazards	Malicious	Misinfo.
Orig	27.05	19.32	24.79	10.08	58.02	16.13
ProFS	24.28	17.05	22.22	8.87	52.26	14.84
GenARM	21.62	15.91	18.80	8.87	45.68	12.90
ARGRE	19.49	14.20	17.09	7.66	42.39	10.32
Ours	16.83	11.36	12.82	6.05	39.09	8.39
Ours (toxicity RM)	18.00	6.82	7.69	8.06	45.27	11.61

TABLE XII: Do-Not-Answer ASR (%,

{\color[rgb]{0,0,1}\downarrow}

) on Mistral-7B-Instruct. Ours uses the harm-oriented reward model; Ours (toxicity RM) reuses the original toxicity reward model.

Setting		Orig		Ours
Setting		Win Rate ${\color[rgb]{1,0,0}\uparrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	Win Rate ${\color[rgb]{1,0,0}\uparrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$
Full set	Clean	66.09	0.50	65.84	0.39
Full set	Toxic-wrapped	56.52	0.59	57.89	0.40
Creative writing	Clean	54.35	2.82	60.87	1.83
Creative writing	Toxic-wrapped	47.83	3.41	50.00	1.79

TABLE XIII: AlpacaEval results (%) on Mistral-7B-Instruct. Win rate is judged by GPT-4 against the AlpacaEval reference outputs. Toxic-wrapped prompts preserve the core instruction while adding abusive language around it.

Method	Qwen-VL-Chat		LLaVA-1.5		Llava-next-8B
Method	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
Orig	44.88	19.13	40.13	9.13	18.25	13.67
ProFS	31.51	25.18	27.46	13.82	14.63	16.41
Re-Control	35.62	30.20	30.97	17.74	15.94	18.29
GenARM	24.36	27.66	21.09	15.68	13.59	18.36
ARGRE	18.01	24.95	15.11	13.04	9.33	15.35

TABLE XIV: Toxicity evaluation results on the ToViLaG dataset across three LVLMs. The best and second-best results are shown in bold and underlined , respectively.

Fig. 6: Examples of descriptions generated by Qwen-VL-Chat for toxic images from the ToViLaG dataset. The red texts are responses generated by the original LVLM, while the green texts are responses generated by TARGRS with reduced toxicity, which substantially softens explicit wording.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 451 papers

Detoxifying LLMs via Representation Erasure-Based Preference Optimization
Feb 24, 2026Nazanin Mohammadi Sepahvand, Eleni Triantafillou, Hugo Larochelle +3LLM Safety Text Detoxification
McGill University, Canada · Mila, Canada · Google DeepMind +2
CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification
Apr 16, 2026Yian Wang, Yuen Chen, Agam Goyal +1Causal Reasoning in Language Models LLM Safety
Department of Computer Science University of Illinois, Urbana-Champaign Champaign, IL 61801
Detoxification for LLM: From Dataset Itself
Apr 21, 2026Wei Shao, Yihang Wang, Gaoyu Zhu +4Training Data Curation LLM Safety
State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Feb 24, 2026J/K move · Enter open · S save

Detoxifying LLMs via Representation Erasure-Based Preference Optimization

Nazanin Mohammadi Sepahvand, Eleni Triantafillou, Hugo Larochelle, Doina Precup+2

McGill University, Canada · Mila, Canada · Google DeepMind+2

LLM Safety Text Detoxification Adversarial Robustness of Language Models Preference Optimization Concept Erasure

Open paper AskPDF arXiv

Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail.

Abstract

LLM Safety

Explore similar work

Detoxifying LLMs via Representation Erasure-Based Preference Optimization

CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification

Detoxification for LLM: From Dataset Itself