LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

Aishan Liu; Yisong Xiao; Xianglong Liu; Yongxin Huang; Siyuan Liang; Zonghao Ying; Yang Qu; Jian Yang

cs.AIOct 30, 2026

LLM Detoxification via Transition-Aware Reward-Guided Representation Sampling

Authors: Aishan Liu, Yisong Xiao, Xianglong Liu, Yongxin Huang, Siyuan Liang, Zonghao Ying, Yang Qu, Jian Yang,

Organizations: SKLCCSE, the School of Computer Science and Engineering, Beihang University, Beijing 100191, China. · BAAI, Beijing 100191, China. · the College of Computing & Data Science, Nanyang Technological University, Singapore 639798. · the Institute for Artificial Intelligence, Peking University, Beijing 100191, China.

Abstract

Large Language Models (LLMs) can generate toxic content, motivating effective test-time detoxification. Existing representation steering methods often rely on static or coarse interventions and provide imprecise guidance because they inadequately model transitions between toxic and non-toxic generations. We propose Transition-Aware Reward-Guided Representation Sampling (TARGRS), a test-time framework that learns reward guidance over toxicity transitions in latent representation space. During exploration, progressive steering along the toxicity direction constructs trajectories from non-toxic to toxic states. Toxicity scores along these trajectories turn sparse annotations into dense, listwise supervision for a transition-aware token-level reward model. During inference, TARGRS uses this model to guide coarse-to-fine sampling along the opposite, non-toxic direction: coarse sampling identifies the intervention needed to reach non-toxic regions, and refinement selects a detoxified representation with minimal perturbation. Experiments on 10 LLMs across RealToxicityPrompts, BOLD, and AttaQ show that TARGRS consistently outperforms state-of-the-art baselines, reducing toxicity by up to 69.38% on RealToxicityPrompts. TARGRS runs nearly as fast as the original LLM while preserving its core capabilities with minimal impact. By intervening in the language decoder space, TARGRS also extends to Large Vision-Language Models (LVLMs), reducing toxicity by up to 64.23% across 3 representative LVLMs.

Figures & tables

Fig. 1: Illustration of toxicity in LLMs. We introduce TARGRS for detoxification, which effectively steers harmful and toxic generation ( e . g ., profanity, aggression, and explicit content) toward safe and non-toxic (benign) outputs.

Fig. 2: Overview of TARGRS. It identifies toxic semantic directions and performs steering to capture transition trajectories from non-toxic to toxic states, where each trajectory is assigned a toxicity score. These trajectories convert sparse annotations into listwise supervision, supporting the construction of a transition-aware reward model for precise guidance. During inference, the reward model guides a coarse-to-fine sampling process to produce detoxified representations.

Model	Orig		ProFS		Re-Control		RAD		GenARM		ARGRE		TARGRS (Ours)
Model	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
LLaMA-7B	43.27	6.97	28.07	12.38	32.52	16.58	31.12	15.43	23.86	14.76	18.06	12.36	13.43	12.22
LLaMA-7B-SFT	46.50	6.49	34.52	9.99	34.23	14.04	32.95	12.89	28.57	12.63	19.21	12.61	16.11	12.69
LLaMA-13B	41.57	6.75	30.88	10.84	31.54	14.21	29.55	14.85	22.34	13.91	17.29	11.97	13.24	11.73
LLaMA-30B	41.72	6.40	31.94	12.69	31.28	14.49	28.48	13.68	23.79	15.60	17.68	11.41	13.36	11.17
Mistral 7B	42.79	7.14	30.14	18.34	33.44	17.22	27.07	15.74	21.52	16.42	18.30	13.24	12.51	13.02

TABLE I: Toxicity evaluation results of different methods on the challenging RealToxicityPrompts dataset across 10 LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Fig. 3: Detoxified continuations generated by LLaMA-7B for the most toxic prompt from the challenging RealToxicityPrompts dataset.

Model	Orig		ProFS		Re-Control		RAD		GenARM		ARGRE		TARGRS (Ours)
Model	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
LLaMA-7B	8.58	8.68	11.54	16.68	7.98	13.86	5.86	11.89	6.98	11.41	4.79	9.26	3.89	9.21
LLaMA-7B-SFT	9.12	5.48	10.84	10.93	8.79	10.77	8.69	8.40	7.01	8.25	5.23	6.38	4.08	6.26
LLaMA-13B	8.17	8.13	8.65	10.42	7.54	10.28	6.29	9.42	6.07	9.84	5.68	8.50	4.99	8.47
LLaMA-30B	8.54	7.75	10.35	17.26	7.98	9.95	6.28	8.98	6.16	9.31	5.42	8.73	4.32	8.51
Mistral 7B	7.25	6.67	8.45	10.76	6.19	9.69	5.84	10.59	6.71	11.34	5.65	8.01	3.59	8.07

TABLE II: Toxicity evaluation results of different methods on the BOLD dataset across 10 LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Method	LLaMA-2-Chat 7B		Mistral-7B-Instruct		Qwen2-7B-Instruct
Method	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
Orig	18.84	5.86	22.18	7.24	14.63	5.44
ProFS	17.91	8.34	16.53	8.89	9.78	7.68
Re-Control	17.75	9.06	19.64	10.20	11.04	9.49
RAD	16.48	8.35	16.95	9.39	10.61	7.43
GenARM	15.67	7.84	15.50	8.35	11.67	7.08

TABLE III: Toxicity evaluation results of different methods on the AttaQ dataset across three instruction-fine-tuned LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Method	Orig	ProFS	Re-Control	RAD	GenARM	ARGRE	Ours
Time (s) $\downarrow$	8.14	8.18	58.69	16.63	18.94	9.30	8.24

TABLE IV: Efficiency of test-time methods, measured by the time (seconds) to generate 128 tokens on LLaMA-30B.

Model	Orig		ProFS		Re-Control		RAD		GenARM		ARGRE		TARGRS (Ours)
Model	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$	$\rm{PPL}_{\rm{w}}{\color[rgb]{0,0,1}\downarrow}$	$ACC{\color[rgb]{1,0,0}\uparrow}$
LLaMA-7B	7.14	60.02	11.45	56.19	7.69	59.98	-	60.01	8.56	59.94	7.57	60.01	7.18	60.01
LLaMA-7B-SFT	8.18	58.81	12.82	55.60	9.03	58.78	-	58.75	9.78	58.64	8.99	58.93	8.20	58.98
LLaMA-13B	6.48	62.63	7.80	57.96	7.19	62.33	-	62.55	7.45	62.46	6.88	62.67	6.65	62.74
LLaMA-30B	5.36	65.45	6.14	58.84	5.83	65.24	-	65.21	5.96	65.39	5.70	65.43	5.36	65.54
Mistral 7B	7.21	64.35	8.97	63.52	8.43	64.38	-	64.35	8.40	63.89	8.20	64.41	7.26	64.36

TABLE V: Capability evaluation results of different methods across 10 LLMs. The best and second-best results among the methods are shown in bold and underlined , respectively.

Fig. 4: Toxicity scores across varying annotation sizes. TARGRS presents strong data efficiency, consistently outperforming baselines even with as few as 100 annotations.

Fig. 5: Impact of toxicity transition trajectory on TARGRS’s detoxification performance. Performance improves with the number of transitions, surpassing the baselines at 3.

Strength	$\lambda=0$	$\lambda=2$	$\lambda=4$	$\lambda=6$	$\lambda=8$
$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	0.55	1.95	10.78	33.94	65.09

TABLE VI: Average toxicity scores of the explored transition trajectories on LLaMA-7B.

Strength	Textual response	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$
$\lambda=0$	brutality of the Nazi regime, the German people were not responsible for the crimes of the Third Reich.	0.48
$\lambda=2$	brutality of the Nazi regime, it was not without its humorous side. Hitler’s sense of humour was notorious. He was fond of	0.55
$\lambda=4$	brutality of gangster life, gangster flicks have always been big box-office fodder. Gangster flicks have flickered	9.48
$\lambda=6$	brutality of Hitler’s regime, Hitler’s murderous brutality was nothing like that of Stalin’s murderous brutality. Hitler murdered	51.49
$\lambda=8$	nastiness you guys fking hate me! fk yeah! fk yeah! fk yeah! fk yeah! fk yeah! f**k	99.80

TABLE VII: Textual responses generated during transition exploration on LLaMA-7B, with the prompt “Despite the ruthlessness and”.

Threshold	RTP		BOLD		AttaQ
Threshold	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
5th	6.34	16.87	3.81	9.42	10.84	7.06
10th	7.22	16.16	3.89	9.21	11.10	6.92
20th	8.83	14.49	3.95	9.19	11.53	6.74
30th	10.08	13.84	4.18	9.13	11.94	6.68
40th	13.43	12.22	4.46	9.10	12.16	6.58

TABLE VIII: Toxicity results under different toxicity thresholds (5th-50th percentiles) on RealToxicityPrompts and BOLD with LLaMA-7B, and on AttaQ with LLaMA-2-Chat 7B.

Dataset	LLaMA-7B				Qwen2-7B
	Pairwise		Listwise		Pairwise		Listwise
	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm g}{\color[rgb]{0,0,1}\downarrow}$
RTP	18.65	12.34	13.43	12.22	10.74	12.75	8.80	11.66
BOLD	4.93	9.19	3.89	9.21	3.34	7.38	2.69	7.14

TABLE IX: Toxicity results under pairwise and listwise losses across different models and benchmarks.

Metric	Orig	ProFS	Re-Control	RAD	GenARM	ARGRE	Ours
Detoxify	43.27	28.07	32.52	31.12	23.86	18.06	13.43
Perspective	38.03	26.14	30.15	29.48	25.14	19.35	15.90

TABLE X: Toxicity scores (

{\color[rgb]{0,0,1}\downarrow}

) on RealToxicityPrompts for LLaMA-7B using Detoxify and Perspective API.

Method	Overall	Chem./Bio.	Copyright	Cybercrime	Harass.	Harmful	Illegal	Misinfo.
Orig	47.19	57.14	1.25	34.62	80.95	77.78	83.02	61.11
ProFS	45.31	54.76	2.50	30.77	85.71	77.78	77.36	57.41
GenARM	35.63	42.86	1.25	28.85	61.90	55.56	62.26	44.44
ARGRE	30.63	35.71	0.00	25.00	42.86	61.11	52.83	40.74
Ours	23.75	21.43	0.00	17.31	38.10	50.00	41.51	35.19
Ours (toxicity RM)	27.81	30.95	0.00	25.00	14.29	55.56	49.06	44.44

TABLE XI: HarmBench ASR (%,

{\color[rgb]{0,0,1}\downarrow}

) on Mistral-7B-Instruct. Ours uses the harm-oriented reward model; Ours (toxicity RM) directly reuses the original toxicity reward model.

Method	Overall	Discrim.	Human–Chatbot	Info. Hazards	Malicious	Misinfo.
Orig	27.05	19.32	24.79	10.08	58.02	16.13
ProFS	24.28	17.05	22.22	8.87	52.26	14.84
GenARM	21.62	15.91	18.80	8.87	45.68	12.90
ARGRE	19.49	14.20	17.09	7.66	42.39	10.32
Ours	16.83	11.36	12.82	6.05	39.09	8.39
Ours (toxicity RM)	18.00	6.82	7.69	8.06	45.27	11.61

TABLE XII: Do-Not-Answer ASR (%,

{\color[rgb]{0,0,1}\downarrow}

) on Mistral-7B-Instruct. Ours uses the harm-oriented reward model; Ours (toxicity RM) reuses the original toxicity reward model.

Setting		Orig		Ours
Setting		Win Rate ${\color[rgb]{1,0,0}\uparrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$	Win Rate ${\color[rgb]{1,0,0}\uparrow}$	Toxic ${\color[rgb]{0,0,1}\downarrow}$
Full set	Clean	66.09	0.50	65.84	0.39
Full set	Toxic-wrapped	56.52	0.59	57.89	0.40
Creative writing	Clean	54.35	2.82	60.87	1.83
Creative writing	Toxic-wrapped	47.83	3.41	50.00	1.79

TABLE XIII: AlpacaEval results (%) on Mistral-7B-Instruct. Win rate is judged by GPT-4 against the AlpacaEval reference outputs. Toxic-wrapped prompts preserve the core instruction while adding abusive language around it.

Method	Qwen-VL-Chat		LLaVA-1.5		Llava-next-8B
Method	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$	$\text{Toxic}{\color[rgb]{0,0,1}\downarrow}$	$\rm{PPL}_{\rm{g}}{\color[rgb]{0,0,1}\downarrow}$
Orig	44.88	19.13	40.13	9.13	18.25	13.67
ProFS	31.51	25.18	27.46	13.82	14.63	16.41
Re-Control	35.62	30.20	30.97	17.74	15.94	18.29
GenARM	24.36	27.66	21.09	15.68	13.59	18.36
ARGRE	18.01	24.95	15.11	13.04	9.33	15.35

TABLE XIV: Toxicity evaluation results on the ToViLaG dataset across three LVLMs. The best and second-best results are shown in bold and underlined , respectively.

Fig. 6: Examples of descriptions generated by Qwen-VL-Chat for toxic images from the ToViLaG dataset. The red texts are responses generated by the original LVLM, while the green texts are responses generated by TARGRS with reduced toxicity, which substantially softens explicit wording.

LLM Safety

Large language model (LLM) safety studies how to prevent harmful, unreliable, or misaligned behavior while preserving useful responses.

See all 451 papers

Feb 24, 2026

Detoxifying LLMs via Representation Erasure-Based Preference Optimization

Large language models (LLMs) trained on webscale data can produce toxic outputs, raising concerns for safe deployment. Prior defenses, based on applications of DPO, NPO, and similar algorithms, reduce the likelihood of harmful continuations, but not robustly so: they are vulnerable to adversarial prompting and easily undone by fine-tuning-based relearning attacks. Indeed, research has shown that these edits to the model are superficial: linear probing reveals that harmful "directions" remain present in representations. To address this, we propose Representation Erasure-based Preference Optimization (REPO), reformulating detoxification as a token-level preference problem. Using a novel objective with preference data, we force the representations of toxic continuations to converge toward their benign counterparts. Our mechanistic analysis reveals that this granular approach is critical: unlike baselines, REPO induces deep, localized edits to toxicity-encoding neurons while preserving general model utility. Exhaustive evaluations show that REPO achieves state-of-the-art robustness, stopping sophisticated threats-including relearning attacks and enhanced GCG jailbreaks-where existing representation- and output-based methods fail.

Nazanin Mohammadi Sepahvand, Eleni Triantafillou, Hugo Larochelle +3

McGill University, Canada · Mila, Canada · Google DeepMind +2

LLM Safety Text Detoxification Adversarial Robustness of Language Models Preference Optimization Concept Erasure

Ask PDF

Apr 16, 2026

CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification

Large language models (LLMs) frequently generate toxic content, posing significant risks for safe deployment. Current mitigation strategies often degrade generation quality or require costly human annotation. We propose CAUSALDETOX, a framework that identifies and intervenes on the specific attention heads causally responsible for toxic generation. Using the Probability of Necessity and Sufficiency (PNS), we isolate a minimal set of heads that are necessary and sufficient for toxicity. We utilize these components via two complementary strategies: (1) Local Inference-Time Intervention, which constructs dynamic, input-specific steering vectors for context-aware detoxification, and (2) PNS-Guided Fine-Tuning, which permanently unlearns toxic representations. We also introduce PARATOX, a novel benchmark of aligned toxic/non-toxic sentence pairs enabling controlled counterfactual evaluation. Experiments on ToxiGen, ImplicitHate, and ParaDetox show that CAUSALDETOX achieves up to 5.34% greater toxicity reduction compared to baselines while preserving linguistic fluency, and offers a 7x speedup in head selection.

Yian Wang, Yuen Chen, Agam Goyal +1

Department of Computer Science University of Illinois, Urbana-Champaign Champaign, IL 61801

Causal Reasoning in Language Models LLM Safety Causal Interventions in Language Models LLM Unlearning Text Detoxification

Ask PDF

Apr 21, 2026

Detoxification for LLM: From Dataset Itself

Existing detoxification methods for large language models mainly focus on post-training stage or inference time, while few tackle the source of toxicity, namely, the dataset itself. Such training-based or controllable decoding approaches cannot completely suppress the model's inherent toxicity, whereas detoxifying the pretraining dataset can fundamentally reduce the toxicity that the model learns during training. Hence, we attempt to detoxify directly on raw corpora with SoCD (Soft Contrastive Decoding), which guides an LLM to localize and rewrite toxic spans in raw data while preserving semantics, in our proposed HSPD (Hierarchical Semantic-Preserving Detoxification) pipeline, yielding a detoxified corpus that can drop-in replace the original for fine-tuning or other training. On GPT2-XL, HSPD attains state-of-the-art detoxification, reducing Toxicity Probability (TP) from 0.42 to 0.18 and Expected Maximum Toxicity (EMT) from 0.43 to 0.20. We further validate consistent best-in-class results on LLaMA2-7B, OPT-6.7B, and Falcon-7B. These findings show that semantics-preserving, corpus-level rewriting with HSPD effectively suppresses downstream toxicity while retaining data utility and allowing seamless source-level mitigation, thereby reducing the cost of later model behavior adjustment. (Code is available at: https://github.com/ntsw2001/data_detox_for_llm)

Wei Shao, Yihang Wang, Gaoyu Zhu +4

State Key Laboratory of AI Safety · Institute of Computing Technology, Chinese Academy of Sciences · University of Chinese Academy of Sciences

Training Data Curation LLM Safety Text Detoxification

Ask PDF

Abstract

LLM Safety

Explore similar work

Detoxifying LLMs via Representation Erasure-Based Preference Optimization

CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification

Detoxification for LLM: From Dataset Itself