CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification

Yian Wang; Yuen Chen; Agam Goyal; Hari Sundaram

cs.CLApr 16, 2026

CausalDetox: Causal Head Selection and Intervention for Language Model Detoxification

Authors: Yian Wang, Yuen Chen, Agam Goyal, Hari Sundaram

Organizations: Department of Computer Science University of Illinois, Urbana-Champaign Champaign, IL 61801

Abstract

Large language models (LLMs) frequently generate toxic content, posing significant risks for safe deployment. Current mitigation strategies often degrade generation quality or require costly human annotation. We propose CAUSALDETOX, a framework that identifies and intervenes on the specific attention heads causally responsible for toxic generation. Using the Probability of Necessity and Sufficiency (PNS), we isolate a minimal set of heads that are necessary and sufficient for toxicity. We utilize these components via two complementary strategies: (1) Local Inference-Time Intervention, which constructs dynamic, input-specific steering vectors for context-aware detoxification, and (2) PNS-Guided Fine-Tuning, which permanently unlearns toxic representations. We also introduce PARATOX, a novel benchmark of aligned toxic/non-toxic sentence pairs enabling controlled counterfactual evaluation. Experiments on ToxiGen, ImplicitHate, and ParaDetox show that CAUSALDETOX achieves up to 5.34% greater toxicity reduction compared to baselines while preserving linguistic fluency, and offers a 7x speedup in head selection.

Explore similar work

Test-Time Detoxification without Training or Learning Anything
Jan 14, 2026Baturay Saglam, Dionysis KalogeriasToxicity Detoxification
Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models
May 27, 2026Himanshu Beniwal, Mayank SinghToxicity Large Language Model Safety
Detoxification for LLM: From Dataset Itself
Apr 21, 2026Wei Shao, Yihang Wang, Gaoyu Zhu +4Detoxification Toxicity

Jan 14, 2026 · cs.CLJ/K move · Enter open · S save

Test-Time Detoxification without Training or Learning Anything

Baturay Saglam, Dionysis Kalogerias

Department of Electrical and Computer Engineering, Yale University, New Haven, CT.

Large language models can produce toxic or inappropriate text even for benign inputs, creating risks when deployed at scale. Detoxification is therefore important for safety and user trust, particularly when we want to reduce harmful content without sacrificing the model's generation quality. Many existing approaches rely on model retraining, gradients, or learned auxiliary components, which can be costly and may not transfer across model families or to truly black-box settings. We introduce a test-time procedure that approximates the gradient of completion toxicity with respect to the input embeddings and uses a small number of descent steps to steer generation toward less toxic continuations. This is achieved with zeroth-order optimization that requires only access to input embeddings, a toxicity scoring function, and forward evaluations of the model. Empirically, the approach delivers robust toxicity reductions across models and prompts and, in most settings, achieves the best overall toxicity-quality trade-off. More broadly, our work positions word embeddings as effective control variables and encourages wider use of black-box optimization to guide autoregressive language models toward scalable, safer text generation, without requiring any training or access to intermediate computations.

Toxicity Detoxification Large Language Model Safety Harmful Content Text Generation Autoregressive Language Models Learn

Open paper Ask about it PDF

Abstract

Explore similar work

Test-Time Detoxification without Training or Learning Anything

Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models

Detoxification for LLM: From Dataset Itself