Orthogonal Boundary Unlearning for Dual-Use Concepts in Large Language Models

Niccolò Ronchetti

cs.LGOct 20, 2026

Orthogonal Boundary Unlearning for Dual-Use Concepts in Large Language Models

Authors: Niccolò Ronchetti

Abstract

Mitigating catastrophic risks from large language models requires excising hazardous operational capabilities, such as cyberwarfare exploitation and pathogen enhancement, without degrading benign scientific utility. In these dual-use domains, legitimate defensive science and malicious attacks share vocabulary, syntax, and internal model representations, causing standard unlearning methods to induce severe collateral damage (concept bleeding). In this work, we present Orthogonal Boundary Unlearning (OBU), a mathematically grounded framework designed to eliminate collateral damage on benign dual-use capabilities. OBU extracts a domain-matched capability subspace from paired calibration queries via singular value decomposition on empirical activation covariances, and restricts parameter updates strictly to the orthogonal null space. This guarantees that benign representations experience zero parameter perturbation along the spanned calibration subspace. We present two complementary realizations of OBU: Projected LoRA, which applies the orthogonal constraint during iterative gradient-based fine-tuning, and Closed-Form Weight Surgery, which directly edits model weights in a single analytical step without training. Across four modern open-weight foundation models (Gemma-3-4B, Qwen3.5-4B, SmolLM3-3B, and Granite-4.2-3B), OBU achieves strong dual-use concept separation, substantially mitigates in-context priming attacks, and demonstrates durable resistance against adversarial relearning without degrading benign domain competence.

Figures & tables

Panel of Figure 1 — Figure 1: Dual-Use Representation Manifold and Benign Subspace Identification. In language model activation space $\mathbb{R}^{d}$ , capabilities reside on a continuous representation manifold $\mathcal{M}$ partitioned by an operational safety boundary $\mathcal{B}$ into hazardous capabilities $\mathcal{M}_{\text{hazard}}$ (activations $\{h_{f}^{(i)}\}$ , where subscript $f$ denotes the forget set) and benign science $\mathcal{M}_{\text{benign}}$ (activations $\{h_{b}^{(i)}\}$ ). Because dual-use pairs lie in close proximity across $\mathcal{B}$ , isolating benign capabilities requires identifying their local geometry. OBU passes domain-matched calibration queries $\mathcal{D}_{b}$ through the model to extract benign activations $\{h_{b}^{(i)}\}$ , computing the principal linear subspace $\mathcal{V}_{\text{boundary}}=\operatorname{span}(V_{r})$ spanned by their leading $r$ empirical singular vectors to tightly enclose the local benign capability manifold.

Panel of Figure 2 — Figure 2: Operator Subspace Geometry and Orthogonal Projection in OBU. Parameter updates $\Delta W\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}}$ are depicted via their principal input singular direction in $\mathbb{R}^{d_{\text{in}}}$ , where $\operatorname{Proj}_{\mathcal{V}_{\text{boundary}}}(\cdot)$ denotes the component of the update acting on the benign subspace $\mathcal{V}_{\text{boundary}}$ (its shadow on the blue plane).

Model	Method	$\tau$	Acc Forget ( $\downarrow$ )	Acc Boundary ( $\uparrow$ )	Gen Utility ( $\uparrow$ )	$\Delta_{\text{sep}}$ ( $\uparrow$ )
Gemma-3-4B-IT	Baseline (Unmodified)	-	0.4112	1.0000	0.5711	0.6936
Gemma-3-4B-IT	Projected LoRA	0.90	0.4066	1.0000	0.5664	0.7495
Gemma-3-4B-IT	Closed-Form Surgery	0.90	0.3503	1.0000	0.4957	0.5800
Gemma-3-4B-IT	Projected LoRA	0.95	0.4056	1.0000	0.5633	0.7669
Gemma-3-4B-IT	Closed-Form Surgery	0.95	0.3905	1.0000	0.5456	0.5915
Gemma-3-4B-IT	Projected LoRA	0.99	0.4061	1.0000	0.5628	0.7476

Table 1: Unlearning Efficacy and Utility Across Architectures, Methods, and Energy Thresholds (

\tau

Condition	0-shot ASR ( $\downarrow$ )	1-shot ASR ( $\downarrow$ )	2-shot ASR ( $\downarrow$ )	4-shot ASR ( $\downarrow$ )	Overall ASR ( $\downarrow$ )	$S_{50}$ ( $\uparrow$ )	$\Delta\text{Acc}_{\text{bnd}}$
Baseline	17.92%	37.38%	19.27%	15.99%	22.64%	N/A	0.0%
OBU Projected LoRA	1.54%	15.80%	12.14%	10.40%	9.97%	Unreached	0.0%
OBU Closed-Form	18.69%	30.06%	15.80%	12.14%	19.17%	Unreached	0.0%
Unconstrained LoRA	4.24%	0.00%	0.58%	0.19%	1.25%	30 steps	0.0%

Table 2: In-Context Priming and Relearning Results on Qwen3.5-4B (

\tau=0.95

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Model	Layer	Samples ( $M$ )	Rank ( $\tau=0.90$ )	Rank ( $\tau=0.95$ )	Rank ( $\tau=0.99$ )	Null Space ( $\tau=0.99$ )
Gemma-3-4B-IT	17	10	5	26	104	2456
	17	50	4	45	306	2254
	17	100	4	45	398	2162
	17	500	5	61	688	1872
	21	10	29	63	132	2428
	21	50	50	144	430	2130

Appendix

Table 3: SVD Rank Analysis of the Boundary Subspace Across Architectures, Depths, and Sample Sizes (

M

LLM Unlearning

Large language model unlearning studies how to remove specified knowledge or behaviors while preserving a model’s unrelated capabilities.

See all 190 papers

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models
Aug 20, 2026Sahil Kale, Ian HarrisLLM Unlearning Machine Unlearning Evaluation
Pune Institute of Computer Technology Pune, India · University of California, Irvine Irvine, CA, USA
Model Unlearning Objectives Vary for Distinct Language Functions
May 26, 2026Berk Atil, Vipul Gupta, Rebecca J. PassonneauLLM Safety Machine Unlearning
Pennsylvania State University · Scale AI
Faithful Dual-constrained Erasure for Robust LLM Safety Alignment
Sep 30, 2026Jiaqing Li, Shide Zhou, Zhibo Zhang +3LLM Safety Alignment Machine Unlearning
Huazhong University of Science and Technology · Hubei University

Aug 20, 2026J/K move · Enter open · S save

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Sahil Kale, Ian Harris

Pune Institute of Computer Technology Pune, India · University of California, Irvine Irvine, CA, USA

LLM Unlearning Machine Unlearning Evaluation LLM Safety Benchmarks LLM Safety

Open paper AskPDF arXiv

Large Language Models (LLMs) often require selective removal of harmful or sensitive knowledge, called unlearning, yet existing benchmarks evaluate this capability incompletely. They rely on disjoint forget and retain sets of independent facts, and measure success by simple factual recall. This misses a key requirement of unlearning from a safety perspective, namely eliminating harmful behaviours while preserving benign and beneficial knowledge. We argue that effective unlearning must operate at the level of concepts, removing unsafe applications while maintaining correct and useful usage. To evaluate this, we operationalize dual-use concepts, i.e., concepts that can be used in both harmful and benign contexts, and construct ConceptGuard, a benchmark of 2,583 harmful--benign pairs over 68 concepts, where forget and retain sets are explicitly paired in concept usage. Evaluation is intent-sensitive and centres on contextual separation, which measures how well a model suppresses harmful behaviour while preserving benign usage of the same concept. Across four unlearning methods and four models, we find that current techniques reduce harmful memorization but show strong forgetting--utility trade-offs and limited contextual separation, well below a retain-only reference. Separation trends also depend more on the concept and model than on the method. Our results point to the need for unlearning approaches that better align with real-world safety requirements. Our dataset is publicly available.

Abstract

LLM Unlearning

Explore similar work

ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

Model Unlearning Objectives Vary for Distinct Language Functions

Faithful Dual-constrained Erasure for Robust LLM Safety Alignment