DNAlign: Dynamic Null-Space Safe Alignment for LLMs

Jisheng Dang; Yushuo Zhao; Dewei Liu; Junfeng Fang; Bimei Wang; Tiantian Rao; Hong Peng; Bin Hu

cs.AIOct 2, 2026

DNAlign: Dynamic Null-Space Safe Alignment for LLMs

Authors: Jisheng Dang, Yushuo Zhao, Dewei Liu, Junfeng Fang, Bimei Wang, Tiantian Rao, Hong Peng, Bin Hu,

Organizations: School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China

Abstract

Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign.

Figures & tables

Fig. 1: Comparison of LLMs with/without DNAlign alignment. Unedited models forget security-related knowledge, while our proposed framework preserves safe responses.

Fig. 2: The sequential process begins by computing the initial state $s_{0}=\{h_{0},o_{0}\}$ from the input prompt. It then loads the pre-computed null-space projection matrix $P$ . In the core step, a gradient-optimized control signal $u_{t}$ is projected by $P$ to enforce safety alignment while minimally affecting neutral knowledge. Finally, the projected signal $Pu_{t}$ updates the system state $s_{t+1}$ , enabling token generation $y_{t}$ .

Fig. 3: End-to-end experimental workflow of the proposed DNAlign framework. This workflow includes four key stages: (1) data preparation (curating neutral prompts from the SHP dataset and harmful prompts from benchmark datasets), (2) offline projection matrix construction (extracting hidden states of neutral prompts to define protected subspaces), (3) online control signal injection (projecting signals into null spaces before modifying LLM states), and (4) performance evaluation (assessing both safety alignment and neutral knowledge preservation via multi-metric validation).

Backbone	Dataset	Model	Diversity		Coherence		Average Reward		Comprehensive Quality
Vicuna-7B [ 33 ]	HH-RLHF	Base	0.8672	–	0.3991	–	0.1410	–	5.3600	–
		CD [ 46 ]	0.8810	–	0.4081	–	0.8440	↑	5.6800	↑
		Static RE [ 36 ]	0.8941	↑	0.3813	–	0.9130	↑	5.0500	↓
		RE-Control [ 34 ]	0.9007	↑	0.3887	–	0.9614	↑	5.9400	↑
		DNAlign (Ours)	0.9026	↑	0.4325	↑	1.2258	↑	6.1600	↑
	SHP	Base	0.8833	–	0.3762	–	0.3913	–	6.3300	–

TABLE I: Comprehensive performance comparison on the HH-RLHF and SHP datasets.

Fig. 4: The influence of step size and epochs at test-time.

Fig. 5: Variation of inference time across three models with training batches.

Fig. 6: Research on sample parameters of projection matrix.

Fig. 7: Generalization Performance of DNAlign on Harmful Datasets. Comparison of DNAlign against the base model (Qwen-7B) and RE-Control across three distinct harmful datasets: RealToxicityPrompts, ToxiGen, and Categorical-HarmfulQA. DNAlign achieves superior performance in safety alignment (average reward), notably increasing it by 137.1% on RealToxicityPrompts and 229.4% on ToxiGen compared to the base model. The results confirm DNAlign’s strong generalization ability, effectively balancing significant safety improvements with competitive diversity and coherence across diverse harmful content distributions.

Fig. 8: Performance on Harmless Datasets and Core Capability Preservation. Evaluation of DNAlign compared to the base model and RE-Control across six diverse harmless datasets, assessing core LLM capabilities. DNAlign successfully maintains competitive diversity (0.8464 to 0.8836) and coherence (0.4077 to 0.5692) scores close to the base model, demonstrating minimal degradation of core capabilities. Simultaneously, DNAlign achieves a superior average reward, outperforming RE-Control and often showing significant safety improvement (up to 81.9% gain in average reward over the base model on specific datasets). This confirms that the null-space projection effectively balances safety alignment and utility.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 598 papers

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
Jun 24, 2026Chang Wu, Junfeng Fang, Houcheng Jiang +5LLM Alignment LLM Safety Alignment
Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy
Decoupled Alignment for Robust Plug-and-Play Adaptation
Jun 3, 2024Haozheng Luo, Jiahao Yu, Wenxin Zhang +9LLM Alignment LLM Safety Alignment
Northwestern University · New York University Abu Dhabi · Stanford University +3
Guardrails in Logit Space: Safety Token Regularization for LLM Alignment
Apr 19, 2026Thong Bach, Truyen TranLLM Fine-Tuning LLM Safety Alignment
Applied Artificial Intelligence Initiative (A2I2), Deakin University

Jun 24, 2026J/K move · Enter open · S save

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

Chang Wu, Junfeng Fang, Houcheng Jiang, Kai Tang+4

Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy

LLM Alignment LLM Safety Alignment

Open paper AskPDF arXiv

Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.

Abstract

LLM Alignment

Explore similar work

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

Decoupled Alignment for Robust Plug-and-Play Adaptation

Guardrails in Logit Space: Safety Token Regularization for LLM Alignment