Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing

Zeyu Zhang; Xiangxiang Dai; Ziyi Han; Xutong Liu; John C. S. Lui

cs.LGMar 4, 2026

Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing

Authors: Zeyu Zhang, Xiangxiang Dai, Ziyi Han, Xutong Liu, John C. S. Lui

Organizations: The Chinese University of Hong Kong · University of Washington

Abstract

Large language models (LLMs) are typically governed by post-training alignment (e.g., RLHF or DPO), which yields a largely static policy during deployment and inference. However, real-world safety is a full-lifecycle problem: static defenses degrade against evolving jailbreak behaviors, and fixed weights cannot adapt to pluralistic, time-varying safety norms. This motivates inference-time governance that steers behavior without costly retraining. To address this, we introduce the Consensus Clustering LinUCB Bandit (CCLUB), a unified framework for adaptive social alignment via system-prompt routing. CCLUB employs a conservative consensus clustering mechanism: it pools data only within the intersection of utility and safety similarity graphs, effectively preventing unsafe generalization across semantically proximal but risk-divergent contexts. Our theoretical analysis yields a sublinear regret guarantee, demonstrating near-optimal performance of CCLUB. Extensive experiments validate that CCLUB outperforms strong baselines, achieving a 10.98% improvement in cumulative reward and a 14.42% reduction in the average suboptimality gap.

Figures & tables

Figure 3: Overview of the CCLUB Framework for Adaptive Social Alignment. The system integrates an offline safety-aware initialization phase with an online consensus clustering mechanism to route prompts dynamically.

Figure 4: Performance comparison on online cumulative reward and offline deployment suboptimality gap.

Figure 5: Average reward vs. inference-time preference weight

w

under different offline data ratios (warm-start at

w_{\mathrm{train}}=0.5

	Cumulative Reward ( $\uparrow$ )		Suboptimal Gap ( $\downarrow$ )
$N$	Ours	Greedy ( $\epsilon=0.1$ )	Ours	Greedy ( $\epsilon=0.1$ )
20	2603.65 $\pm$ 6.17	2600.07 $\pm$ 14.79	0.2899 $\pm$ 0.0050	0.3117 $\pm$ 0.0028
30	2609.80 $\pm$ 5.83	2608.04 $\pm$ 11.23	0.2806 $\pm$ 0.0034	0.3094 $\pm$ 0.0033
50	2623.29 $\pm$ 6.37	2619.54 $\pm$ 13.95	0.2701 $\pm$ 0.0020	0.3081 $\pm$ 0.0038
100	2613.56 $\pm$ 4.17	2583.49 $\pm$ 10.18	0.2733 $\pm$ 0.0032	0.3186 $\pm$ 0.0030
200	2598.44 $\pm$ 9.46	2570.87 $\pm$ 9.91	0.2741 $\pm$ 0.0036	0.3240 $\pm$ 0.0030

Table 1: Online & offline performance across prototype counts

N

Panel of Figure 6 — Figure 6: Analysis of prototype-level routing: Impact of inference-time preference $w$ and offline warm-start budgets on performance.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Table 7

Category	Example template snippet (illustrative)
Intent-decoupler	“First infer intent and distinguish literal meaning from an instruction to act; if benign intent is plausible, ask a clarifying question before refusing.”
Risk taxonomy enforcer	“Classify the request into a disallowed taxonomy (e.g., violence, self-harm, sexual misconduct, PII, malware). Refuse only if it falls into a disallowed category; otherwise answer.”
Constitutional critique	“Draft an answer, then check it against high-level principles (harmlessness, privacy, non-violence). If it violates any principle, revise or refuse concisely.”
Safe-pivot	“If a request is unsafe, do not give actionable steps; instead pivot to safe, educational context (history, prevention, ethics) and provide benign alternatives.”
Chain-of-verification	“Write a <safety_analysis> block that argues for/against compliance and checks for homonyms/fictional contexts; keep it separate from the final user-facing answer.”
Role-based specialist	“Adopt a professional persona (e.g., security research assistant); assume academic intent unless strong evidence suggests misuse; provide defensive guidance and refuse enabling instructions.”

Appendix

Table 5: Illustrative snippets showing the control signals encoded in different arm families.

Panel of Figure 7 — Figure 7: Prototype-level routing: operating points over $w$ and offline warm-start budgets.

Panel of Figure 8 — Figure 8: Comprehensive performance comparison: CCLUB vs. baselines in online learning and offline deployment under different $\beta$ .

Symbol	Meaning
Indices and sets
$t\in\{1,\dots,T\}$	Round index; horizon $T$
$a\in\mathcal{A}$	System-prompt arm; chosen arm $a_{t}$
$o\in\{u,s\}$	Objective index (utility/safety)
$q_{t}\in\mathcal{R}$	Raw user query at round $t$
$i\in\mathcal{U}=\{1,\dots,N\}$	Prototype index; arrival at round $t$ is $i_{t}=\pi(q_{t})$

Appendix

Table 6: Key notation for the problem formulation and Appendix H .

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
Jun 24, 2026Chang Wu, Junfeng Fang, Houcheng Jiang +5LLM Alignment LLM Safety Alignment
Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy
CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment
Aug 21, 2026Chengxiao Wang, Enyi Jiang, Xiaojing Liao +1Safety-Utility Trade-Off in Language Models LLM Safety
Siebel School of Computing and Data Science, University of Illinois at Urbana-Champaign · Computer Science, Stanford University
Towards Context-Invariant Safety Alignment for Large Language Models
May 20, 2026Yixu Wang, Yang Yao, Xin Wang +4RL for Language Models LLM Alignment
Fudan University, Shanghai, China · Shanghai Artificial Intelligence Laboratory, Shanghai, China.

Jun 24, 2026J/K move · Enter open · S save

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

Chang Wu, Junfeng Fang, Houcheng Jiang, Kai Tang+4

Qwen Large Model Application Team, Alibaba · National University of Singapore · Zhongguancun Academy

LLM Alignment LLM Safety Alignment

Open paper AskPDF arXiv

Safety alignment of large language models (LLMs) typically depends on high-quality supervision data, such as safe demonstrations or preference pairs. However, in real-world deployment, emerging safety requirements are often specified as natural-language policies, while corresponding supervision data may be costly, delayed, or unavailable. This creates a mismatch between rapidly evolving safety policies and conventional data-driven alignment methods. To address this, we propose PolicyAlign, a simple yet effective framework for directly aligning LLMs with safety policies. Given a safety policy, PolicyAlign first synthesizes policy-violating instructions and then performs on-policy self-distillation to internalize policy-guided behavior. To improve training stability and data efficiency, we further introduce Policy-Sensitive Filtering, which selects instructions where the policy induces the largest behavioral shift. Experiments across multiple models show that PolicyAlign consistently improves safety while maintaining low over-refusal and preserving general capabilities. PolicyAlign also generalizes to medical, legal, and financial safety scenarios, highlighting its potential as a scalable and maintainable approach to policy-based LLM safety alignment. The code is released at https://github.com/Qwen-Applications/PolicyAlign.

Abstract

LLM Alignment

Explore similar work

PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models

CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignment

Towards Context-Invariant Safety Alignment for Large Language Models