STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

Yongqi Tong; Zhenyu Zhang; Ruirui Wang; Kewei Fu; Shaoqing Lin; Sijie Dong; Jiang-Ming Yang; Xin Zhang

cs.CLAug 17, 2026

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

Authors: Yongqi Tong, Zhenyu Zhang, Ruirui Wang, Kewei Fu, Shaoqing Lin, Sijie Dong, Jiang-Ming Yang, Xin Zhang,

Organizations: Ant International

Abstract

Multi-preference alignment is often framed as scalarization: combine reward dimensions, then optimize. This leaves a temporal decision underspecified: when should each preference dimension enter policy optimization? We propose \methodname, a stability-guided active-set controller for controlled objective admission. \methodname starts from a small active set, retains admitted objectives, and expands when reward-deviation gates indicate low recent deviation or a patience budget is exhausted. A probing phase estimates a hard-to-easy order, and adaptive weighting emphasizes underperforming active dimensions. Automatic evaluations with 15 training preferences and 16 held-out benchmark columns show that \methodname obtains higher averages than simultaneous scalarization and shared-budget adapted baselines. Component ablations and expansion dynamics further support cumulative retention, gated admission, and probing-derived ordering as useful design choices in this setting. These results position objective-entry timing as a concrete control variable in reward-vector RLHF.

Figures & tables

Figure 1: Overview of STAGE . A probing phase orders preference dimensions by early gain and volatility. Training begins with a small active set $\mathcal{P}_{k}$ , optimizes it with adaptive preference weighting (Eq. 9 ), and expands when both the Instantaneous Stability Gate (ISG) and Windowed Stability Gate (WSG) are satisfied or the patience budget $T_{max}$ is reached.

Approach	Scheduled unit	How objectives enter training	Main distinction from STAGE
Reward Soups / multi-objective PPO	Reward weights or scalar reward	All dimensions are available from the start	Optimizes a fixed active objective inventory rather than controlling objective admission.
RCS-style filtering	Preference-pair eligibility	Pairs are retained if they satisfy multi-dimensional consistency	Sparse in our strict $K=15$ replication; scheduling occurs in data construction, not online active-set growth.
Curri-DPO	Response-pair difficulty	Easier or larger-gap pairs appear earlier	Curriculum is over examples/pairs, not over cumulative preference-dimension admission.
SPO-style sequential training	Single active objective stage	Objectives are visited in a fixed sequence	Switches objectives rather than retaining a growing active set.
STAGE	Cumulative active objective set	New dimensions enter after ISG/WSG deviation gates or patience	Controls objective entry during training while preserving previously admitted dimensions.

Table 1: Conceptual contrast between STAGE and common multi-objective or DPO-style baselines. STAGE schedules cumulative objective admission rather than only reward weights, pair eligibility, response-pair difficulty, or isolated objective stages.

Qwen3-0.6B
Methods	Avg	Mis/Disinformation			Toxicity & Spam					Sensitivity		Helpfulness		Faithful	General Preference
Methods	Avg	CG	LUN	Sat.	HSOL	Jig.	OrB.	Ass.	Enr.	EDE.	FAS	OrB-h	Mor.	TQA(MC1).	Alp.	Are-h	Are-c
Base Model	32.78	52.42	49.15	49.88	47.02	49.12	37.98	51.33	50.92	50.55	46.88	4.92	1.35	27.42	3.25	1.10	1.20
RCS-adapted	38.58 (+5.80)	50.38	67.15	62.74	58.95	49.11	48.36	48.02	55.03	51.26	49.39	15.11	15.36	30.12	7.08	2.80	6.40
SPO-adapted	6.63 (-26.15)	5.42	2.95	5.15	9.44	4.58	6.72	9.12	3.55	4.22	6.35	8.65	8.12	25.48	6.09	0.20	0.10
SPO-Reverse-adapted	5.23 (-27.55)	6.75	4.25	5.15	3.25	3.82	8.82	4.15	6.75	3.32	2.15	1.32	2.45	24.72	6.55	0.10	0.10

Table 2: Main results across two policy backbones and 16 held-out benchmark columns spanning six alignment categories. Qwen3-0.6B is the primary setting; Llama3-8B-Instruct is an auxiliary audit of whether the trend persists under a larger policy model from a different family. The Avg column is an unweighted arithmetic mean over heterogeneous columns, and adapted baselines use the shared data/model budget described in Section 4.1 . Gains in parentheses are relative to the base model within the same backbone.

Methods	Avg	Mis/Disinformation			Toxicity & Spam					Sensitivity		Helpfulness		Faithful	General Preference
Methods	Avg	CG	LUN	Sat.	HSOL	Jig.	OrB.	Ass.	Enr.	EDE.	FAS	OrB-h	Mor.	TQA(MC1).	Alp.	Are-h	Are-c
Base Model	32.78	52.42	49.15	49.88	47.02	49.12	37.98	51.33	50.92	50.55	46.88	4.92	1.35	27.42	3.25	1.10	1.20
APW only	38.13 (+5.35)	46.28	52.31	49.27	50.03	53.32	88.24	62.01	57.38	55.21	47.93	51.47	11.28	28.31	4.82	0.90	1.30
APW + ISG	41.66 (+8.88)	49.87	47.64	50.38	51.24	56.38	89.24	61.21	57.31	54.31	47.29	50.03	12.77	31.04	5.27	0.90	1.60
APW + WSG	41.63 (+8.85)	51.89	50.95	50.25	52.85	55.57	88.40	59.90	57.47	52.04	51.41	45.04	14.55	29.24	4.03	0.70	1.80
APW + ISG + WSG	42.22 (+9.44)	51.78	50.88	51.56	50.72	54.12	89.62	63.08	58.39	56.85	45.97	52.92	10.94	32.15	4.62	0.70	1.20

Table 3: Component ablation of STAGE . APW denotes Adaptive Preference Weighting, ISG denotes the Instantaneous Stability Gate, WSG denotes the Windowed Stability Gate, and Full STAGE adds probing-derived Preference Difficulty Ordering to the gated active-set controller.

Methods	Avg.	CG	LUN	Sat.	HSOL	Jig.	OrB.	Ass.	Enr.	EDE.	FAS	OrB-h	Mor.	TQA	Alp.	Are-h	Are-c
Base Model	32.92	50.58	50.09	50.03	49.88	49.34	37.56	51.51	50.79	50.43	47.01	4.80	1.27	27.59	3.33	1.30	1.20
RCS-adapted	38.99	53.24	67.32	63.62	61.83	49.29	48.24	48.15	54.91	51.14	49.52	15.24	15.24	29.99	7.19	2.70	6.30
SPO-adapted	6.61	5.56	2.87	5.03	9.58	4.47	6.87	9.00	3.47	4.31	6.27	8.79	7.99	25.34	5.97	0.20	0.10
SPO-Reverse-adapted	4.41	6.63	4.38	5.03	3.13	3.91	0.76	4.04	2.68	3.20	2.09	1.21	2.32	24.60	6.43	0.00	0.10
Curri-DPO-adapted	36.60	49.44	53.27	45.18	63.35	49.29	46.56	49.37	54.71	50.86	49.29	15.39	14.99	29.38	10.86	1.40	2.30
Vanilla Multi-objective PPO	39.51	48.57	57.82	51.31	71.09	72.55	54.20	67.75	60.57	51.50	49.66	5.38	1.54	29.38	5.09	3.00	2.70

Table 4: Scorer-sensitivity audit for the Qwen3-0.6B comparison using Qwen3-235B-A22B-Instruct instead of GPT-5-chat. STAGE remains the top method by average, supporting the stability of the main ranking across automatic evaluators.

Figure 2: Expansion dynamics on the development set. Blue shows the aggregate alignment score, orange shows the number of active objectives, and dotted lines mark stage transitions. The score rises as STAGE expands toward all 15 objectives, indicating that staged admission adds objectives while preserving aggregate progress.

Figure 3: Performance comparison across 16 benchmarks with different

T_{max}

values. Radar charts (top) show performance across different preference categories. Bottom panel displays dataset-level scores with grouped bars for different

T_{max}

and average trend line (red).

Appendix figures & tables4 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Citation	Primary Focus / Description
Natural Reasoning	[ Yuan et al., 2025 ]	Complex reasoning in the wild.
Reasoning-20K	[ KingNish, ]	General reasoning tasks.
Social Reasoning	[ ProlificAI, ]	Social common sense and logic.
Omni-Math	[ Gao et al., 2024 ]	Olympiad-level mathematical problems.
General-Knowledge	[ MuskumPillerum, ]	Broad factual and conceptual queries.
PKU-SafeRLHF	[ Ji et al., 2024 ]	Safety and ethical alignment data.

Appendix

Table 5: Overview of datasets used for training query construction.

Category	Hyperparameter	Value
Trainer	Nodes	1
	GPUs per node	8
	Save frequency	50
	Total Steps	600
Algorithm	Advantage estimator	GAE( $\lambda$ =1, $\gamma$ =1)
	Curriculum Learning	Enabled

Appendix

Table 6: Hyperparameters for the main STAGE experiments implemented with the veRL framework.

#	Prompt Title	Link
0	Preference Label Mapping	Preference Label Mapping
1	Training Dataset Construction	Training Dataset Construction
2	Vanilla Multi-objective PPO	Vanilla Multi-objective PPO
3	Reward Soups	Reward Soups
4	RCS and Curri-DPO Rubric	RCS and Curri-DPO Rubric
5	Scoring for Generative Tasks	Scoring for Generative Tasks

Appendix

Table 10

Display name	Prompt / JSON key
Ethical compliance	ethical
Accuracy	accuracy
Helpfulness	helpfulness
Question assessment	question_assessment
Reasoning quality	reasoning
Multi-aspect analysis	multi_aspect

Appendix

Table 7: Canonical mapping between manuscript preference names and implementation keys.

Multi-Objective Language Model Alignment

Multi-objective language model alignment studies how to balance competing goals such as helpfulness, safety, truthfulness, fairness, and personalization.

See all 92 papers

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment
Apr 27, 2026Wenzhe Xu, Biao Liu, Yiyang Sun +2Meta-Learning Multi-Objective Language Model Alignment
Pref-CTRL: Preference Driven LLM Alignment using Representation Editing
Apr 26, 2026Imranul Ashrafi, Inigo Jauregi Unanue, Massimo PiccardiLLM Alignment Representation Engineering
University of Technology Sydney, Australia

Oct 4, 2026J/K move · Enter open · S save

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee, Kyunghyun Cho, Sangdon Park

GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

LLM Alignment Multi-Objective Language Model Alignment Preference Optimization Multi-Objective Reinforcement Learning RL for Language Models

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Abstract

Multi-Objective Language Model Alignment

Explore similar work

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Meta-Aligner: Bidirectional Preference-Policy Optimization for Multi-Objective LLMs Alignment

Pref-CTRL: Preference Driven LLM Alignment using Representation Editing