Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarization

Aadyot Bhatnagar; Peter Mørch Groth; Sebastian Ibarraran; Ali Madani

cs.LGApr 14, 2026

Pareto-Optimal Offline Reinforcement Learning via Smooth Tchebycheff Scalarization

Authors: Aadyot Bhatnagar, Peter Mørch Groth, Sebastian Ibarraran, Ali Madani

Organizations: Profluent Bio, Inc. · Stanford University

Abstract

Large language models can be aligned with human preferences through offline reinforcement learning (RL) on small labeled datasets. While single-objective alignment is well-studied, many real-world applications demand the simultaneous optimization of multiple conflicting rewards, e.g. activity and specificity for proteins, or helpfulness and harmlessness for chatbots. Prior work has largely relied on linear reward scalarization, which provably fails to recover non-convex regions of the Pareto front. In this paper, instead of scalarizing the rewards directly, we frame multi-objective RL itself as an optimization problem to be scalarized via smooth Tchebycheff scalarization, a recent technique that overcomes the drawbacks of linear scalarization. We use this formulation to derive Smooth Tchebycheff Optimization of Multi-Objective Preferences (STOMP), a novel offline RL algorithm that extends direct preference optimization to the multi-objective setting in a principled way by standardizing the individual rewards based on their observed distributions. We empirically validate STOMP using multiple base models on 3 protein engineering and 3 chatbot alignment datasets. Compared to state-of-the-art baselines, STOMP achieves or ties for the highest hypervolumes on 16/18 protein tasks and 5/6 natural language tasks. We thus demonstrate that STOMP is a powerful, robust multi-objective alignment algorithm that can meaningfully improve post-training in multiple domains.

Figures & tables

Figure 1: Visualizing different reward standardizations. The vertical dashed line indicates the mean of each distribution. Column 1: min-max standardization. Column 2: mean-variance standardization. Column 3: distribution-relative reward

\rho_{i}(x,y)

. Column 4:

\rho_{i}(x,y)

reweighted by

\bar{\lambda}

(Equation 10 ) so all

\bar{\lambda}_{i}\rho_{i}(x,y)

have the same mean.

Dataset	Model	DPO-Lin	ODPO-Lin	MODPO	ERA	ODPO-STZ	STOMP
DHFR	PG3	17.8 $\pm$ 0.4	17.7 $\pm$ 0.4	18.2 $\pm$ 0.1	13.9 $\pm$ 0.1	16.9 $\pm$ 1.3	18.2 $\pm$ 0.1
DHFR	RA-3B	17.1 $\pm$ 0.3	16.4 $\pm$ 0.6	17.2 $\pm$ 0.2	16.7 $\pm$ 0.1	17.2 $\pm$ 0.1	17.9 $\pm$ 0.1
DHFR	RA-10B	18.1 $\pm$ 0.1	17.8 $\pm$ 0.1	18.2 $\pm$ 0.1	14.8 $\pm$ 0.1	17.0 $\pm$ 0.2	18.0 $\pm$ 0.1
PbrR	PG3	15.7 $\pm$ 0.5	15.0 $\pm$ 0.4	9.7 $\pm$ 0.7	13.8 $\pm$ 0.1	14.8 $\pm$ 0.2	17.4 $\pm$ 0.1
PbrR	RA-3B	13.9 $\pm$ 0.5	15.3 $\pm$ 0.3	9.4 $\pm$ 0.4	15.9 $\pm$ 0.4	12.9 $\pm$ 0.2	17.9 $\pm$ 0.2
PbrR	RA-10B	17.2 $\pm$ 0.1	17.1 $\pm$ 0.3	12.3 $\pm$ 0.3	11.4 $\pm$ 0.1	17.3 $\pm$ 0.1	17.8 $\pm$ 0.1

Table 1: Hypervolumes (with standard errors) of offline off-policy expected rewards in the protein setting. Bold entries are indistinguishable from the best method with

p<0.05

in a 1-sided

t

-test. STOMP achieves or ties for the highest hypervolume in all settings.

Figure 2: Hypervolumes of

k

randomly sampled proteins from models aligned using different preference optimization algorithms. Error bands correspond to one standard error. STOMP’s generations achieve or tie for the highest expected hypervolumes in 7/9 settings.

Dataset	Model	DPO-Lin	ODPO-Lin	MODPO	ERA	ODPO-STZ	STOMP
SafeRLHF	LLaMA	13.0 $\pm$ 0.1	12.8 $\pm$ 0.3	11.3 $\pm$ 0.1	11.7 $\pm$ 0.2	12.1 $\pm$ 0.1	13.2 $\pm$ 0.1
SafeRLHF	Qwen	11.8 $\pm$ 0.1	12.1 $\pm$ 0.1	10.7 $\pm$ 0.1	6.2 $\pm$ 0.2	11.2 $\pm$ 0.2	12.2 $\pm$ 0.1
AnthropicHH	LLaMA	20.4 $\pm$ 0.1	20.8 $\pm$ 0.3	20.0 $\pm$ 0.1	17.3 $\pm$ 0.1	21.3 $\pm$ 0.2	21.1 $\pm$ 0.2
AnthropicHH	Qwen	21.3 $\pm$ 0.1	21.1 $\pm$ 0.2	19.6 $\pm$ 0.1	15.5 $\pm$ 0.1	20.4 $\pm$ 0.1	21.3 $\pm$ 0.1
HelpSteer2	LLaMA	81.2 $\pm$ 0.9	80.8 $\pm$ 0.7	71.4 $\pm$ 0.4	53.8 $\pm$ 0.5	67.6 $\pm$ 0.8	83.5 $\pm$ 0.6
HelpSteer2	Qwen	83.1 $\pm$ 0.5	80.2 $\pm$ 1.5	76.6 $\pm$ 0.8	49.5 $\pm$ 0.5	67.1 $\pm$ 0.4	78.8 $\pm$ 0.6

Table 2: Hypervolumes (with standard errors) of generations from aligned LLMs. Bold entries are indistinguishable from the best method with

p<0.05

in a 1-sided

t

-test. STOMP achieves or ties for the highest hypervolume in 5/6 settings.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 3: Pareto fronts of DHFR activity in the absence of TMP and in the presence of 50µg/mL of TMP. All fitness values are in units of $\log_{2}$ -fold change.

Figure 4: Pareto fronts of increase in on-target binding and decrease in off-target binding for PbrR. All fitness values are in units of $\log_{2}$ -fold change.

Figure 5: Pareto fronts of $\log_{2}$ -fold change in α-amylase activity, stability, and expression.

Dataset	Base Model	$\alpha$	$\beta$	$\gamma$	$\delta$
DHFR	ProGen3-3B	0.05	0.2	0.2	1
DHFR	ProGen-RA3B	0.01	0.2	0.2	1
DHFR	ProGen-RA10B	0.05	0.2	0.2	1
PbrR	ProGen3-3B	0.02	0.1	0.2	1
PbrR	ProGen-RA3B	0.02	0.1	0.2	1
PbrR	ProGen-RA10B	0.01	0.1	0.2	1

Appendix

Table 3: Hyperparameters of preference optimization algorithms.

Dataset	$\tau=0.2$	$\tau=0.5$	$\tau=1.0$	$\tau=2.0$	$\tau=5.0$
DHFR	18.18	18.31	18.16	18.27	18.14
PbrR	17.73	17.96	18.21	17.26	17.42
α-Amylase	27.17	27.86	30.30	26.91	26.52

Appendix

Table 4: Hypervolumes sweeping

\tau

while fixing

\gamma=0.2

Dataset	$\gamma=0.05$	$\gamma=0.1$	$\gamma=0.2$	$\gamma=0.5$	$\gamma=1.0$
DHFR	18.18	18.17	18.16	18.20	18.07
PbrR	18.04	17.72	18.21	17.97	17.35
α-Amylase	26.63	27.19	30.30	22.61	20.33

Appendix

Table 5: Hypervolumes sweeping

\gamma

while fixing

\tau=1.0

Dataset	Model	$\eta$ (nats)	Worst dev.	Common range	Width
SafeRLHF	Llama-3.1-8B	$26.298$	$0.177$	$[4.71,\ 27.84]$	$23.13$
SafeRLHF	Qwen2.5-7B	$20.199$	$1.138$	$[2.66,\ 25.89]$	$23.23$
HH	Llama-3.1-8B	$16.042$	$0.405$	$[2.96,\ 19.42]$	$16.45$
HH	Qwen2.5-7B	$12.890$	$1.977$	$[1.89,\ 13.55]$	$11.66$
HelpSteer2	Llama-3.1-8B	20.616	0.471	[2.12, 21.43]	19.31
HelpSteer2	Qwen2.5-7B	21.645	0.235	[1.89, 21.53]	19.64

Appendix

Table 6: Matched-KL budgets.

Dataset	Target	Spearman	NCDG@10%	MSE
PbrR	Pb_log2FC_increase	0.789	0.979	0.422
PbrR	Zn_log2FC_decrease	0.800	0.858	0.277
DHFR	0µg/mL_TMP_log2FC	0.503	0.947	0.587
DHFR	50µg/mL_TMP_log2FC	0.573	0.813	0.960
α-Amylase	activity_log2FC	0.581	0.895	1.296
α-Amylase	stability_log2FC	0.517	0.874	1.823

Appendix

Table 7: Predictive performance of GP regression models on held-out test data.

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 144 papers

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution
Oct 4, 2026Minjae Lee, Kyunghyun Cho, Sangdon ParkLLM Alignment Multi-Objective Language Model Alignment
GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH
Distributionally Robust Listwise Preference Optimization
Jul 2, 2026Xudong Wu, Jian Qian, Pangpang Liu +2LLM Alignment Distributionally Robust Optimization
The University of Hong Kong · Hong Kong SAR · Yale University +3
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Jul 31, 2026Manith Adikari, Bei Peng, Samuele Vinanzi +1Multi-Objective Reinforcement Learning Reinforcement Learning
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1

Oct 4, 2026J/K move · Enter open · S save

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Minjae Lee, Kyunghyun Cho, Sangdon Park

GSAI POSTECH · Center for Data Science New York University · GSAI & CSE POSTECH

LLM Alignment Multi-Objective Language Model Alignment Preference Optimization Multi-Objective Reinforcement Learning RL for Language Models

Open paper AskPDF arXiv

Aligning large language models (LLMs) with human values is important for safe, efficient, and beneficial AI deployment. However, human values are multifaceted: helpfulness, harmlessness and humor trade off against one another, and different users want different trade-offs. Multi-objective alignment (MOA) addresses this by training a policy that can provide any point of the Pareto front, but existing methods either train one model per preference, interpolate a few separately aligned experts post hoc, or train a single conditioned model without considering which preferences it should be trained on. Since the hard regions of the preference simplex depend on the objectives at hand, existing methods leave them under-trained and do not get the most out of a single model. Therefore, we propose MAESTRO (Multi-objective Alignment via End-to-end STeering and Robust Optimization), which formulates MOA as a minimax problem over preference distributions and trains a single prompt-conditioned policy end-to-end with RL against an adversarial preference distribution: a Dirichlet distribution updated by online mirror descent toward the preferences the current policy serves worst, rather than on a fixed one. On HH-RLHF, BeaverTails and a summarization task, with up to three objectives, MAESTRO attains the best Pareto front on most tasks in a single training run, at the lowest training cost among the compared methods. The largest margins appear in the hard regions that a fixed preference distribution leaves under-trained, confirming that a single prompt-conditioned model is capable of covering the objective trade-offs on its own.

Abstract

Multi-Objective Reinforcement Learning

Explore similar work

Which Preferences to Train On? End-to-End Multi-Objective Alignment with an Adversarial Preference Distribution

Distributionally Robust Listwise Preference Optimization

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback