Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Chenyang Zhao; Vinny Cahill; Ivana Dusparic

cs.AIFeb 24, 2026

Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

Authors: Chenyang Zhao, Vinny Cahill, Ivana Dusparic

Organizations: Trinity College Dublin, Dublin, Ireland

Abstract

Reward design has been one of the central challenges for real world reinforcement learning (RL) deployment, especially in settings with multiple objectives. Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes. More recently, RL from AI feedback (RLAIF) has demonstrated that large language models (LLMs) can generate preference labels at scale, mitigating the reliance on human annotators. However, existing RLAIF work typically focuses only on single-objective tasks, leaving the open question of how RLAIF handles systems that involve multiple objectives. In such systems trade-offs among conflicting objectives are difficult to specify, and policies risk collapsing into optimizing for a dominant goal. In this paper, we explore the extension of the RLAIF paradigm to multi-objective self-adaptive systems. We show that multi-objective RLAIF can produce policies that yield balanced trade-offs reflecting different user priorities without laborious reward engineering. We argue that integrating RLAIF into multi-objective RL offers a scalable path toward user-aligned policy learning in domains with inherently conflicting objectives.

Figures & tables

Figure 1. Our framework of RLAIF for multi-objective tasks: the policy interacts with the environment, transitions enter the replay buffer $\mathcal{B}$ ; segment pairs are annotated by an LLM and stored into a preference buffer $\mathcal{D}$ ; the reward model $r_{\psi}$ updates from $\mathcal{D}$ ; re-scores $\mathcal{B}$ , and the policy is trained with RL algorithms with the updated replay buffer. During annotation, we make the optimization criteria explicit, stating the task objectives and desired trade-offs, so the learned reward reflects user expectations and aligns behaviour across multiple objectives without bespoke reward engineering.

Figure 2. An illustrated workflow of the preference annotation process. The annotator is given considered objectives, the overall goal, and descriptions of the sampled pair of segments

\{\sigma^{1},\sigma^{2}\}

, and outputs the preference label

y\in\{0,1,2\}

Figure 3. An exemplar text description of a traffic segment with length 1 in the task of balancing traffic throughput and environmental impact. The text description includes the observations, explained with their semantic meanings, as well as information on throughput and carbon emissions.

Panel of Figure 4 — Figure 4. Left and middle: Average traffic throughput and CO2 emission in throughput-emission scenario through learning process. RLAIF learns better performances compared with single objective baselines, and falls short only of the Linear baseline, which requires reward engineering. Right: Comparison between 5 policies trained with different user specifications in lane priorities scenario with instructions towards different lane priorities.

Multi-Objective Reinforcement Learning

Multi-objective reinforcement learning studies how agents can balance competing goals and learn policies for different trade-offs.

See all 145 papers

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Jul 31, 2026Manith Adikari, Bei Peng, Samuele Vinanzi +1Multi-Objective Reinforcement Learning Reinforcement Learning
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1
Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution
Oct 26, 2026Zhenshuo Zhang, Ruoxuan Xiong, Hongyang R. ZhangMulti-Objective Reinforcement Learning Reinforcement Learning
Northeastern University, Boston, MA · Emory University, Atlanta, GA
MAPL: Multi-Objective Preference Learning for Robot Locomotion
Jun 24, 2026Xiyue Chen, Muhan Lin, Shuyang Shi +1Multi-Objective Reinforcement Learning Reward Modeling
Department of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47906, USA · Department of Computer Science, Purdue University, West Lafayette, IN 47906, USA · independent researcher

Jul 31, 2026J/K move · Enter open · S save

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

Manith Adikari, Bei Peng, Samuele Vinanzi, Angelo Cangelosi

Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield+1

Multi-Objective Reinforcement Learning Reinforcement Learning Reward Design for RL Preference-Based RL RL from Human Feedback

Open paper AskPDF arXiv

Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.

Abstract

Multi-Objective Reinforcement Learning

Explore similar work

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

Multi-Objective Reinforcement Learning Using Routed Ensembles and Trajectory Attribution

MAPL: Multi-Objective Preference Learning for Robot Locomotion