Emotional Preferences as Goal-Priority Regulation

Shiqi Liu; Yihua Tan; Hu Fu; Guanyu Qi

cs.LGAug 27, 2026

Emotional Preferences as Goal-Priority Regulation

Authors: Shiqi Liu, Yihua Tan, Hu Fu, Guanyu Qi

Organizations: School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Luoyu Street, Wuhan, 430074, Hubei, China

Abstract

A core question in decision-making for agents is whether the relative priorities of competing lower-level objectives can be determined by emotional preferences autonomously generated by higher-level goals, rather than being externally prespecified. Under changing external environments and evolving internal states, emotions play an important functional role in regulating the relative priorities of competing goals. Inspired by the goal-directed theory of emotion, this paper studies how such preference regulation can be computationally realized through reinforcement learning. We first propose a conception of emergent emotional preference: a high-level goal autonomously induces state-dependent preferences over competing lower-level objectives. This conception is built upon a framework consisting of a multi-objective reinforcement learning inner controller and an outer preference generator. The inner controller provides a repertoire of preference-conditioned goal-directed behaviors, while the outer preference generator learns a mapping from the current state to objective preferences through reinforcement learning on a high-level goal. We operationalize emotional preference as a state-dependent regulation of relative goal priorities that emerges through optimization. Furthermore, we characterize the policy space induced by preference regulation and derive an upper bound on the optimality gap in terms of the representation error of the inner behavioral repertoire. We show that the gap vanishes when the optimal policy can be represented by the available preference-conditioned policies. Experiments in self-constructed multi-objective exploration environments show that the learned preference function exhibits contextual priority switching, graded trade-offs, and temporal persistence, and outperforms the evaluated fixed-preference and handcrafted-preference strategies.

Figures & tables

Panel of Figure 1 — Figure 1: The overall framework for emergent emotional preference, comprising inner MORL and outer emotional preference learning.

GDT construct / mechanism	Computational interpretation in our framework
Valued goal / Goal at stake	The high-level task objective $G$ , e.g., long-term survival
Actual or anticipated stimulus / state	The current or anticipated agent–environment state $s_{t}$
Stimulus–goal discrepancy	A mismatch between the current situation and states favorable to the high-level goal, e.g., low energy or elevated safety risk
Discrepancy-reduction strategies	Preference-conditioned MORL policies representing alternative lower-level goal-directed behaviors
Strategy selection based on expected utility	Preference-conditioned strategy execution through the inner MORL controller
Goal value / Goal competition	The relative priority among competing lower-level objectives, represented by the preference vector $\mathbf{w}_{t}\in\Delta^{m-1}$

Table 1: Correspondence between the Goal-Directed Theory of Emotion and Our Computational Framework

Panel of Figure 2 — Figure 2: The goal-directed computational cycle: a high-level valued goal and a goal-relevant state drive emotional preference regulation, which selects a strategy that produces behavior. $\mathbf{e}_{\theta}(s_{t},{\color[rgb]{0.5,0.5,0.5}G})$ denotes that $\mathbf{e}_{\theta}$ is indeed goal-relevant.

Figure 3: Scatter plot of two-dimensional average reward for various preferences. The rewards are obtained from statistics over 200,000 episodes.

Panel of Figure 4 — Figure 4: Distribution of visited states across learned emotional preference regions in the basic environment.

Panel of Figure 5 — Figure 5: Representative trajectory showing state-dependent emotional preference regulation. The preference shifts between energy- and achievement-oriented priorities as the goal-relevant environmental state changes.

Target Preference	Average Durations	Median	Min	Max	Count
[1.0,0.0]	$2.46\pm 0.85$	3.0	1.0	9.0	1124
[0.9,0.1]	$1.12\pm 0.34$	1.0	1.0	3.0	195
[0.8,0.2]	$1.10\pm 0.33$	1.0	1.0	3.0	234
[0.7,0.3]	$1.12\pm 0.37$	1.0	1.0	4.0	263
[0.6,0.4]	$1.17\pm 0.45$	1.0	1.0	4.0	367
[0.5,0.5]	$1.15\pm 0.46$	1.0	1.0	5.0	443

Table 2: Emotional Preference Persistence Analysis: The duration of the corresponding emotional preference when the target preference value falls within the range of ±0.1. The table records the mean, median, minimum, and maximum durations of emotional preferences, as well as the total number of continuous segments. The specific data is derived from the results of the previous 1,000 trials.

Model	Avg Survival Steps	Avg Help	Avg Batteries	Avg Fruits
Emotional Preference Model	$14.97\pm 4.18$	$0.64\pm 0.48$	$2.11\pm 0.92$	$1.37\pm 1.17$
Survival Model	$16.00\pm 4.08$	$0.03\pm 0.17$	$2.12\pm 0.94$	$0.09\pm 0.35$
Handcraft Model(energy)	$11.81\pm 5.21$	$0.90\pm 0.30$	$1.61\pm 1.16$	$1.47\pm 1.09$
Handcraft Model(battery)	$14.53\pm 4.11$	$1.00\pm 0.00$	$2.10\pm 0.91$	$1.24\pm 1.03$
Fixed Preference [1.0,0.0]	$10.97\pm 5.19$	$1.00\pm 0.00$	$1.43\pm 1.22$	$1.99\pm 1.02$
Fixed Preference [0.9,0.1]	$11.58\pm 5.20$	$1.00\pm 0.00$	$1.55\pm 1.18$	$1.97\pm 1.03$

Table 3: Performance comparison of preference-regulation strategies in the basic environment. The statistical data is derived from 200,000 episodes.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Outer learning rate (outer_lr)	$3\times 10^{-5}$
Soft update coefficient ( $\tau_{soft}$ )	$0.005$
Outer discount factor ( $\gamma_{outer}$ )	$0.99$
Experience replay buffer size (mem_size)	$30000$
Batch size (batch_size)	$256$
Number of sampled weights (weight_num)	$32$

Appendix

Table 4: Outer Network Training Hyperparameters

Panel of Figure 6 — Figure 6: Training Statistics (Mean $\pm$ 3 $\sigma$ across rounds). Training Results with Moving Average (window=100).

Panel of Figure 7 — Figure 7: Training statistics during outer preference learning in five randomized runs. Curves report the mean $\pm 3\sigma$ , with a moving-average window of 100 episodes.

Figure 8: Average three-dimensional reward vectors obtained under different fixed preferences in the advanced environment. Results are computed over 20,000 evaluation episodes.

Model	Avg Steps	Avg Batteries	Avg Fruit	Avg Help	Avg Danger Steps
Emotional Preference Model	$14.81\pm 9.56$	$1.14\pm 1.12$	$0.76\pm 1.24$	$0.59\pm 0.49$	$0.61\pm 1.01$
Handcrafted-Energy Preference	$13.62\pm 8.86$	$1.04\pm 1.13$	$1.78\pm 1.67$	$0.68\pm 0.47$	$0.51\pm 0.92$
Handcrafted-Battery Preference	$14.39\pm 9.44$	$1.19\pm 1.12$	$1.12\pm 1.54$	$0.64\pm 0.48$	$0.75\pm 1.10$
Survival Model	$16.26\pm 9.04$	$1.18\pm 1.13$	$0.03\pm 0.25$	$0.64\pm 0.48$	$0.44\pm 0.85$
Fixed Preference $(0.0,0.0,1.0)$	$9.40\pm 6.53$	$0.36\pm 0.76$	$0.23\pm 0.66$	$0.88\pm 0.32$	$0.24\pm 0.67$
Fixed Preference $(0.0,0.2,0.8)$	$12.29\pm 9.48$	$1.08\pm 1.15$	$0.40\pm 0.91$	$0.90\pm 0.29$	$0.25\pm 0.62$

Appendix

Table 5: Comparison of multi-objective performance between the proposed emotional preference model, handcrafted preference-regulation baselines, the direct survival policy, and fixed-preference strategies in the advanced environment.

Panel of Figure 9 — Figure 9: Distribution of visited states across learned emotional-preference regions in the advanced environment. Continuous preference outputs are assigned to the nearest reference preference for visualization.

Panel of Figure 10 — Figure 10: Representative episode illustrating state-dependent emotional preference regulation in the advanced environment.

Target Preference	Average Duration	Median	Min	Max	Count
$[0.00,0.00,1.00]$	$3.99\pm 2.07$	4.0	1.0	12.0	92
$[0.00,0.20,0.80]$	$1.60\pm 0.86$	1.0	1.0	4.0	20
$[0.00,0.40,0.60]$	$1.00\pm 0.00$	1.0	1.0	1.0	11
$[0.00,0.60,0.40]$	$1.00\pm 0.00$	1.0	1.0	1.0	11
$[0.00,0.80,0.20]$	$1.07\pm 0.26$	1.0	1.0	2.0	14
$[0.00,1.00,0.00]$	$3.28\pm 3.21$	2.0	1.0	23.0	173

Appendix

Table 6: Preference persistence analysis in the advanced environment. Durations are measured for states whose learned preference lies within a Chebyshev distance of

\pm 0.2

from each reference preference. Statistics are computed from the evaluation trajectories.

Application Setting	Potential Benefit	Potential Risk
Assistive and Home Robotics	Context-dependent regulation of energy, task completion, and safety priorities may improve adaptation to changing household conditions.	The learned preference may prioritize self-preservation or task completion in situations where human safety should dominate.
Medical and Care Systems	A preference representation may expose explicit trade-offs among competing operational objectives and facilitate human auditing.	Incorrect preference regulation could produce inappropriate prioritization, unequal treatment, or unsafe behavior. Such systems therefore require domain-specific constraints and human oversight.
Industrial and Collaborative Robotics	Dynamic regulation of productivity, energy efficiency, and safety may help adapt behavior to changing operational conditions.	Reward exploitation or inappropriate activation of learned skills could increase physical or operational risk.
Autonomous Decision Systems	Explicit preference trajectories provide an additional interface for monitoring and intervention.	Out-of-distribution states may trigger unexpected preference shifts, particularly when the outer objective is underspecified.

Appendix

Table 7: Potential benefits and risks of state-dependent emotional preference regulation across representative application settings.

Explore similar work

Mar 24, 2026

Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts

Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective function. In contrast, most computational decision-making and multi-objective RL methods assume static preference weights or a known scalar reward. In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. Specifically, we propose Dynamic Preference Inference (DPI), a cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. We instantiate DPI as a variational preference inference module trained jointly with a preference-conditioned actor-critic, using vector-valued returns as evidence about latent trade-offs. In queueing, maze, and multi-objective continuous-control environments with event-driven changes in objectives, DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and heuristic envelope baselines.

Xianwei Cao, Dou Quan, Zhenliang Zhang +1

School of Artificial Intelligence, Xidian University, Xi’an, 710071, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

Preference Learning Multi-Objective Reinforcement Learning User Preference Modeling

Ask PDF

Jul 31, 2026

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

Reinforcement Learning (RL) systems are typically trained using a single, well-specified scalar reward function. However, real-world decision-making tasks often involve multiple, competing objectives, such as performance versus efficiency, where ground-truth reward functions are difficult to specify or inaccessible. While Multi-Objective RL (MORL) addresses such trade-offs by modeling rewards as vectors, existing approaches typically assume access to a well-specified reward function for each objective, inheriting the same challenges faced by single-objective RL. Meanwhile, Preference-based RL (PbRL) has shown great potential in solving complex tasks without access to a pre-defined reward function through reward learning from human feedback, yet has largely been studied in single-objective settings. In this work, we bridge this gap with LEMUR: Learning to Align with Multi-Objective Reinforcement Learning with Preference feedback, a novel framework where an agent interactively learns from the preferences of multiple humans to learn optimal multi-objective policies. Our approach jointly learns policies and multiple objective-specific reward models from human feedback, enabling agents to effectively balance competing objectives during learning. We evaluate LEMUR on a variety of benchmark multi-objective tasks, and empirical results demonstrate its superior performance over baseline methods. Our method presents a promising direction for solving multi-objective decision-making tasks without pre-defined reward functions.

Manith Adikari, Bei Peng, Samuele Vinanzi +1

Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1

Multi-Objective Reinforcement Learning Reinforcement Learning Reward Design for RL Preference-Based RL RL from Human Feedback

Ask PDF

Oct 9, 2026

EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning

Multimodal emotion preference learning uses pairwise comparisons to identify which open-ended emotion descriptions align more closely with human judgments. However, existing methods primarily focus on the final preference decision and provide little explicit supervision for localizing and correcting erroneous content. To address this limitation, we propose EmoGavel, a Grounding-Aware reinforcement learning framework with Verifiable rewards for multimodal Emotion preference Learning. We first introduce grounded supervised fine-tuning (Grounded SFT), which exploits controlled-error construction information to jointly learn preference prediction, error localization, repair, and missing-evidence recovery. We then optimize these verifiable outcomes with grounding-aware group relative policy optimization (GA-GRPO), combining preference-gated grounding rewards with dynamic zero-variance rescue to prioritize preference correctness and recover learning signals from all-wrong groups. Experiments on EmoPrefer and controlled-error evaluation sets show that EmoGavel improves both natural preference prediction and controlled-error discrimination while maintaining strong error-grounding performance. Our code and model checkpoints will be released once the paper is accepted.

Zilong Huang, Kong Aik Lee, Zezhong Jin +2

The Hong Kong Polytechnic University, Hong Kong SAR, China · Kyoto University, Japan

Pairwise Preference Learning Reinforcement Learning with Verifiable Rewards Multimodal Grounding

Ask PDF