Emotional Preferences as Goal-Priority Regulation

Shiqi Liu; Yihua Tan; Hu Fu; Guanyu Qi

cs.LGAug 27, 2026

Emotional Preferences as Goal-Priority Regulation

Authors: Shiqi Liu, Yihua Tan, Hu Fu, Guanyu Qi

Organizations: School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Luoyu Street, Wuhan, 430074, Hubei, China

Abstract

A core question in decision-making for agents is whether the relative priorities of competing lower-level objectives can be determined by emotional preferences autonomously generated by higher-level goals, rather than being externally prespecified. Under changing external environments and evolving internal states, emotions play an important functional role in regulating the relative priorities of competing goals. Inspired by the goal-directed theory of emotion, this paper studies how such preference regulation can be computationally realized through reinforcement learning. We first propose a conception of emergent emotional preference: a high-level goal autonomously induces state-dependent preferences over competing lower-level objectives. This conception is built upon a framework consisting of a multi-objective reinforcement learning inner controller and an outer preference generator. The inner controller provides a repertoire of preference-conditioned goal-directed behaviors, while the outer preference generator learns a mapping from the current state to objective preferences through reinforcement learning on a high-level goal. We operationalize emotional preference as a state-dependent regulation of relative goal priorities that emerges through optimization. Furthermore, we characterize the policy space induced by preference regulation and derive an upper bound on the optimality gap in terms of the representation error of the inner behavioral repertoire. We show that the gap vanishes when the optimal policy can be represented by the available preference-conditioned policies. Experiments in self-constructed multi-objective exploration environments show that the learned preference function exhibits contextual priority switching, graded trade-offs, and temporal persistence, and outperforms the evaluated fixed-preference and handcrafted-preference strategies.

Figures & tables

Panel of Figure 1 — Figure 1: The overall framework for emergent emotional preference, comprising inner MORL and outer emotional preference learning.

GDT construct / mechanism	Computational interpretation in our framework
Valued goal / Goal at stake	The high-level task objective $G$ , e.g., long-term survival
Actual or anticipated stimulus / state	The current or anticipated agent–environment state $s_{t}$
Stimulus–goal discrepancy	A mismatch between the current situation and states favorable to the high-level goal, e.g., low energy or elevated safety risk
Discrepancy-reduction strategies	Preference-conditioned MORL policies representing alternative lower-level goal-directed behaviors
Strategy selection based on expected utility	Preference-conditioned strategy execution through the inner MORL controller
Goal value / Goal competition	The relative priority among competing lower-level objectives, represented by the preference vector $\mathbf{w}_{t}\in\Delta^{m-1}$

Table 1: Correspondence between the Goal-Directed Theory of Emotion and Our Computational Framework

Panel of Figure 2 — Figure 2: The goal-directed computational cycle: a high-level valued goal and a goal-relevant state drive emotional preference regulation, which selects a strategy that produces behavior. $\mathbf{e}_{\theta}(s_{t},{\color[rgb]{0.5,0.5,0.5}G})$ denotes that $\mathbf{e}_{\theta}$ is indeed goal-relevant.

Figure 3: Scatter plot of two-dimensional average reward for various preferences. The rewards are obtained from statistics over 200,000 episodes.

Panel of Figure 4 — Figure 4: Distribution of visited states across learned emotional preference regions in the basic environment.

Panel of Figure 5 — Figure 5: Representative trajectory showing state-dependent emotional preference regulation. The preference shifts between energy- and achievement-oriented priorities as the goal-relevant environmental state changes.

Target Preference	Average Durations	Median	Min	Max	Count
[1.0,0.0]	$2.46\pm 0.85$	3.0	1.0	9.0	1124
[0.9,0.1]	$1.12\pm 0.34$	1.0	1.0	3.0	195
[0.8,0.2]	$1.10\pm 0.33$	1.0	1.0	3.0	234
[0.7,0.3]	$1.12\pm 0.37$	1.0	1.0	4.0	263
[0.6,0.4]	$1.17\pm 0.45$	1.0	1.0	4.0	367
[0.5,0.5]	$1.15\pm 0.46$	1.0	1.0	5.0	443

Table 2: Emotional Preference Persistence Analysis: The duration of the corresponding emotional preference when the target preference value falls within the range of ±0.1. The table records the mean, median, minimum, and maximum durations of emotional preferences, as well as the total number of continuous segments. The specific data is derived from the results of the previous 1,000 trials.

Model	Avg Survival Steps	Avg Help	Avg Batteries	Avg Fruits
Emotional Preference Model	$14.97\pm 4.18$	$0.64\pm 0.48$	$2.11\pm 0.92$	$1.37\pm 1.17$
Survival Model	$16.00\pm 4.08$	$0.03\pm 0.17$	$2.12\pm 0.94$	$0.09\pm 0.35$
Handcraft Model(energy)	$11.81\pm 5.21$	$0.90\pm 0.30$	$1.61\pm 1.16$	$1.47\pm 1.09$
Handcraft Model(battery)	$14.53\pm 4.11$	$1.00\pm 0.00$	$2.10\pm 0.91$	$1.24\pm 1.03$
Fixed Preference [1.0,0.0]	$10.97\pm 5.19$	$1.00\pm 0.00$	$1.43\pm 1.22$	$1.99\pm 1.02$
Fixed Preference [0.9,0.1]	$11.58\pm 5.20$	$1.00\pm 0.00$	$1.55\pm 1.18$	$1.97\pm 1.03$

Table 3: Performance comparison of preference-regulation strategies in the basic environment. The statistical data is derived from 200,000 episodes.

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
Outer learning rate (outer_lr)	$3\times 10^{-5}$
Soft update coefficient ( $\tau_{soft}$ )	$0.005$
Outer discount factor ( $\gamma_{outer}$ )	$0.99$
Experience replay buffer size (mem_size)	$30000$
Batch size (batch_size)	$256$
Number of sampled weights (weight_num)	$32$

Appendix

Table 4: Outer Network Training Hyperparameters

Panel of Figure 6 — Figure 6: Training Statistics (Mean $\pm$ 3 $\sigma$ across rounds). Training Results with Moving Average (window=100).

Panel of Figure 7 — Figure 7: Training statistics during outer preference learning in five randomized runs. Curves report the mean $\pm 3\sigma$ , with a moving-average window of 100 episodes.

Figure 8: Average three-dimensional reward vectors obtained under different fixed preferences in the advanced environment. Results are computed over 20,000 evaluation episodes.

Model	Avg Steps	Avg Batteries	Avg Fruit	Avg Help	Avg Danger Steps
Emotional Preference Model	$14.81\pm 9.56$	$1.14\pm 1.12$	$0.76\pm 1.24$	$0.59\pm 0.49$	$0.61\pm 1.01$
Handcrafted-Energy Preference	$13.62\pm 8.86$	$1.04\pm 1.13$	$1.78\pm 1.67$	$0.68\pm 0.47$	$0.51\pm 0.92$
Handcrafted-Battery Preference	$14.39\pm 9.44$	$1.19\pm 1.12$	$1.12\pm 1.54$	$0.64\pm 0.48$	$0.75\pm 1.10$
Survival Model	$16.26\pm 9.04$	$1.18\pm 1.13$	$0.03\pm 0.25$	$0.64\pm 0.48$	$0.44\pm 0.85$
Fixed Preference $(0.0,0.0,1.0)$	$9.40\pm 6.53$	$0.36\pm 0.76$	$0.23\pm 0.66$	$0.88\pm 0.32$	$0.24\pm 0.67$
Fixed Preference $(0.0,0.2,0.8)$	$12.29\pm 9.48$	$1.08\pm 1.15$	$0.40\pm 0.91$	$0.90\pm 0.29$	$0.25\pm 0.62$

Appendix

Table 5: Comparison of multi-objective performance between the proposed emotional preference model, handcrafted preference-regulation baselines, the direct survival policy, and fixed-preference strategies in the advanced environment.

Panel of Figure 9 — Figure 9: Distribution of visited states across learned emotional-preference regions in the advanced environment. Continuous preference outputs are assigned to the nearest reference preference for visualization.

Panel of Figure 10 — Figure 10: Representative episode illustrating state-dependent emotional preference regulation in the advanced environment.

Target Preference	Average Duration	Median	Min	Max	Count
$[0.00,0.00,1.00]$	$3.99\pm 2.07$	4.0	1.0	12.0	92
$[0.00,0.20,0.80]$	$1.60\pm 0.86$	1.0	1.0	4.0	20
$[0.00,0.40,0.60]$	$1.00\pm 0.00$	1.0	1.0	1.0	11
$[0.00,0.60,0.40]$	$1.00\pm 0.00$	1.0	1.0	1.0	11
$[0.00,0.80,0.20]$	$1.07\pm 0.26$	1.0	1.0	2.0	14
$[0.00,1.00,0.00]$	$3.28\pm 3.21$	2.0	1.0	23.0	173

Appendix

Table 6: Preference persistence analysis in the advanced environment. Durations are measured for states whose learned preference lies within a Chebyshev distance of

\pm 0.2

from each reference preference. Statistics are computed from the evaluation trajectories.

Application Setting	Potential Benefit	Potential Risk
Assistive and Home Robotics	Context-dependent regulation of energy, task completion, and safety priorities may improve adaptation to changing household conditions.	The learned preference may prioritize self-preservation or task completion in situations where human safety should dominate.
Medical and Care Systems	A preference representation may expose explicit trade-offs among competing operational objectives and facilitate human auditing.	Incorrect preference regulation could produce inappropriate prioritization, unequal treatment, or unsafe behavior. Such systems therefore require domain-specific constraints and human oversight.
Industrial and Collaborative Robotics	Dynamic regulation of productivity, energy efficiency, and safety may help adapt behavior to changing operational conditions.	Reward exploitation or inappropriate activation of learned skills could increase physical or operational risk.
Autonomous Decision Systems	Explicit preference trajectories provide an additional interface for monitoring and intervention.	Out-of-distribution states may trigger unexpected preference shifts, particularly when the outer objective is underspecified.

Appendix

Table 7: Potential benefits and risks of state-dependent emotional preference regulation across representative application settings.

Explore similar work

Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts
Mar 24, 2026Xianwei Cao, Dou Quan, Zhenliang Zhang +1Preference Learning Multi-Objective Reinforcement Learning
School of Artificial Intelligence, Xidian University, Xi’an, 710071, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback
Jul 31, 2026Manith Adikari, Bei Peng, Samuele Vinanzi +1Multi-Objective Reinforcement Learning Reinforcement Learning
Department of Computer Science, University of Manchester · Centre for Robotics & AI, University of Manchester · School of Computer Science, University of Sheffield +1
EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning
Oct 9, 2026Zilong Huang, Kong Aik Lee, Zezhong Jin +2Pairwise Preference Learning Reinforcement Learning with Verifiable Rewards
The Hong Kong Polytechnic University, Hong Kong SAR, China · Kyoto University, Japan

Mar 24, 2026J/K move · Enter open · S save

Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts

Xianwei Cao, Dou Quan, Zhenliang Zhang, Shuang Wang

School of Artificial Intelligence, Xidian University, Xi’an, 710071, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China

Preference Learning Multi-Objective Reinforcement Learning User Preference Modeling

Open paper AskPDF arXiv

Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective function. In contrast, most computational decision-making and multi-objective RL methods assume static preference weights or a known scalar reward. In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. Specifically, we propose Dynamic Preference Inference (DPI), a cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. We instantiate DPI as a variational preference inference module trained jointly with a preference-conditioned actor-critic, using vector-valued returns as evidence about latent trade-offs. In queueing, maze, and multi-objective continuous-control environments with event-driven changes in objectives, DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and heuristic envelope baselines.

Abstract

Explore similar work

Learning What Matters Now: Dynamic Preference Inference under Contextual Shifts

LEMUR: Learning to Align with Multi-Objective Reinforcement Learning from Preference Feedback

EmoGavel: Grounding-Aware Reinforcement Learning with Verifiable Rewards for Multimodal Emotion Preference Learning