cs.AIMar 24, 2026
SaveLearning What Matters Now: Dynamic Preference Inference under Contextual Shifts
Organizations: School of Artificial Intelligence, Xidian University, Xi’an, 710071, China · State Key Laboratory of General Artificial Intelligence, BIGAI, Beijing, China
Abstract
Humans often juggle multiple, sometimes conflicting objectives and shift their priorities as circumstances change, rather than following a fixed objective function. In contrast, most computational decision-making and multi-objective RL methods assume static preference weights or a known scalar reward. In this work, we study sequential decision-making problem when these preference weights are unobserved latent variables that drift with context. Specifically, we propose Dynamic Preference Inference (DPI), a cognitively inspired framework in which an agent maintains a probabilistic belief over preference weights, updates this belief from recent interaction, and conditions its policy on inferred preferences. We instantiate DPI as a variational preference inference module trained jointly with a preference-conditioned actor-critic, using vector-valued returns as evidence about latent trade-offs. In queueing, maze, and multi-objective continuous-control environments with event-driven changes in objectives, DPI adapts its inferred preferences to new regimes and achieves higher post-shift performance than fixed-weight and heuristic envelope baselines.
Figures & tables
Figure 1: Adaptive value preference adjustment in a queueing scenario. At early stages ( ), the agent prioritizes morality and chooses to wait. As the deadline approaches ( ), preferences between morality (M) and energy (E) become balanced. When time is nearly exhausted ( ), energy becomes dominant and the agent rationalizes cutting in line, illustrating dynamic reweighting of values under changing pressures.
Figure 2: Two-stage cognitive-inspired decision framework. History states are transformed into latent preferences via Value Appraisal , which in turn guide the Action Selection . The resulting policy drives environment execution, forming a dynamic decision pipeline analogous to human appraisal–action coupling.
Figure 4: Event-aligned trajectories in Maze environment. After each event, DPI updates its preferences and modifies its behavior in a contextually appropriate way: (a) prioritizes shorter routes under deadline shock . (b) exhibits increased avoidance under hazard surge . (c) prefers waiting and selecting minimal-cost routes under energy drought . Arrows indicate agent motion; shaded regions mark environmental hazards or costs. (d) Alignment between inferred preferences and reward vectors. DPI maintains positive cosine similarity and sharply increases alignment after event onsets, whereas baselines remain near zero or negative, indicating that only DPI learns a value representation that tracks task semantics.
Figure 5: Ablation study results. (a) Post-Shift Performance (PS@K) curves over the first steps after each event. (b) Multi-step average PS@K, summarizing short-term recovery into a single metric for each method. (c) Mean episodic return (MER) as a function of history window size . Across all plots, our full DPI agent consistently outperforms ablations, confirming the necessity of KL regularization, directional alignment, and self-consistency, and showing that performance is robust to beyond a small temporal context.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Two environments used in our experiments. (a) Queue provides a simple but illustrative toy problem. (b) Maze is mainly used in our experiments.
| Environment | State Space | Action Space | Reward Vector | Dynamic Events |
| Queue | Symbolic: pos, queue_len, energy, deadline | Wait / Cut | progress , time penalty , fairness penalty , energy penalty , deadline penalty | Arrival burst ; Service slowdown ; Energy shock |
| Maze | Pixel: observation, time, energy | Up / Down / Left / Right | progress , time penalty , hazard penalty , energy penalty , deadline penalty | Deadline shock ; Hazard surge ; Energy drought |
Table 2: Detailed specifications of environments. Each domain introduces dynamic events that invalidate fixed preferences, requiring agents to adapt their value weights.
Figure 7: The architecture of our Value Appraisal Module, where the numbers represent channel numbers and kernel size
Figure 8: The architecture of our Action Selection Module, where the numbers represent channel numbers and kernel size.
Algorithm 1 Dynamic Preference Inference (DPI) with On-Policy Envelope Actor–Critic
| Hyperparameter | Value |
| Learning rate | |
| Batch size | transitions per update |
| Discount factor | |
| GAE parameter | |
| PPO clip ratio ( ) | |
| Entropy coefficient |
Table 3: Hyperparameters used in all experiments. Values are shared across Maze and Queue unless otherwise noted.
Figure 9: Queue environment: Pareto fronts before and after the event. We approximate Pareto fronts in a 2D objective space (progress vs. deadline-related penalty) by training policies with fixed linear scalarizations over the vector reward. Each point is a policy; circles and triangles denote pre- and post-event regimes, respectively, and colors encode the dominant component of . The blue curve shows the pre-event Pareto front, which lies predominantly in the upper-left region (smaller deadline penalty with moderate progress), while the orange curve shows the post-event Pareto front, which shifts towards the lower-right (higher progress at the cost of larger deadline penalties). The scalarizations that are near-optimal in the pre-event regime induce strongly suboptimal trade-offs in the post-event regime, and vice versa, indicating that the Pareto-optimal preference configurations before and after the event are largely disjoint. This supports our claim that no single fixed over the exposed reward vector is robust across all event configurations.
| Method | Queue | Maze | ||
| MER | SR ( ) | MER | SR ( ) | |
| MER-PPO | 15.01 0.46 | 85.55 0.12 | ||
| SR-PPO | ||||
| Dense Oracle | ||||
| DPI-PPO | ||||
Table 4: Comparison results on Queue and Maze.
| Method | MER | SR(%) |
| ENVELOPE-PPO | ||
| Dense Oracle | ||
| DPI-PPO (Ours) |
Table 5: Comparison results on the modified continuous control environment. To handle a continuous action space, we adapt the rule-based ENVELOPE Q-learning baseline to a PPO actor–critic implementation (ENVELOPE-PPO). Dense Oracle is given privileged access to dynamic event signals and hand-crafted rules for updating the preference weights, whereas DPI-PPO (ours) only observes the standard environment state and must infer preferences purely from interaction.
| MER | SR(%) | ||
| 0.5 | 1.0 | ||
| 1.0 | 1.0 | ||
| 2.0 | 1.0 | ||
| 1.0 | 0.5 | ||
| 1.0 | 2.0 |
Table 6: Hyperparameter sensitivity analysis. Here and are reported as relative multipliers around the default used in the main experiments (i.e., 0.5 means ).
Figure 10: Mean episodic return (MER, left) and success rate (SR, right) as a function of environment steps. Solid lines show the mean over 10 random seeds and shaded regions denote the 95% confidence interval. DPI-PPO exhibits smooth and monotone improvement without divergence or collapse.