A Comparison of Heuristic, Human-Sensitive, and Self-Play DRL Policies in Human-Agent Teaming
Authors: Ayman bin Kamruddin, Christopher Lam, Gaurav Patil, Mirco Musolesi, Mario di Bernardo, Michael J. Richardson
Organizations: Scuola Superiore Meridionale, Naples, Italy · University of Naples Federico II, Naples, Italy · School of Psychological Sciences, Macquarie University, Sydney, NSW 2109, Australia · School of Psychological Sciences and the Performance and Expertise Research Centre, Macquarie University, Sydney, NSW 2109, Australia · University College London, London, U.K.
Dynamical Perceptual-Motor Primitives (DPMPs) offer a tractable framework for generating human-like movement dynamics in artificial agents (AAs) and have been used to support coordination in human-AI teaming tasks. Traditionally, DPMP controllers rely on heuristic (rule-based) decision policies derived from human behavior to guide goal selection and action sequencing. Although effective in structured tasks, such heuristics can be difficult to generalize to more complex settings, motivating interest in deep reinforcement learning (DRL) approaches for learning decision policies for human-AI interaction. A key challenge, however, is ensuring that DRL policies remain aligned with human strategies, particularly when agents are trained through self-play. To examine this issue, we compared three target-selection policies governing DPMP-based AAs in a first-person herding task: (1) a human-derived heuristic policy, (2) a self-play DRL policy (SP-DRL), and (3) a human-sensitive DRL policy (HS-DRL) trained alongside a heuristic agent. Results showed that heuristic agents most closely matched typical human-human coordination patterns, followed by HS-DRL agents, whereas SP-DRL agents deviated most. These findings highlight the importance of human-informed training when developing DRL-based teammates and suggest that simple heuristic policies may remain an effective approach for designing human-aligned AAs capable of human-AI collaboration.
Figures & tables
Fig. 1 : (a) First-person herding phases. The HA first approaches the TA, then transitions to corralling the TA towards the containment zone. The median angle at which human HAs begin influencing a TA is 1.1°. The TA influence radius is di=10 m . (b) Model variables. C denotes the containment zone. Positions are given by x (TA) and y (HA). Offset goal location is c , and dc(x,y) is the HA-offset distance. The HA heading angle is ϕ . θG,θT,θC denote the angles of the offset goal, TA, and containment zone, respectively, measured relative to the vertical axis centered on the HA.
Fig. 2 : Task environment. (a) Example first-person view of the game field from a HA. Visible are TAs, the containment zone (red), boundary walls (gray), and partner HA (red capsule) in the distance. (b) Bird’s-eye view of the field (not visible to participants), showing HA positions (red dots) and TA positions (white dots).
Fig. 3 : Example binary trace maps (Trial 7) with participant and AA trajectories. Binary trace maps were derived from the human-human dataset [ 20 ] . Red lines indicate the HA paths used to compute trajectory overlap with typical human navigation.
Fig. 4 : Average decision and trajectory overlap scores for AAs (Panels A–B) and participants (Panels C–D) across target-number conditions; error bars represent 95% confidence intervals.
Appendix figures & tables56 assets
Supplementary material from the paper’s appendix.
Appendix
Fig. 1 : Justification of choice of ϵ=5∘ . ζ=0 when ∠(HA,TA)=5∘ , ensuring the repulsive term decays as the HA approaches the TA.
Step
Condition
Action
1
HA current target = null (otherwise skip to Step 2)
Select the closest TA to the HA based on the angular distance of TAs from the HA. – If the selected TA is closest to other-HA, skip this TA and select the next closest TA. – If all TAs are closer to other-HA, select TA furthest from the other-HA. – Assign (return) the selected TA as HA’s current target.
2
Check for near-collinear TAs
Identify if other TAs are within 20° of the currently selected TA (with respect to CZ). – If no collinear TAs, skip to Step 3, otherwise… – Order the collinear TAs based on their Euclidean distance to CZ. – Assign (return) furthest TA from CZ as HA’s current target (resulting in target switching when corralling collinear TAs).
3
Check for TA containment
If HA’s currently selected TA is in the containment zone and stationary, current target = null, wait 1.0 seconds.
4
Pause and Return
Wait 0.25 seconds then return to Step 1.
Appendix
TABLE I : Rule logic of the successive collinear angle (SCA) target-selection policy adapted for multiple herders
Multi-agent collaboration, especially in human-AI teaming, requires agents that can adapt to novel partners with diverse and dynamic behaviors. Conventional Deep Hierarchical Reinforcement Learning (DHRL) methods focus on agent-centric rewards and overlook partner behavior, leading to shortcut learning, where skills exploit spurious information instead of adapting to partners' dynamic behaviors. This limitation undermines agents' ability to adapt and coordinate effectively with novel partners. We introduce Partner-Aware Skill Discovery (PASD), a DHRL framework that learns skills conditioned on partner behavior. PASD introduces a contrastive intrinsic reward to capture patterns emerging from partner interactions, aligning skill representations across similar partners while maintaining discriminability across diverse strategies. By structuring the skill space based on partner interactions, this approach mitigates shortcut learning and promotes behavioral consistency, enabling robust and adaptive coordination. We extensively evaluate PASD in the Overcooked-AI benchmark with a diverse population of partners characterized by varying skill levels and play styles. We further evaluate the approach with human proxy models trained from human-human gameplay trajectories. PASD consistently outperforms existing population-based and hierarchical baselines, demonstrating transferable skill learning that generalizes across a wide range of partner behaviors. Analysis of learned skill representations shows that PASD adapts effectively to diverse partner behaviors, highlighting its robustness in human-AI collaboration.
Adnan Ahmad, Bahareh Nakisa, Mohammad Naim Rastgoo
aSchoold of Information Technology, Deakin University, 75 Pidgons Rd, Waurn Ponds, 3216, VIC, Australia · bFaculty of Information Technology, Monash University, Wellington Rd, Clayton, 3800, VIC, Australia
Mutual adaptation is a central challenge in human-AI teaming, as humans naturally adjust their strategies in response to an AI agent's behavior. Existing approaches attempt to approximate human behavior by diversifying training partners; however, these partners are typically static and fail to capture the adaptive nature of human teammates. When agents are trained jointly in standard multi-agent settings, they often converge to opaque coordination strategies that work only with their co-trained partners, leading to poor generalization. To model adaptive human behavior, we formulate human-AI teaming as an Interactive Partially Observable Markov Decision Process (I-POMDP). We propose NestRL, a nested training regime that learns the solution to a finite-level I-POMDP by training agents at each level against adaptive agents from the level below. This exposes agents to adaptive behavior while preventing emergence of opaque coordination strategies. We provide theoretical analysis showing that NestRL agents avoid convergence to partner-specific strategies, and validate this empirically in the Overcooked domain against state-of-the-art baselines. NestRL achieves higher task performance with both unseen adaptive agents and real human teammates, while exhibiting significantly greater adaptability over the course of interaction.
School of Computing and AI, Arizona State University, Tempe, AZ, USA · Department of Computer Science, Colorado State University, Fort Collins, CO, USA
Human-AI collaboration requires agents that can adapt to diverse partner behaviors and skill levels while remaining robust to unseen partners. Existing methods often collapse to a single dominant behavior or learn poorly aligned skills, limiting effective coordination. We propose Intrinsic Action Disentanglement (IAD), a deep hierarchical reinforcement learning (DHRL) framework that learns distinct, partner-aware low-level action sequences conditioned on high-level latent skills. IAD introduces an intrinsic reward that explicitly encourages disentangled action distributions of the agent's low-level policy across skills, yielding an interpretable mapping between high-level decisions and partner-specific behavioral responses. By capturing temporally extended interaction patterns, IAD enables flexible adaptation to heterogeneous partner dynamics under distributional shift. We evaluate IAD in the Overcooked-AI domain across multiple layouts and diverse partner settings, including unseen simulated partners, a human-proxy model trained on human-human gameplay, and real human partners. Results show that IAD consistently outperforms strong baselines and achieves more reliable, adaptive coordination across all settings.
Adnan Ahmad, Bahareh Nakisa, Mohammad Naim Rastgoo
School of Information Technology Deakin University, Waurn Ponds, 3216, VIC, Australia · Faculty of Information Technology, Monash University, 3800, VIC, Australia