A Comparison of Heuristic, Human-Sensitive, and Self-Play DRL Policies in Human-Agent Teaming
Organizations: Scuola Superiore Meridionale, Naples, Italy · University of Naples Federico II, Naples, Italy · School of Psychological Sciences, Macquarie University, Sydney, NSW 2109, Australia · School of Psychological Sciences and the Performance and Expertise Research Centre, Macquarie University, Sydney, NSW 2109, Australia ·
Abstract
Dynamical Perceptual-Motor Primitives (DPMPs) offer a tractable framework for generating human-like movement dynamics in artificial agents (AAs) and have been used to support coordination in human-AI teaming tasks. Traditionally, DPMP controllers rely on heuristic (rule-based) decision policies derived from human behavior to guide goal selection and action sequencing. Although effective in structured tasks, such heuristics can be difficult to generalize to more complex settings, motivating interest in deep reinforcement learning (DRL) approaches for learning decision policies for human-AI interaction. A key challenge, however, is ensuring that DRL policies remain aligned with human strategies, particularly when agents are trained through self-play. To examine this issue, we compared three target-selection policies governing DPMP-based AAs in a first-person herding task: (1) a human-derived heuristic policy, (2) a self-play DRL policy (SP-DRL), and (3) a human-sensitive DRL policy (HS-DRL) trained alongside a heuristic agent. Results showed that heuristic agents most closely matched typical human-human coordination patterns, followed by HS-DRL agents, whereas SP-DRL agents deviated most. These findings highlight the importance of human-informed training when developing DRL-based teammates and suggest that simple heuristic policies may remain an effective approach for designing human-aligned AAs capable of human-AI collaboration.
Figures & tables
Appendix figures & tables56 assets
Supplementary material from the paper’s appendix.
Appendix
| Step | Condition | Action |
|---|---|---|
| 1 | HA current target = null (otherwise skip to Step 2) | Select the closest TA to the HA based on the angular distance of TAs from the HA. – If the selected TA is closest to other-HA, skip this TA and select the next closest TA. – If all TAs are closer to other-HA, select TA furthest from the other-HA. – Assign (return) the selected TA as HA’s current target. |
| 2 | Check for near-collinear TAs | Identify if other TAs are within 20° of the currently selected TA (with respect to CZ). – If no collinear TAs, skip to Step 3, otherwise… – Order the collinear TAs based on their Euclidean distance to CZ. – Assign (return) furthest TA from CZ as HA’s current target (resulting in target switching when corralling collinear TAs). |
| 3 | Check for TA containment | If HA’s currently selected TA is in the containment zone and stationary, current target = null, wait 1.0 seconds. |
| 4 | Pause and Return | Wait 0.25 seconds then return to Step 1. |
Human-AI Collaboration
Human-AI collaboration studies how people and AI systems divide, coordinate, and adapt work to support decisions, creativity, and learning.
See all 476 papers