NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming

Upasana Biswas; Durgesh Kalwar; Subbarao Kambhampati; Sarath Sreedharan

cs.ROFeb 18, 2026

NestRL: A Nested Training Regime for Mutual Adaptation in Human-AI Teaming

Authors: Upasana Biswas, Durgesh Kalwar, Subbarao Kambhampati, Sarath Sreedharan

Organizations: School of Computing and AI, Arizona State University, Tempe, AZ, USA · Department of Computer Science, Colorado State University, Fort Collins, CO, USA

Abstract

Mutual adaptation is a central challenge in human-AI teaming, as humans naturally adjust their strategies in response to an AI agent's behavior. Existing approaches attempt to approximate human behavior by diversifying training partners; however, these partners are typically static and fail to capture the adaptive nature of human teammates. When agents are trained jointly in standard multi-agent settings, they often converge to opaque coordination strategies that work only with their co-trained partners, leading to poor generalization. To model adaptive human behavior, we formulate human-AI teaming as an Interactive Partially Observable Markov Decision Process (I-POMDP). We propose NestRL, a nested training regime that learns the solution to a finite-level I-POMDP by training agents at each level against adaptive agents from the level below. This exposes agents to adaptive behavior while preventing emergence of opaque coordination strategies. We provide theoretical analysis showing that NestRL agents avoid convergence to partner-specific strategies, and validate this empirically in the Overcooked domain against state-of-the-art baselines. NestRL achieves higher task performance with both unseen adaptive agents and real human teammates, while exhibiting significantly greater adaptability over the course of interaction.

Figures & tables

Figure 1: Multi-recipe Overcooked domain. Each agent selects one of three recipes ( PotatoBroccoliSalad , LettuceOnionSalad , or TomatoCarrotSalad ) per episode. Each recipe requires both agents to contribute one ingredient from their respective sides, place ingredients on a shared plate, and deliver the completed dish to the serving station. Multiple episodes occur within each round, requiring repeated coordination to establish a shared convention on which recipe to prepare.

Figure 2: NestRL: Nested training for mutual adaptation: Level-1 human policies are trained against fixed robot policies, producing diverse adaptive behaviors corresponding to different coordination conventions. A level-2 robot is then trained against these adaptive partners, learning to maintain uncertainty over partner types and to act in a way that anticipates how the partner will adapt in return. This models bounded recursive reasoning i+n an I-POMDP framework and promotes coordination across multiple equilibria.

	Non-adaptive	Adaptive partners (held out)					Aggregate over P1–P5
Agent	L0	P1	P2	P3	P4	P5	Avg. success	CV
NestRL (ours)	0.95	0.67	0.80	0.93	0.87	0.80	0.81 (0.71–0.89)	0.11
LiLi	0.95	0.73	0.40	0.67	0.53	0.67	0.60 (0.49–0.70) ∗∗	0.20
PACE	0.96	0.73	0.33	0.67	0.60	0.40	0.55 (0.43–0.65) ∗∗∗	0.28
Generalist	0.94	0.40	0.53	0.73	0.60	0.47	0.55 (0.43–0.65) ∗∗∗	0.21

Table 1: (10 rounds

\times

5 episodes) Each cell is the agent’s success rate (fraction of episodes solved). L0 denotes average success rate when NestRL is paired with non-adaptive partners and P1–P5 denote average success rate when NestRL is paired with held-out adaptive L1 partners. The parenthesised range is the 95% confidence interval (Wilson score). CV

=\sigma/\mu

measures sensitivity to partner identity, where

\sigma

and

\mu

are the standard deviation and mean of success rates across L1 partners measures sensitivity to partner identity.

{}^{\ast\ast}p{<}0.01

{}^{\ast\ast\ast}p{<}0.001

(pooled test vs. NestRL).

1 2 3 4 5 6 7 8 9 1 100 0 0 0 10 20 10 80 0 2 0 100 0 0 0 0 0 0 0 3 0 0 100 0 0 0 0 0 40 4 0 0 0 100 50 100 90 0 0 5 100 0 0 100 100 100 50 10 0 6 0 0 0 80 50 100 100 0 0 7 0 0 0 90 50 100 100 0 0 8 100 0 0 0 0 0 0 100 0 9 0 0 20 0 0 0 0 0 100 (a) Self-play cross-play matrix

\mathcal{R}_{1}

(LO) 3 9 3 100 40 9 20 100

\mathcal{R}_{2}

(TC) 1 4 5 6 7 8 1 100 0 10 20 10 80 4 0 100 50 100 90 0 5 100 100 100 100 50 10 6 0 80 50 100 100 0 7 0 90 50 100 100 0 8 100 0 0 0 0 100 (b–c) Within-recipe cross-play NestRL

\times

L0 Partner Success

L0-R_{1}

95.2

L0-R_{2}

98.5

L0-R_{3}

94.7 Mean 96.1 (e) NestRL generalization

Figure 3: Convention collapse in self-play vs. generalization under nested training. (a) Cross-play matrix for 9 self-play agents grouped by recipe; darker = higher success. Diagonal (bold) shows self-coordination (

100\%

); off-diagonal shows cross-play failure. (b-c) Within-recipe submatrices isolate task-irrelevant signalling: agents sharing the same recipe still fail to coordinate (e.g., sp5

\times

sp8 =

10\%

\mathcal{R}_{2}

; sp9 fails with sp3, sp9 in

\mathcal{R}_{1}

). (e) The NestRL agent trained via nested adaptation achieves

>94\%

success with all three L0 convention types, demonstrating robustness to both task-specific and task-irrelevant convention variation.

Figure 4: Emergent coordination patterns. Recipe commitment trajectories for representative episodes. Horizontal bars show which recipe each agent works on over time; vertical dashed lines mark key coordination moments. (a) L1 explores multiple recipes while waiting for L0 to reveal its commitment, then follows L0’s lead on the Potato-Broccoli recipe. (b) L2 takes initiative by committing to Tomato–Carrot at $t{=}1$ ; L1 initially explores Lettuce–Onion, then follows L2’s lead at $t{=}17$ .

Panel of Figure 5 — Figure 5: Delivery rate improves across episodes within a round as the robot adapts to the partner.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Phase 1: Informed consent. Upon opening the study link, participants are briefed on the study duration ( ${\sim}15$ minutes), confidentiality and data protection policies, remuneration structure, and their right to withdraw at any time. The page displays the IRB approval code and explains how participants will receive their completion code and payment via the Prolific platform. A consent checkbox must be actively checked before proceeding, confirming that the participant understands the study conditions and agrees to participate.

Figure 7: Phase 2: Demographics questionnaire. Participants report their age, prior experience with the Overcooked game (on a four-point scale from “Never played” to “Expert / very experienced”), and highest educational qualification. These data are used to characterize the participant pool and to check whether prior game experience moderates the observed effects.

Figure 8: Phase 3: Game instructions. The left panel displays the game environment, showing the kitchen layout with ingredient stations (tomato, lettuce, onion, potato, broccoli, carrot), chopping boards, a shared center plate, and the delivery station (star). Two chef avatars represent the human participant and the AI partner. The right panel explains the game mechanics in four steps: (1) pick up an ingredient, (2) chop it on a cutting board, (3) place it on the shared plate, and (4) deliver the completed dish. The three valid recipes are listed (Lettuce + Onion, Potato + Broccoli, Tomato + Carrot). The study structure is also explained: 1 practice round, 2 trial rounds (5 episodes each), and 4 main rounds (5 episodes each), with each episode lasting 40 seconds.

Figure 9: Phase 4: Practice round. The participant controls one chef avatar using arrow keys (movement) and spacebar (interaction), while a rule-based AI partner controls the other. A persistent sidebar displays the control scheme and a step-by-step “How to Play” guide. This single-episode round is explicitly marked as practice and does not affect scoring or payment, allowing participants to build familiarity with the interface before the evaluation begins.

Figure 10: Phase 5a: Trial phase introduction. Before the trial rounds begin, participants are briefed on what to expect: 2 trial rounds of 5 episodes each, with each episode lasting 40 seconds. The page explains that the AI partner may prefer different recipes and clarifies the success criterion—an episode is successful only if the recipe containing the ingredient the participant picked up is delivered. This incentivizes coordination with the AI partner. The remuneration structure for the upcoming main rounds is also previewed.

Figure 11: Phase 5b: Trial round gameplay (Trial Block 1). The header displays the current block, episode counter (e.g., “Episode 1/5”), and a countdown timer. The right sidebar lists the three available recipes. Participants play 2 trial blocks of 5 episodes each with rule-based AI partners, developing fluency with the interface and game dynamics before the main evaluation.

Figure 12: Phase 6a: Main study introduction. A transition page informs participants that the performance-contingent bonus is now active: $0.625 per successful round (all 5 episodes delivered), up to a maximum bonus of $2.50 on top of the $2.50 base payment. The success criterion is reiterated with a highlighted reminder. Participants will play 4 main rounds, each with a different AI partner. — Figure 12: Phase 6a: Main study introduction. A transition page informs participants that the performance-contingent bonus is now active: $0.625 per successful round (all 5 episodes delivered), up to a maximum bonus of$ 2.50 on top of the $2.50 base payment. The success criterion is reiterated with a highlighted reminder. Participants will play 4 main rounds, each with a different AI partner.

Figure 13: Phase 6b: Main round gameplay (Round 1). The interface includes the game canvas, episode and time indicators, available recipes, control reference, and step-by-step instructions. The participant is paired with one of the evaluated AI agents (PACE or L2); each agent is played for 2 rounds. The agent assignment order is counterbalanced across participants to control for order effects.

Figure 14: Phase 6c: Post-round assessment. After each of the 4 main rounds, participants rate their experience on eight 7-point Likert scales. The first six items correspond to the NASA Task Load Index: mental demand, physical demand, temporal demand, performance, effort, and frustration. Two additional items assess perceived partner quality: proactiveness (“How proactive was your AI partner in completing the task?”) and adaptiveness (“How well did your AI partner adapt to your actions and strategy?”). These items test whether participants can subjectively distinguish agents trained with nested adaptation from those that are not.

Question	PACE	NestRL (ours)
How mentally demanding was the task? ∗∗ $\downarrow$	6.05	4.89
How physically demanding was the task? $\downarrow$	5.18	4.09
How rushed or hurried did you feel? ∗∗ $\downarrow$	6.64	4.91
How successful was the team at completing the task? ∗∗ $\uparrow$	3.59	5.48
How hard did you have to work to accomplish the task? ∗∗ $\downarrow$	7.77	5.59
How insecure, discouraged, or irritated did you feel? ∗∗ $\downarrow$	6.23	4.32

Appendix

Table 2: Subjective ratings from the user study (

n{=}25

) on a 7-point scale. The first six items correspond to NASA-TLX workload measures, and the last two evaluate perceived robot behavior. Arrows indicate whether lower (

\downarrow

) or higher (

\uparrow

) values correspond to better outcomes. Participants report significantly lower temporal demand and frustration with NestRL, and rate it higher on perceived team performance and adaptiveness compared to PACE (

{}^{*}p{<}.05

{}^{**}p{<}.01

Algorithm	Learning Rate	PPO Clip $\epsilon$	Entropy Coef	$\gamma$	GAE $\lambda$	Batch Size	# Epochs	# Mini-batches	Grad Clip (L2)	Activation	Hidden Dims
Level-n Training	1e-3	0.2	0.03	0.99	0.95	72000	4	18	15.0	ReLU	[128,128]
PACE	1e-3	0.2	0.03	0.99	0.95	72000	4	18	15.0	ReLU	[128,128]
Generalist	1e-3	0.2	0.03	0.99	0.95	72000	4	18	15.0	ReLU	[128,128]
LILI	1e-3	0.2	0.03	0.99	0.95	72000	4	18	15.0	ReLU	[128,128]
LIAM	1e-3	0.2	0.03	0.99	0.95	72000	4	18	15.0	ReLU	[128,128]
LIAMX	1e-3	0.2	0.03	0.99	0.95	72000	4	18	15.0	ReLU	[128,128]

Appendix

Table 3: Hyperparameters for all algorithms in the Overcooked environment.

Explore similar work

Adaptive Human-AI Coordination via Hierarchical Action Disentanglement
May 23, 2026Adnan Ahmad, Bahareh Nakisa, Mohammad Naim RastgooDisentangled Representation Learning Hierarchical RL
School of Information Technology Deakin University, Waurn Ponds, 3216, VIC, Australia · Faculty of Information Technology, Monash University, 3800, VIC, Australia
Partner-Aware Hierarchical Skill Discovery for Robust Human-AI Collaboration
May 23, 2026Adnan Ahmad, Bahareh Nakisa, Mohammad Naim RastgooHierarchical RL Multi-Agent Collaboration
aSchoold of Information Technology, Deakin University, 75 Pidgons Rd, Waurn Ponds, 3216, VIC, Australia · bFaculty of Information Technology, Monash University, Wellington Rd, Clayton, 3800, VIC, Australia
Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork
May 23, 2026Yuheng Jing, Kai Li, Ziwen Zhang +8RL Benchmarks Multi-Agent Coordination
C2DL, Institute of Automation, Chinese Academy of Sciences · School of Artificial Intelligence, University of Chinese Academy of Sciences · University of Science and Technology of China +5

May 23, 2026J/K move · Enter open · S save

Adaptive Human-AI Coordination via Hierarchical Action Disentanglement

Adnan Ahmad, Bahareh Nakisa, Mohammad Naim Rastgoo

School of Information Technology Deakin University, Waurn Ponds, 3216, VIC, Australia · Faculty of Information Technology, Monash University, 3800, VIC, Australia

Disentangled Representation Learning Hierarchical RL Multi-Agent Collaboration Agent Skill Learning Ad Hoc Teamwork Cooperative MARL+1

Open paper AskPDF arXiv

Human-AI collaboration requires agents that can adapt to diverse partner behaviors and skill levels while remaining robust to unseen partners. Existing methods often collapse to a single dominant behavior or learn poorly aligned skills, limiting effective coordination. We propose Intrinsic Action Disentanglement (IAD), a deep hierarchical reinforcement learning (DHRL) framework that learns distinct, partner-aware low-level action sequences conditioned on high-level latent skills. IAD introduces an intrinsic reward that explicitly encourages disentangled action distributions of the agent's low-level policy across skills, yielding an interpretable mapping between high-level decisions and partner-specific behavioral responses. By capturing temporally extended interaction patterns, IAD enables flexible adaptation to heterogeneous partner dynamics under distributional shift. We evaluate IAD in the Overcooked-AI domain across multiple layouts and diverse partner settings, including unseen simulated partners, a human-proxy model trained on human-human gameplay, and real human partners. Results show that IAD consistently outperforms strong baselines and achieves more reliable, adaptive coordination across all settings.

Abstract

Explore similar work

Adaptive Human-AI Coordination via Hierarchical Action Disentanglement

Partner-Aware Hierarchical Skill Discovery for Robust Human-AI Collaboration

Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork