Human-Human & Human-Robot Interaction Transformer (H2INT) for Robot Navigation in Dense and Uncertain Crowds

Ao Shen; Kaixi Chen; Shiwei Liu; Fang Deng; Chen Chen

cs.ROSep 4, 2026

Human-Human & Human-Robot Interaction Transformer (H2INT) for Robot Navigation in Dense and Uncertain Crowds

Authors: Ao Shen, Kaixi Chen, Shiwei Liu, Fang Deng, Chen Chen

Organizations: School of Automation, Beijing Institute of Technology, Beijing 100081, China · State Key Laboratory of Autonomous Intelligent Unmanned Systems, Beijing 100081, China

Abstract

Safe robot navigation in dense crowds requires reasoning about pedestrian motion and how it may change in response to a robot. However, many learning-based approaches generate pedestrian motion independently of the robot or assume uniform reciprocity, omitting an important source of interaction uncertainty. This paper presents a Human-Human & Human-Robot Interaction Transformer (H2INT), a reinforcement learning framework that retains robot-conditioned changes in pedestrian motion during policy learning while allowing responsiveness to vary across pedestrians. Responsiveness affects the crowd dynamics when the robot is visible but is not supplied as a policy input; the policy must instead infer its consequences from robot-centered relative positions. A two-stage gated Transformer progressively encodes human-human and human-robot relations, while a recurrent policy captures their temporal evolution. A curriculum gradually reduces pedestrian responsiveness to increase interaction difficulty. Simulation experiments demonstrate improved navigation safety and robustness over representative baselines across response conditions and crowd densities, and show transfer without retraining to structurally distinct crowd-flow layouts. Ablations support the hierarchical relational encoding and gated updates. Real-robot deployment further verifies that the learned policy can operate with sparse observations in a physical environment.

Figures & tables

Fig. 1: An illustration of interaction relationships in crowd navigation.

Fig. 2: Overview of the proposed policy learning framework. The robot and pedestrians are represented by relative spatial edges and embedded into a shared latent space. A two-stage gated Transformer progressively refines the interaction representation, and the resulting context vector is processed by an Actor-Critic policy network for navigation.

Fig. 3: Two-stage gated Transformer encoder. The first stage constructs an interaction representation, and the second refines the encoded relational features for robot decision making.

Fig. 4: Training performance with and without responsiveness annealing. Dashed lines mark reductions of $\rho_{\mathrm{resp}}$ from 0.8 to 0.0. The annealed policy (blue) converges faster and more stably than the non-annealed baseline (orange); shaded regions denote $\pm$ one standard deviation.

Set	Method	$r_{s}\uparrow$	$r_{c}\downarrow$	$r_{t}\downarrow$	$t_{s}\downarrow$	$l_{p}\downarrow$	$\Delta v\downarrow$
High	ORCA	97	3	0	14.68	18.14	1.00
	SF	65	32	3	20.77	21.47	0.99
	CADRL	84	13	3	15.12	19.98	1.21
	SARL	88	10	2	14.01	19.11	1.15
	DS-RNN	91	8	1	16.36	22.23	1.58
	AIG-GST	92	6	2	15.77	22.48	1.63

TABLE I: Comparison under Different Response Regimes

Set	Method	$r_{s}\uparrow$	$r_{c}\downarrow$	$r_{t}\downarrow$	$t_{s}\downarrow$	$l_{p}\downarrow$	$\Delta v\downarrow$
$N=10$	ORCA	93	7	0	12.14	17.61	1.00
	SF	60	40	0	16.31	18.41	1.00
	CADRL	85	10	5	13.91	19.50	1.10
	SARL	93	7	0	13.10	18.05	1.16
	DS-RNN	89	11	0	12.39	18.69	1.04
	AIG-GST	95	5	0	14.37	20.48	1.51

TABLE II: Comparison under Different Crowd Scales

Fig. 5: Ablation trends under different pedestrian response regimes and crowd sizes. High, Medium, and Low correspond to mean responsiveness probabilities $\rho_{\mathrm{resp}}$ of 0.7, 0.5, and 0.3, respectively.

Set	Method	$r_{s}\uparrow$	$r_{c}\downarrow$	$r_{t}\downarrow$	$t_{s}\downarrow$	$l_{p}\downarrow$	$\Delta v\downarrow$
High	ORCA	93	5	2	17.39	22.16	0.99
	SF	56	32	12	23.91	26.04	0.94
	CADRL	77	17	6	18.23	24.19	1.25
	SARL	85	11	4	16.59	23.51	1.20
	DS-RNN	76	16	8	23.57	32.38	1.93
	AIG-GST	91	8	1	16.18	25.48	1.62

TABLE III: Comparison in Unseen Crowded Environments

Panel of Figure 9 — Fig. 6: Navigation trajectories in four crowd layouts. The robot and goal are shown in yellow and red, respectively; red and multicolored dashed curves denote the robot and pedestrian trajectories.

Panel of Figure 10 — Fig. 7: Snapshots of real-world indoor navigation at successive timesteps. The illustrated run is completed without collision among naturally moving pedestrians.

Robot Navigation

Robot navigation studies how robots perceive, localize, plan, and move safely toward goals in static or changing environments.

See all 554 papers

Learning Robot Visual Navigation in Crowds via Intention-Aware Scene Representations
Jun 24, 2026Han Bao, Bingyi Xia, Hanjing Ye +5Pedestrian Intention Prediction Robot Navigation
Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, SUSTech, Shenzhen, China · Xiamen Key Laboratory of Visual Perception Technology and Application, and the Algorithm Research Center at Reconova Information Technology Co., Ltd. in Xiamen, China · Research Institute of MA&EI, Peng Cheng Laboratory, Shenzhen, China +1
PRISM: Predictive Representation of Interaction Style and Motion for Social Robot Navigation
Sep 16, 2026Bo-Han Chen, Hiromu Taketsugu, Norimichi UkitaPedestrian Behavior Modeling Social Robot Navigation
National Chung Hsing University · Toyota Technological Institute
Don't Freeze, Don't Crash: Extending the Safe Operating Range of Neural Navigation in Dense Crowds
Mar 6, 2026Jiefu Zhang, Yang Xu, Vaneet AggarwalSafe Robot Navigation Robot Navigation
Purdue University, USA

Jun 24, 2026J/K move · Enter open · S save

Learning Robot Visual Navigation in Crowds via Intention-Aware Scene Representations

Han Bao, Bingyi Xia, Hanjing Ye, Yu Zhan+4

Shenzhen Key Laboratory of Robotics Perception and Intelligence, Department of Electronic and Electrical Engineering, SUSTech, Shenzhen, China · Xiamen Key Laboratory of Visual Perception Technology and Application, and the Algorithm Research Center at Reconova Information Technology Co., Ltd. in Xiamen, China · Research Institute of MA&EI, Peng Cheng Laboratory, Shenzhen, China+1

Pedestrian Intention Prediction Robot Navigation Robotic RL Social Robot Navigation Vision-Based Robot Control

Open paper AskPDF arXiv

Robot crowd navigation requires the ability to infer human intentions while accounting for the structural constraints of the environment. Currently, deep reinforcement learning (DRL) provides a promising method for learning navigation policies that understand human intentions. However, most of them rely on limited scene representations, treating pedestrians as simple 2D points and ignoring rich visual cues from both humans and the environment. To address this issue, we introduce iCrowdNav, a novel visual crowd navigation method with intention-aware scene representations, to encode behavioral and structural context from egocentric visual observations. Our method employs two key components: a spatio-temporal encoder for extracting occupancy features of the scene, and Intent-Interact Former (I

^2

Former), an attention-based module that encodes human poses to infer pedestrians' motion intentions. These features are integrated into a compact state embedding that supports effective DRL policy training. Extensive experiments show that our method achieves superior performance over baselines, and real-world deployment demonstrates vision-based crowd navigation.

Abstract

Robot Navigation

Explore similar work

Learning Robot Visual Navigation in Crowds via Intention-Aware Scene Representations

PRISM: Predictive Representation of Interaction Style and Motion for Social Robot Navigation

Don't Freeze, Don't Crash: Extending the Safe Operating Range of Neural Navigation in Dense Crowds