FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations

Konstantinos Foteinos; Georgios Angelidis; Aggelos Psiris; Vasileios Argyriou; Panagiotis Sarigiannidis; Georgios Th. Papadopoulos

cs.ROFeb 19, 2026

FR-GESTURE: An RGBD Dataset For Gesture-based Human-Robot Interaction In First Responder Operations

Authors: Konstantinos Foteinos, Georgios Angelidis, Aggelos Psiris, Vasileios Argyriou, Panagiotis Sarigiannidis, Georgios Th. Papadopoulos

Organizations: ∗Department of Informatics and Telematics, Harokopio University of Athens, Athens, Greece · † Department of Networks and Digital Media, Kingston University, Kingston upon Thames, Surrey, United Kingdom · ‡ Department of Electrical and Computer Engineering, University of Western Macedonia, Kozani, Greece · § Archimedes, Athena Research Center, Athens, Greece

Abstract

The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: https://doi.org/10.5281/zenodo.18131333.

Figures & tables

Name	Year	Task	Robot	Classes	Samples	Subjects	Environment	Range	Modality	Public
URGR Bamani et al. (2024)	2024	SHGR	UGV	5	347483	16	Indoor & outdoor	1-25 m	RGB	✗
MD-UHGRD Xu et al. (2024)	2024	SHGR	UAV	19+1	20000	5	-	3-10 m	RGB	✗
LRHG Zhou et al. (2021)	2021	SHGR	UAV	10	4320	8	Indoor	1-7 m	RGB	✓
Wang and Zhu (2021)	2021	IDHGR	Construction Machinery	11	364	-	Indoor & outdoor	-	Video RGBD	✗
AUTH Patrona et al. (2021)	2021	IDHGR	UAV	6	4930	20	Indoor & outdoor	-	Video RGB	✗
UAV GESTURE Perera et al. (2019)	2019	IDHGR	UAV	13	119	8	Outdoor	-	Video RGB	✗

TABLE I : Existing datasets for gesture-based AMR guidance.

Panel of Figure 2 — Fig. 1 : Indicative samples from the FR-GESTURE dataset. Each gesture instance is captured by two cameras, positioned at different heights to increase diversity. Participants performed signs in various scenes to enhance generalization ability.

Panel of Figure 3 — Fig. 2 : Examples of the 12 defined gestures. All of them correspond to specific UGV commands, tailored for first response scenarios.

Method	Modality	Pretrained	Train	Val	Test
ResNet-18	RGB	HaGRID	99.78	92.26	92.46
ResNet-50	RGB	HaGRID	99.01	90.27	90.07
ResNeXt-50	RGB	HaGRID	100.0	94.24	90.87
EfficientNet-B0	RGB	HaGRID	99.86	97.22	96.42

TABLE II : Reported F1-Score (in percentage) for the uniform protocol.

Method	Modality	Pretrained	Train	Val	Test
ResNet-18	RGB	HaGRID	99.95	88.88	65.67
ResNet-50	RGB	HaGRID	94.53	75.79	52.18
ResNeXt-50	RGB	HaGRID	98.56	57.93	39.68
EfficientNet-B0	RGB	HaGRID	99.86	95.43	87.73

TABLE III : Reported F1-Score (in percentage) for the subject independent protocol. The performance is seriously affected when the models are enforced to recognize gestures performed by unseen signers.

Human-Robot Interaction

Human-robot interaction studies how people and robots communicate, coordinate, and safely work or live together.

See all 452 papers

R2RI: A Multi-View Event and RGB Dataset for Robot-to-Robot Interaction
Oct 5, 2026Gabriele Magrini, Riccardo Catalini, Federico Becattini +4Robotic Data Collection Robotics
Department of Information Engineering, University of Florence, Florence, Italy · Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, Modena, Italy · Department of Information Engineering and Mathematics (DIISM), University of Siena, Siena, Italy
Intuitive Human-Robot Interaction: Development and Evaluation of a Gesture-Based User Interface for Object Selection
Apr 7, 2026Bijan Kavousian, Oliver Petrovic, Werner HerfsHuman-Robot Interaction Human-Robot Collaboration
Laboratory for Machine Tools and Production Engineering WZL of RWTH Aachen, Steinbachstraße 25, D-52072 Aachen
On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective
Feb 19, 2026Domonkos VargaGesture Recognition Data Leakage

Oct 5, 2026J/K move · Enter open · S save

R2RI: A Multi-View Event and RGB Dataset for Robot-to-Robot Interaction

Gabriele Magrini, Riccardo Catalini, Federico Becattini, Guido Borghi+3

Department of Information Engineering, University of Florence, Florence, Italy · Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, Modena, Italy · Department of Information Engineering and Mathematics (DIISM), University of Siena, Siena, Italy

Robotic Data Collection Robotics Social Robotics

Open paper AskPDF arXiv

Understanding and modeling interactions between autonomous agents is a fundamental challenge in robotics, with broad implications for collaborative systems, social robotics, and human-robot coexistence. Although the study of robot interactions has emerged as a compelling research direction, progress has been severely hampered by the absence of large-scale benchmarks. In this paper, we introduce Robot-to-Robot Interaction (R2RI), the first dataset specifically designed to address the Robot-Robot Interaction (RRI) task. R2RI consists of different humanoid robots and realistic interactions modeled on real human social behaviors. Complementary viewpoints are available, i.e., an egocentric perspective from each robot's onboard sensors, and an exocentric perspective from external fixed cameras, thus enabling rich spatial and contextual understanding of the interaction dynamics. The dataset comprises more than

6.5

M frames and

\approx5000

videos at

120

fps, including Event and RGB domains. We investigate pros and cons of each domain, comparing state-of-the-art approaches for a number of key sensing and interaction based tasks. We publicly release the dataset and its annotations for all tasks and modalities at https://github.com/MagriniGabriele/R2RI.

Abstract

Human-Robot Interaction

Explore similar work

R2RI: A Multi-View Event and RGB Dataset for Robot-to-Robot Interaction

Intuitive Human-Robot Interaction: Development and Evaluation of a Gesture-Based User Interface for Object Selection

On the Evaluation Protocol of Gesture Recognition for UAV-based Rescue Operation based on Deep Learning: A Subject-Independence Perspective