Organizations: ∗Department of Informatics and Telematics, Harokopio University of Athens, Athens, Greece · † Department of Networks and Digital Media, Kingston University, Kingston upon Thames, Surrey, United Kingdom · ‡ Department of Electrical and Computer Engineering, University of Western Macedonia, Kozani, Greece · § Archimedes, Athena Research Center, Athens, Greece
The ever increasing intensity and number of disasters make even more difficult the work of First Responders (FRs). Artificial intelligence and robotics solutions could facilitate their operations, compensating these difficulties. To this end, we propose a dataset for gesture-based UGV control by FRs, introducing a set of 12 commands, drawing inspiration from existing gestures used by FRs and tactical hand signals and refined after incorporating feedback from experienced FRs. Then we proceed with the data collection itself, resulting in 3312 RGBD pairs captured from 2 viewpoints and 7 distances. To the best of our knowledge, this is the first dataset especially intended for gesture-based UGV guidance by FRs. Finally we define evaluation protocols for our RGBD dataset, termed FR-GESTURE, and we perform baseline experiments, which are put forward for improvement. We have made data publicly available to promote future research on the domain: https://doi.org/10.5281/zenodo.18131333.
Figures & tables
Name
Year
Task
Robot
Classes
Samples
Subjects
Environment
Range
Modality
Public
URGR Bamani et al. (2024)
2024
SHGR
UGV
5
347483
16
Indoor & outdoor
1-25 m
RGB
✗
MD-UHGRD Xu et al. (2024)
2024
SHGR
UAV
19+1
20000
5
-
3-10 m
RGB
✗
LRHG Zhou et al. (2021)
2021
SHGR
UAV
10
4320
8
Indoor
1-7 m
RGB
✓
Wang and Zhu (2021)
2021
IDHGR
Construction Machinery
11
364
-
Indoor & outdoor
-
Video RGBD
✗
AUTH Patrona et al. (2021)
2021
IDHGR
UAV
6
4930
20
Indoor & outdoor
-
Video RGB
✗
UAV GESTURE Perera et al. (2019)
2019
IDHGR
UAV
13
119
8
Outdoor
-
Video RGB
✗
TABLE I : Existing datasets for gesture-based AMR guidance.
Fig. 1 : Indicative samples from the FR-GESTURE dataset. Each gesture instance is captured by two cameras, positioned at different heights to increase diversity. Participants performed signs in various scenes to enhance generalization ability.
Fig. 2 : Examples of the 12 defined gestures. All of them correspond to specific UGV commands, tailored for first response scenarios.
Fig. 3 : Distribution of scenes.
Fig. 4 : Distribution of subjects.
Method
Modality
Pretrained
Train
Val
Test
ResNet-18
RGB
HaGRID
99.78
92.26
92.46
ResNet-50
RGB
HaGRID
99.01
90.27
90.07
ResNeXt-50
RGB
HaGRID
100.0
94.24
90.87
EfficientNet-B0
RGB
HaGRID
99.86
97.22
96.42
TABLE II : Reported F1-Score (in percentage) for the uniform protocol.
Method
Modality
Pretrained
Train
Val
Test
ResNet-18
RGB
HaGRID
99.95
88.88
65.67
ResNet-50
RGB
HaGRID
94.53
75.79
52.18
ResNeXt-50
RGB
HaGRID
98.56
57.93
39.68
EfficientNet-B0
RGB
HaGRID
99.86
95.43
87.73
TABLE III : Reported F1-Score (in percentage) for the subject independent protocol. The performance is seriously affected when the models are enforced to recognize gestures performed by unseen signers.
Understanding and modeling interactions between autonomous agents is a fundamental challenge in robotics, with broad implications for collaborative systems, social robotics, and human-robot coexistence. Although the study of robot interactions has emerged as a compelling research direction, progress has been severely hampered by the absence of large-scale benchmarks. In this paper, we introduce Robot-to-Robot Interaction (R2RI), the first dataset specifically designed to address the Robot-Robot Interaction (RRI) task. R2RI consists of different humanoid robots and realistic interactions modeled on real human social behaviors. Complementary viewpoints are available, i.e., an egocentric perspective from each robot's onboard sensors, and an exocentric perspective from external fixed cameras, thus enabling rich spatial and contextual understanding of the interaction dynamics. The dataset comprises more than 6.5M frames and ≈5000 videos at 120 fps, including Event and RGB domains. We investigate pros and cons of each domain, comparing state-of-the-art approaches for a number of key sensing and interaction based tasks. We publicly release the dataset and its annotations for all tasks and modalities at https://github.com/MagriniGabriele/R2RI.
Department of Information Engineering, University of Florence, Florence, Italy · Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, Modena, Italy · Department of Information Engineering and Mathematics (DIISM), University of Siena, Siena, Italy
Gestures are a natural form of communication between humans and can also be leveraged for human-robot interaction. This work presents a gesture-based user interface for object selection using pointing and click gestures. An experiment with 20 participants evaluates accuracy and selection time, demonstrating the potential for efficient collaboration.
Bijan Kavousian, Oliver Petrovic, Werner Herfs
Laboratory for Machine Tools and Production Engineering WZL of RWTH Aachen, Steinbachstraße 25, D-52072 Aachen
This paper presents a methodological analysis of the gesture-recognition approach proposed by Liu and Szirányi, with a particular focus on the validity of their evaluation protocol. We show that the reported near-perfect accuracy metrics result from a frame-level random train-test split that inevitably mixes samples from the same subjects across both sets, causing severe data leakage. By examining the published confusion matrix, learning curves, and dataset construction, we demonstrate that the evaluation does not measure generalization to unseen individuals. Our findings underscore the importance of subject-independent data partitioning in vision-based gesture-recognition research, especially for applications - such as UAV-human interaction - that require reliable recognition of gestures performed by previously unseen people.