Design of a Human-Assistance Robot System with Contextual Action Recognition

Amanuel Ergogo; Teresa Zielińska

cs.ROAug 22, 2026

Design of a Human-Assistance Robot System with Contextual Action Recognition

Authors: Amanuel Ergogo, Teresa Zielińska

Organizations: Warsaw University of Technology, Warsaw, Poland · SANO Centre for Computational Personalized Medicine, Krakow, Poland

Abstract

This paper presents a conceptual design for a proactive human assisting robot system capable of recognizing human activities and responding proactively. The system leverages contextual human activity recognition to interpret human actions across diverse contexts, while behavior trees are utilized to define dynamic and interpretable robot behaviors. We outline the system architecture, incorporating contextual human action recognition (HAR), behavior trees (BTs), and ROS, using the Spot robot platform as a representative example. We explain how HAR enables the robot to provide proactive assistance, discuss its limitations, and introduce methodologies for contextual HAR to address these limitations, thereby enhancing the robot's decision-making in complex human activity scenarios.

Figures & tables

Aspect	BTs	FSMs
Structure	Hierarchical tree of nodes	Flat collection of states and transitions
Modularity	High; subtrees can be reused colledanchise2017behavior	Low; difficult to reuse states
Scalability	Scales well with complexity	Poor scalability; prone to state explosion colledanchise2018behavior
Flexibility	Allows dynamic modification at runtime marzinotto2014towards	Static; changes require redesign
Debugging	Easier due to hierarchical nature	Can be difficult with complex transitions

Table 1: Comparison between Behavior Trees and Finite State Machines

Library	Language	ROS Integration	Key Features
BehaviorTree.CPP faconti2020behaviortreecpp	C++	Yes	High performance, Groot visualization, asynchronous actions
PyTrees stonier2015pytrees	Python	Yes	Ease of use, rapid prototyping, ROS support
BehaviorTree.NET	C#	Limited	Suitable for .NET applications

Table 2: Comparison of Behavior Tree Libraries

Aspect	Spot SDK	Spot ROS integration
Communication Protocol	gRPC-based API	ROS messages and services
Language Support	Multiple languages (Python, C++, Java, etc.) bostondynamics2020spotSDK	Primarily C++ and Python within ROS
Ease of Integration	Direct access to robot functionalities via SDK	Seamless integration with ROS-based systems and tools
Community Support	Official support from Boston Dynamics	Community-driven support and development
Tools and Libraries	SDK-specific tools; requires custom development	Access to ROS tools ( rviz , rosbag , tf ) and libraries

Table 3: Comparison between Spot SDK (gRPC) and ROS Integration

Figure 1: System’s block diagram for proactive human-assistance robot using contextual HAR and behavior Trees

Figure 2: Behavior tree for object handover: Spot grasps the object, approaches the human, initiates the handover by extending the manipulator, waits for the user to grasp, and opens the gripper to release and retract the arm.

Assistive Robotics

Assistive robotics develops robots and wearable devices that support mobility, daily activities, rehabilitation, and care.

See all 208 papers

SASI: Leveraging Sub-Action Semantics for Robust Early Action Recognition in Human-Robot Interaction
Apr 30, 2026Yongpeng Cao, Masahiro Hirano, Hyuno Kim +1Skeleton-Based Action Recognition Human Activity Recognition
Institute of Industrial Science, The University of Tokyo, Japan · Interfaculty Initiative in Information Studies, Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Japan
Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots
Sep 24, 2026Maithili Patel, Sonia ChernovaHuman-in-the-Loop Evaluation Robotic Policy Evaluation
School of Interactive Computing, Georgia Institute of Technology, USA
PACT: Proactive Asking for Continual Task Assistance in Human-Robot Collaboration
May 23, 2026Chengbo He, Sheng Li, Chenyang Ma +6Proactive Assistance Human-Robot Collaboration
University of Science and Technology Beijing · University of Oxford · 3Tsinghua University

Apr 30, 2026J/K move · Enter open · S save

SASI: Leveraging Sub-Action Semantics for Robust Early Action Recognition in Human-Robot Interaction

Yongpeng Cao, Masahiro Hirano, Hyuno Kim, Yuji Yamakawa

Institute of Industrial Science, The University of Tokyo, Japan · Interfaculty Initiative in Information Studies, Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Japan

Skeleton-Based Action Recognition Human Activity Recognition Video Action Recognition Human-Robot Interaction

Open paper AskPDF arXiv

Understanding human actions is critical for advancing behavior analysis in human-robot interaction. Particularly in tasks that demand quick and proactive feedback, robots must recognize human actions as early as possible from incomplete observations. Sub-actions offer the semantic and hierarchical cues needed for this, since human actions are inherently structured and can be decomposed into smaller, meaningful units. However, conventional approaches focus primarily on holistic actions and often overlook the rich semantic structure embedded in sub-actions, making them poorly suited for early recognition. To address this gap, we introduce SASI (Sub-Action Semantics Integrated cross-modal fusion), a novel framework that integrates existing graph convolution networks to fuse spatiotemporal features with sub-action semantics. SASI exploits a segmentation model with a traditional skeleton-based graph convolution network, capturing both fine-grained sub-action semantics and overall spatial context, while operating in real-time at 29 Hz. Experiments on BABEL, a skeleton-based dataset with frame-level annotations, demonstrate that our method improves recognition accuracy over conventional approaches, with additional gains expected as the quality of sub-action segmentation improves. Notably, SASI also achieves superior performance in understanding partial action sequences, revealing its capability for early recognition, which is essential for proactive and seamless Human-Robot Interaction (HRI). Code is available at https://anonymous.4open.science/r/SASI .

Abstract

Assistive Robotics

Explore similar work

SASI: Leveraging Sub-Action Semantics for Robust Early Action Recognition in Human-Robot Interaction

Robots That Take Initiative: A Framework for Building and Evaluating Proactive Robots

PACT: Proactive Asking for Continual Task Assistance in Human-Robot Collaboration