HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control

Yizhi Chen; Zheng Zhang; Zhanxiang Cao; Yihe Chen; Shengcheng Fu; Liyun Yan; Yang Zhang; Jiali Liu

cs.ROMar 1, 2026

HierKick: Hierarchical Reinforcement Learning for Vision-Guided Soccer Robot Control

Authors: Yizhi Chen, Zheng Zhang, Zhanxiang Cao, Yihe Chen, Shengcheng Fu, Liyun Yan, Yang Zhang, Jiali Liu,

Organizations: Tongji University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Shanghai Jiao Tong University, Shanghai, China. · Jiangxi University of Finance and Economics, China. ·

Abstract

Controlling soccer robots involves multi-time-scale decision-making, which requires balancing long-term tactical planning and short-term motion execution. Traditional end-to-end reinforcement learning (RL) methods face challenges in complex dynamic environments. This paper proposes HierKick, a vision-guided soccer robot control framework based on dual-frequency hierarchical RL. The framework adopts a hierarchical control architecture featuring a 5 Hz high-level policy that integrates YOLOv8 for real-time detection and selects tasks via a coach model, and a pre-trained 50 Hz low-level controller for precise joint control. Through this architecture, the framework achieves the four steps of approaching, aligning, dribbling, and kicking. Experimental results show that the success rates of this framework are 95.2% in IsaacGym, 89.8% in Mujoco, and 80% in the real world. HierKick provides an effective hierarchical paradigm for robot control in complex environments, extendable to multi-time-scale tasks, with its modular design and skill reuse offering a new path for intelligent robot control.

Figures & tables

Figure 1: Robot Trajectory Diagram . This diagram depicts the trajectory of the robot on the football field. Guided by the Coach model and based on the HierKick framework, the robot executes a series of actions, including Approach, Alignment, Dribble, and Shoot.

Figure 2: Hierarchical control framework of HierKick . This framework consists of three interconnected modules: (1) Perception , where a YOLOv8 detector running at 10 Hz processes visual input to determine the positions of the soccer ball and goal (2) High-level control , featuring a “Coach Policy” operating at 5 Hz, which generates acceleration commands through PPO reinforcement learning by fusing perception results and historical control data (last acceleration and velocity commands) (3) Low-level execution , where a “Motion Policy” running at 50 Hz takes inputs like velocity commands, gait phase, and proprioception to execute sequential soccer actions (Approach, Alignment, Dribble, Shoot), with training guided by a multi-stage reward mechanism.

Category	Component	Mathematical Formulation	Description / Condition
Task-Specific Rewards (Phase-Dependent)
Approach	$r_{\text{approach}}$	$(\mathbf{v}_{\text{robot}}\cdot\hat{\mathbf{v}}_{\text{robot-ball}})\cdot M_{\text{approach}}$	Encourage moving towards the ball ( $d>u_{1}$ )
Alignment	$r_{\text{alignment}}$	$[\cos(\theta_{\text{rbg}})-\beta\|\Delta\omega_{\text{err}}\|]\cdot M_{\text{alignment}}$	Align robot-ball-goal vector ( $u_{2}<d\leq u_{1}$ )
Dribble	$r_{\text{dribble}}$	$(\mathbf{v}_{\text{robot}}\cdot\hat{\mathbf{v}}_{\text{ball-goal}})\cdot M_{\text{dribble}}$	Guide ball towards goal gently ( $d_{1}\leq u_{2},d_{2}>u_{3}$ )
Shoot	$r_{\text{shoot}}$	$[\text{Align}_{\text{goal}}+\mu\cdot\min(v_{\text{max}},v_{\text{shoot}})]\cdot M_{\text{shoot}}$	Maximize kick velocity and accuracy ( $d_{1}\leq u_{2},d_{2}<u_{3}$ )
Standard & Regularization Rewards (Global)

Table 1: Summary of Multi-Stage and Standard Reward Functions . The total reward is a weighted sum of task-specific rewards (active only in specific phases) and standard regularization rewards (active globally).

Figure 3: Simulation and Real-World Environments . The environments for the football robot’s kicking task, including both simulated and real-world setups, are used for algorithm development, verification, and deployment.

Category	Parameter	Value
PPO Config	Discount Factor ( $\gamma$ )	0.99
	GAE Parameter ( $\lambda$ )	0.95
	Clip Range ( $\epsilon$ )	0.2
	KL Target	0.01
	Learning Rate	$3\times 10^{-4}$
	Batch Size	4096

Table 2: Hyperparameters and Configuration Details .

Compared Methods	Success Rate
HierKick	95.2%
HierKick w/o $\mathbf{d}_{\text{ball}}$ , $\mathbf{d}_{\text{goal}}$	23.2%
HierKick ( $\mathbf{c}_{\text{prev}}\!\to\!\mathbf{v}_{\text{robot-ball}}$ )	8.2%
End-to-End	25.6%

Table 3: Impact of Component Changes and HierKick Approaches (vs. End-to-End) on Model Success Rate

Figure 4: Success Rate Comparison . This graph denotes the comparison of task success rates across simulation environments and real-world deployment.

Figure 5: Horizontal Distance to Goal Center . Kick distance distributions under three conditions. HierKick achieves the smallest mean and most concentrated distribution, whereas Remove- and HierKick ( $\mathbf{c}_{\text{prev}}\!\to\!\mathbf{v}_{\text{robot-ball}}$ ) experiment exhibit larger means. White diamonds denote means; error bars indicate one standard deviation. The dashed line at 0.5 m marks a reference threshold. Each condition has 5280 samples.

Figure 6: Reward Comparison . Comparison of Reward Evolution Between End-to-End and HierKick.

Robotic Control

Robotic control studies how machines use sensing and computation to produce reliable movement and interaction in changing environments.

See all 567 papers

SkillX: Unified Multi-Skill Policy Learning for Humanoid Soccer
Sep 6, 2026Zhangchen Ye, Enxuan Ruan, Yifei Bao +10Robot Skill Learning Humanoid Robot Control
Noetix Robotics · Tsinghua University
Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots
Apr 21, 2026Yulai Zhang, Yinrong Zhang, Ting Wu +1Legged Robotics Humanoid Robot Control
School of Future Technology, Shanghai University, Shanghai, China
Vision-Based Dribbling for Humanoid Soccer via Privileged Representation Learning
Jul 14, 2026Flavio Maiorana, Valerio Spagnoli, Eugenio Bugli +5RL for Robotics Reinforcement Learning
Dept. of Computer, Control, and Management Engineering Sapienza University of Rome, Rome (Italy) · Institut de Robòtica i Informàtica Industrial (CSIC-UPC), C/ Llorens i Artigas 4-6, 08028, Barcelona, Spain · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

Sep 6, 2026J/K move · Enter open · S save

SkillX: Unified Multi-Skill Policy Learning for Humanoid Soccer

Zhangchen Ye, Enxuan Ruan, Yifei Bao, Runhan Huang+9

Noetix Robotics · Tsinghua University

Robot Skill Learning Humanoid Robot Control Robotics Multi-Task RL Agent Skill Learning

Open paper AskPDF arXiv

Humanoid soccer is a challenging testbed for dynamic whole-body control, requiring robots to coordinate balance, locomotion, object interaction, and skill switching over long horizons. Existing humanoid sports methods often rely on task-specific multi-stage pipelines, making it difficult to jointly learn and compose multiple object-interactive skills within a single deployable policy. To address this, we present SkillX, a unified reinforcement learning framework that learns and composes multiple atomic soccer skills through a single command-conditioned policy. SkillX integrates three core designs: skill-specific adversarial motion priors, skill-specific critics, and an object-aware temporal encoder, enabling the robot to execute atomic skills and transition among them such as dribbling, trapping, and shooting. Experiments in simulation and on a real Noetix E1 humanoid demonstrate robust multi-skill execution, long-horizon skill composition, and successful sim-to-real deployment.

Abstract

Robotic Control

Explore similar work

SkillX: Unified Multi-Skill Policy Learning for Humanoid Soccer

Reinforcement Learning Enabled Adaptive Multi-Task Control for Bipedal Soccer Robots

Vision-Based Dribbling for Humanoid Soccer via Privileged Representation Learning