Organizations: Tongji University, Shanghai, China. · Shanghai Innovation Institute, Shanghai, China. · Shanghai Jiao Tong University, Shanghai, China. · Jiangxi University of Finance and Economics, China. · East China Normal University, Shanghai, China.
Controlling soccer robots involves multi-time-scale decision-making, which requires balancing long-term tactical planning and short-term motion execution. Traditional end-to-end reinforcement learning (RL) methods face challenges in complex dynamic environments. This paper proposes HierKick, a vision-guided soccer robot control framework based on dual-frequency hierarchical RL. The framework adopts a hierarchical control architecture featuring a 5 Hz high-level policy that integrates YOLOv8 for real-time detection and selects tasks via a coach model, and a pre-trained 50 Hz low-level controller for precise joint control. Through this architecture, the framework achieves the four steps of approaching, aligning, dribbling, and kicking. Experimental results show that the success rates of this framework are 95.2% in IsaacGym, 89.8% in Mujoco, and 80% in the real world. HierKick provides an effective hierarchical paradigm for robot control in complex environments, extendable to multi-time-scale tasks, with its modular design and skill reuse offering a new path for intelligent robot control.
Figures & tables
Figure 1: Robot Trajectory Diagram . This diagram depicts the trajectory of the robot on the football field. Guided by the Coach model and based on the HierKick framework, the robot executes a series of actions, including Approach, Alignment, Dribble, and Shoot.
Figure 2: Hierarchical control framework of HierKick . This framework consists of three interconnected modules: (1) Perception , where a YOLOv8 detector running at 10 Hz processes visual input to determine the positions of the soccer ball and goal (2) High-level control , featuring a “Coach Policy” operating at 5 Hz, which generates acceleration commands through PPO reinforcement learning by fusing perception results and historical control data (last acceleration and velocity commands) (3) Low-level execution , where a “Motion Policy” running at 50 Hz takes inputs like velocity commands, gait phase, and proprioception to execute sequential soccer actions (Approach, Alignment, Dribble, Shoot), with training guided by a multi-stage reward mechanism.
Category
Component
Mathematical Formulation
Description / Condition
Task-Specific Rewards (Phase-Dependent)
Approach
rapproach
(vrobot⋅v^robot-ball)⋅Mapproach
Encourage moving towards the ball ( d>u1 )
Alignment
ralignment
[cos(θrbg)−β∣Δωerr∣]⋅Malignment
Align robot-ball-goal vector ( u2<d≤u1 )
Dribble
rdribble
(vrobot⋅v^ball-goal)⋅Mdribble
Guide ball towards goal gently ( d1≤u2,d2>u3 )
Shoot
rshoot
[Aligngoal+μ⋅min(vmax,vshoot)]⋅Mshoot
Maximize kick velocity and accuracy ( d1≤u2,d2<u3 )
Standard & Regularization Rewards (Global)
Table 1: Summary of Multi-Stage and Standard Reward Functions . The total reward is a weighted sum of task-specific rewards (active only in specific phases) and standard regularization rewards (active globally).
Figure 3: Simulation and Real-World Environments . The environments for the football robot’s kicking task, including both simulated and real-world setups, are used for algorithm development, verification, and deployment.
Category
Parameter
Value
PPO Config
Discount Factor ( γ )
0.99
GAE Parameter ( λ )
0.95
Clip Range ( ϵ )
0.2
KL Target
0.01
Learning Rate
3×10−4
Batch Size
4096
Table 2: Hyperparameters and Configuration Details .
Compared Methods
Success Rate
HierKick
95.2%
HierKick w/o dball , dgoal
23.2%
HierKick ( cprev→vrobot-ball )
8.2%
End-to-End
25.6%
Table 3: Impact of Component Changes and HierKick Approaches (vs. End-to-End) on Model Success Rate
Figure 4: Success Rate Comparison . This graph denotes the comparison of task success rates across simulation environments and real-world deployment.
Figure 5: Horizontal Distance to Goal Center . Kick distance distributions under three conditions. HierKick achieves the smallest mean and most concentrated distribution, whereas Remove- and HierKick ( cprev→vrobot-ball ) experiment exhibit larger means. White diamonds denote means; error bars indicate one standard deviation. The dashed line at 0.5 m marks a reference threshold. Each condition has 5280 samples.
Figure 6: Reward Comparison . Comparison of Reward Evolution Between End-to-End and HierKick.
Humanoid soccer is a challenging testbed for dynamic whole-body control, requiring robots to coordinate balance, locomotion, object interaction, and skill switching over long horizons. Existing humanoid sports methods often rely on task-specific multi-stage pipelines, making it difficult to jointly learn and compose multiple object-interactive skills within a single deployable policy. To address this, we present SkillX, a unified reinforcement learning framework that learns and composes multiple atomic soccer skills through a single command-conditioned policy. SkillX integrates three core designs: skill-specific adversarial motion priors, skill-specific critics, and an object-aware temporal encoder, enabling the robot to execute atomic skills and transition among them such as dribbling, trapping, and shooting. Experiments in simulation and on a real Noetix E1 humanoid demonstrate robust multi-skill execution, long-horizon skill composition, and successful sim-to-real deployment.
Developing bipedal football robots in dynamiccombat environments presents challenges related to motionstability and deep coupling of multiple tasks, as well ascontrol switching issues between different states such as up-right walking and fall recovery. To address these problems,this paper proposes a modular reinforcement learning (RL)framework for achieving adaptive multi-task control. Firstly,this framework combines an open-loop feedforward oscilla-tor with a reinforcement learning-based feedback residualstrategy, effectively separating the generation of basic gaitsfrom complex football actions. Secondly, a posture-driven statemachine is introduced, clearly switching between the ballseeking and kicking network (BSKN) and the fall recoverynetwork (FRN), fundamentally preventing state interference.The FRN is efficiently trained through a progressive forceattenuation curriculum learning strategy. The architecture wasverified in Unity simulations of bipedal robots, demonstratingexcellent spatial adaptability-reliably finding and kicking theball even in restricted corner scenarios-and rapid autonomousfall recovery (with an average recovery time of 0.715 seconds).This ensures seamless and stable operation in complex multi-task environments.
Yulai Zhang, Yinrong Zhang, Ting Wu +1
School of Future Technology, Shanghai University, Shanghai, China
Recent advances in humanoid robotics have highlighted the importance of deployable loco-manipulation skills. Dribbling a soccer ball while evading active opponents requires simultaneous balance, precise ball control, and awareness of a dynamic adversary under onboard sensing and real-time constraints. Existing approaches typically separate perception and motion, which can be effective in controlled settings but may fail under occlusions, fast ball movements, and complex opponent interactions, since perception is not directly optimized for control. We propose an integrated approach in which a temporal depth encoder is embedded into a reinforcement learning policy through a task-specific projection layer. We apply this framework to a simulated Booster T1 humanoid robot and show that it is possible to learn vision-based, opponent-aware dribbling directly from depth observations, without explicit state estimation or privileged scene information. The learned policy achieves 100% success in nominal target-driven dribbling and 96% success with a single static obstacle, while reaching 46% success against an actively moving ball-attacker opponent. These results demonstrate that the proposed framework supports robust vision-based dribbling in nominal and moderately dynamic settings, and provides a strong foundation for handling more challenging moving-adversary scenarios.
Dept. of Computer, Control, and Management Engineering Sapienza University of Rome, Rome (Italy) · Institut de Robòtica i Informàtica Industrial (CSIC-UPC), C/ Llorens i Artigas 4-6, 08028, Barcelona, Spain · École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland