Smartphone-Aided VR Exergames for HIIT: Gamifying Fitness with Real-Time Pose Estimation
Authors: Abhiram Kothagundu, Pheelip Pritipal Raipure, Abhijit Chandrakant Patil, Himangshu Sarma, Christian Eichhorn, David A. Plecher, Johanna Pirker
Organizations: Human Computer Interaction Lab, Indian Institute of Information Technology Sri City, Andhra Pradesh 517646, India · Technical University of Munich, Munich, Germany
This paper introduces an innovative virtual reality (VR) exergame system designed to gamify High-Intensity Interval Training (HIIT) workouts through 11 predefined combo moves, including squat-punch combinations, bilateral arm strikes, and diagonal stretch combos. Unlike conventional tracking methods that often rely on wearable motion capture suits or limited headset-based sensors, our system uses a standard smartphone camera placed on a tripod to capture full-body movement. This approach offers a cost-effective and scalable alternative, particularly well-suited to the fast, dynamic motions typical of HIIT routines. The smartphone footage is processed in real time using a pose estimation algorithm, with extracted biomechanical data analyzed by a backend server built on the Model Context Protocol (MCP), Anthropic's open standard for connecting AI models to structured data sources and tools, and a GPT-4o large language model. The system generates personalized assessments of form accuracy, movement efficiency, exercise intensity, and potential injury risks, tailored specifically to high-impact, time-sensitive exercises. A within-subject study involving 20 participants (ages 18-25) compared experiences in VR and Non-VR HIIT sessions. Results indicated significantly higher enjoyment (Mean = 3.5 vs 2.8, p = 0.02), improved immersion/focus (Mean = 3.2 vs 2.8, p = 0.03), and increased likelihood to recommend (Mean = 3.2 vs 2.7, p = 0.04) under the VR condition. Confidence to continue did not differ significantly between conditions (p = 0.15), while perceived intensity remained comparable. The MCP system showed strong agreement with expert evaluation (87.5%, N = 10 sessions, one certified trainer), supporting its reliability as a feedback tool.
Figures & tables
Figure 1: Orb grid layout and combo mapping. Green orbs represent active targets in each combo.
Combo
Orbs
Direction
Movement Type
Combo_1
Orb_1, Orb_2
Horizontal
Lateral punch or side jab
Combo_2
Orb_3, Orb_4
Horizontal
Torso twist or mid strike
Combo_3
Orb_5, Orb_6
Horizontal
Side squat-punch
Combo_4
Orb_1, Orb_4
Diagonal
Upper diagonal hand stretch
Combo_5
Orb_3, Orb_2
Diagonal
Upper diagonal hand stretch
Combo_6
Orb_3, Orb_6
Diagonal
Lower diagonal stretch with squats
Table 1: Orb Position Mapping and Movement Direction
Figure 2: Three-phase interaction pipeline. Setup : smartphone calibration via MR passthrough overlay. Gameplay : reaction-based orb-combo loop with pose scoring per hit (repeats until session timer expires). Feedback : GPT-4o coaching report delivered to the Quest browser.
Figure 3: Setup Phase—player approaching the MR passthrough window in the virtual dojo to position the smartphone camera before session start.
Figure 4: Gameplay Phase—player performing combos as orbs approach in the dojo space.
Figure 5: Dual-stream system architecture. The smartphone streams continuous video frames via WebSocket; the Meta Quest 3 sends per-combo events via HTTP POST. The FastAPI Gateway fuses both streams by wall-clock timestamp. The Classifier MCP performs training-free pose recognition using a 13 × 13 distance matrix and cosine similarity; on session end, GPT-4o generates the coaching report delivered as a self-contained HTML page to the Quest browser.
Metric
VR Group
Non-VR Group
Motivation
2.8
2.6
Enjoyment
3.5
2.8
Focus/Immersion
3.2
2.8
Perceived Intensity
3.3
3.4
Confidence to Continue
2.9
2.8
Recommendation Likelihood
3.2
2.7
Table 2: Mean Post-Study Questionnaire Scores (Scale: 1–5)
Applying Human Pose Estimation (HPE) in real world environments remains a challenging task, this paper explores and surveys real time HPE approaches and their limitations in sports analysis for individuals, alongside developing a practical lightweight prototype for real world testing and usage. The older marker-based motion capture systems evolving to the modern accessible and adaptable markerless deep learning approaches, this survey explores the foundational architectures, which balance precision and efficiency. We also compare algorithmic frameworks (top-down, bottom-up, one-stage approaches, etc.) on practical deployment metrics such as inference latency, frame rate, mean per-joint position error, and temporal jitter to guide model selection process for sports application. As our prime contribution, we are proposing a modular, lightweight software prototype, which uses MediaPipe HPE framework with multiple exercise specific logic to deliver real-time insights and AI based feedback for non-expert users. We derive sports insights and providing feedback with minimal computational resources, while showcasing the performance and reliability metrics. In the end, we suggest other future research directions like combining sensors, and AR/VR. This work caters to researchers, engineers, sport scientists, etc., as both technical resource and a valid blueprint to implement a similar or improved real-time HPE analysis system for athletic performance enhancement or other purposes.
Parth Agrawal, Ronit, Sagar Kumar +1
Department of Computer Science and Engineering, Chandigarh University Punjab, India
We present the first motion generation system for playtesting virtual reality (VR) games. Our player model generates VR headset and handheld controller movements from in-game object arrangements, guided by style exemplars and aligned to maximize simulated gameplay score. We train on the large BOXRR-23 dataset and apply our framework on the popular VR game Beat Saber. The resulting model Robo-Saber produces skilled gameplay and captures diverse player behaviors, mirroring the skill levels and movement patterns specified by input style exemplars. Robo-Saber demonstrates promise in synthesizing rich gameplay data for predictive applications and enabling a physics-based whole-body VR playtesting agent.
Nam Hee Kim, Jingjing May Liu, Jaakko Lehtinen +3
Aalto University, Finland · University of California, Berkeley, United States · NVIDIA +1
Collecting embodied interaction data at scale remains costly and difficult due to the limited accessibility of conventional interfaces. We present a gamified data collection framework based on Unity that combines procedural scene generation, VR-based humanoid robot control, automatic task evaluation, and trajectory logging. A trash pick-and-place task prototype is developed to validate the full workflow.Experimental results indicate that the collected demonstrations exhibit broad coverage of the state-action space, and that increasing task difficulty leads to higher motion intensity as well as more extensive exploration of the arm's workspace. The proposed framework demonstrates that game-oriented virtual environments can serve as an effective and extensible solution for embodied data collection.
Yihan Zhang, Ziyun Huang, Linqi Ye
School of Computer Engineering and Science, Shanghai University, Shanghai, 200444, China · School of Future Technology, Shanghai University, Shanghai, 200444, China