Designing Fatigue-Aware VR Interfaces via Biomechanical Models

Harshitha Voleti; Charalambos Poullis

cs.HCMar 27, 2026

Designing Fatigue-Aware VR Interfaces via Biomechanical Models

Authors: Harshitha Voleti, Charalambos Poullis

Organizations: Immersive & Creative Technologies Lab, Concordia University, Canada

Abstract

Prolonged mid-air interaction in virtual reality (VR) causes arm fatigue and discomfort, negatively affecting user experience. Incorporating ergonomic considerations into VR user interface (UI) design typically requires extensive human-in-the-loop evaluation. Although biomechanical models have been used to simulate human behavior in HCI tasks, their application as surrogate users for ergonomic VR UI design remains underexplored. We propose a hierarchical reinforcement learning framework that leverages biomechanical user models to evaluate and optimize VR interfaces for mid-air interaction. A motion agent is trained to perform button-press tasks in VR under sequential conditions, using realistic movement strategies and estimating muscle-level effort via a validated three-compartment control with recovery (3CC-r) fatigue model. The simulated fatigue output serves as feedback for a UI agent that optimizes UI element layout via reinforcement learning (RL) to minimize fatigue. We compare the RL-optimized layout against a manually-designed centered baseline and a Bayesian optimized baseline. Results show that fatigue trends from the biomechanical model align with human user data. Moreover, the RL-optimized layout using simulated fatigue feedback produced significantly lower perceived fatigue in a follow-up human study. We further demonstrate the framework's extensibility via a simulated case study on longer sequential tasks with non-uniform interaction frequencies. To our knowledge, this is the first work using simulated biomechanical muscle fatigue as a direct optimization signal for VR UI layout design. Our findings highlight the potential of biomechanical user models as effective surrogate tools for ergonomic VR interface design, enabling efficient early-stage iteration with less reliance on extensive human participation.

Figures & tables

Figure 1. Overview of the simulation-driven UI optimization framework. A high-level UI agent proposes discrete interface layouts by selecting positions on a predefined grid. Each candidate layout is evaluated by a pre-trained biomechanical motion agent that executes the interaction task in simulation. Muscle-level fatigue is estimated using the 3CC-r model and aggregated to form a reward signal, which enables iterative optimization of UI layouts without labeled data or hand-crafted ergonomic heuristics. Block diagram illustrating a closed-loop simulation-driven UI optimization process. A high-level UI agent selects three discrete grid positions to place interface elements. These positions are passed to a virtual reality environment, where a pre-trained biomechanical motion agent executes the interaction task. The motion agent estimates muscle-level fatigue using a biomechanical model. Aggregated fatigue values are returned to the UI agent as feedback, forming a loop that supports iterative refinement of interface layouts.

Figure 2. Overview of the proposed hierarchical framework . A high-level UI agent proposes discrete interface layouts and evaluates them using cumulative fatigue feedback from a simulated user. Each candidate layout is instantiated in the Unity-based VR application through the SIM2VR task module. Rendered observations are provided to the simulated user’s perception module, which is fed into a learned control policy that outputs muscle activation signals. A low-level motion agent controls a biomechanical model to execute the interaction sequence under the proposed layout. Muscle activations are logged during execution and converted into fatigue estimates. The cumulative fatigue over the completed sequence is returned to the UI agent as a reward signal, together with the resulting layout state, to guide subsequent layout optimization. Block diagram illustrating a hierarchical VR interface optimization pipeline. A high-level UI agent proposes candidate button layouts. Each layout is evaluated by a simulated user implemented through the SIM2VR pipeline. Within this pipeline, a low-level motion agent receives rendered observations from a Unity-based VR application, processes them through a perception module and learned control policy, and outputs muscle control signals applied to a biomechanical arm model. During task execution, muscle activations are logged and converted into fatigue estimates. Aggregated fatigue values are returned to the UI agent, forming a closed feedback loop for iterative layout optimization.

Figure 3. Overview of the motion agent used to simulate interaction. The motion agent generates muscle control signals using an RL policy and interacts with the VR application, which updates the scene and returns rendered observations, reward, and state information. Muscle activations produced during task execution are logged and later converted into fatigue estimates, which are used to evaluate interface layouts. Diagram illustrating the motion agent control loop used to simulate human interaction. The motion agent receives RGB-D images and proprioceptive state information from a Unity-based VR application. These observations are processed by a learned policy that outputs continuous muscle control signals applied to a biomechanical arm model. As the arm moves and presses UI elements, muscle activations are logged and later converted into fatigue estimates. The environment updates the scene and returns new observations, allowing the loop to repeat over the interaction sequence.

Figure 4. Distance-based shaping term used for reward design.

Figure 5. Overview of the UI agent responsible for interface layout optimization. The agent proposes discrete grid-based button layouts, which are instantiated in the VR environment and evaluated by the motion agent. Aggregated fatigue is returned as a reward signal, along with the next state of the UI agent defined by the grid coordinates of all button positions. Diagram showing the UI agent workflow. The UI agent selects grid-based positions for multiple UI buttons and instantiates the resulting layout in the virtual environment. A simulated user then executes the interaction sequence after which the UI agent receives aggregated fatigue feedback along with the next state, defined by the x,y coordinates of the current button positions. This feedback is used to evaluate and refine future layout proposals.

Panel of Figure 6 — Figure 6. UI configurations evaluated in the study. (a) RL-based layout optimized using biomechanical fatigue feedback, (b) Bayesian optimization (BO) layout minimizing accumulated effort, and (c) static layout with centrally placed buttons. Three UI layout visualizations are shown. The left image shows the RL-based layout, with three buttons placed close together near the right edge of the grid. The middle image shows the Bayesian-optimized layout, where buttons are spread across different grid cells. The right image shows the static baseline layout, with three buttons placed centrally on the grid.

User	Fatigue Type	UI	Mean	Std.	Comparison	Wilcoxon Signed Rank
						Z	p
Simulated User	Accumulated effort ( $F$ )	UI_Static	30.4513	4.0708	$UI_{Static}<UI_{BO}$	-2.910	$<0.01$
		UI_BO	33.0482	1.3801	$UI_{RL}<UI_{BO}$	-4.782	$<0.001$
		UI_RL	22.8955	0.7652	$UI_{RL}<UI_{Static}$	-4.782	$<0.001$
User Study	Borg CR10	UI_Static	3.639	2.2016	$UI_{Static}<UI_{BO}$	-2.875	$<0.01$
		UI_BO	5.94	1.662	$UI_{RL}<UI_{BO}$	-3.598	$<0.001$

Table 1. Descriptive statistics and Wilcoxon signed-rank post-hoc comparisons of fatigue across UI conditions.

Figure 7. NASA-TLX workload ratings across the three UI configurations. Bars show mean scores for each NASA-TLX subscale, aggregated across participants. Lower values indicate lower perceived workload. Bar chart showing mean NASA-TLX subscale scores for three UI configurations: Static, Bayesian-optimized, and RL-based. Subscales include Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.

Subscale	Comparison	Z	p-value
Overall Scores	$UI_{RL}\text{\textendash}UI_{Static}$	-2.172	$<0.05$
	$UI_{RL}\text{\textendash}UI_{BO}$	-2.461	$<0.05$
	$UI_{BO}\text{\textendash}UI_{Static}$	-2.344	$<0.05$
Effort	$UI_{RL}\text{\textendash}UI_{Static}$	-1.998	$<0.05$
	$UI_{RL}\text{\textendash}UI_{BO}$	-2.594	$<0.01$
	$UI_{BO}\text{\textendash}UI_{Static}$	-1.827	n.s

Table 2. Post-hoc Wilcoxon signed-rank test results for NASA-TLX subscales.

Figure 8. Sequential five-button interaction task. All five buttons are displayed simultaneously, and the agent performs selections in the order specified by the visual marker. Image showing the motion agent interacting with the 5-button task. A small spherical marker indicates the current target button. The simulated controller moves toward the marked button and presses it, after which the marker advances to the next button in the predefined sequence. The image also illustrates the low-resolution rendering (120×80 pixels) that represents the input observed by the motion agent.

Virtual Reality

Virtual reality studies immersive, interactive environments and the technologies and human experiences that make them useful.

See all 40 papers

Feb 16, 2026

MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning

Reinforcement learning (RL)-based biomechanical simulations have the potential to revolutionise HCI research and interaction design, but currently lack usability and interpretability. Using the Human Action Cycle as a design lens, we identify key limitations of biomechanical RL frameworks and develop MyoInteract, a novel framework for fast prototyping of biomechanical HCI tasks. MyoInteract allows designers to setup tasks, user models, and training parameters from an easy-to-use GUI within minutes. It trains and evaluates muscle-actuated simulated users within minutes, reducing training times by up to 98%. A workshop study with 12 interaction designers revealed that MyoInteract allowed novices in biomechanical RL to successfully setup, train, and assess goal-directed user movements within a single session. By transforming biomechanical RL from a days-long expert task into an accessible hour-long workflow, this work significantly lowers barriers to entry and accelerates iteration cycles in HCI biomechanics research.

Ankit Bhattarai, Hannah Selder, Florian Fischer +2

Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig University Leipzig, Germany

Human-Computer Interaction Reinforcement Learning Human Behavior Simulation Musculoskeletal Modeling

Ask PDF

Apr 11, 2026

FatigueFusion: Latent Space Fusion for Fatigue-Driven Motion Synthesis

Investigating the impact of fatigue on human physiological function and motor behavior is crucial for developing biomechanics and medical applications aimed at mitigating fatigue, reducing injury risk, and creating sophisticated ergonomic designs, as well as for producing physically-plausible 3D animation sequences. While the former has a prominent position in state-of-the-art literature, fatigue-driven motion generation is still an underexplored area. In this study, we present FatigueFusion, a deep-learning architecture for the fusion of fatigue features within a latent representation space, enabling the creation of a variation of novel fatigued movements, intermediate fatigued states, and progressively fatigued motions. Unlike existing approaches that focus on imitating the effects of fatigue accumulation in motion patterns, our framework incorporates algorithmic and data-driven modules to impose subject-specific temporal and spatial fatigue features on nonfatigued motions, while leveraging PINN-based techniques to simulate fatigue intensity. Since all motion modulation tasks are taking place in latent space, FatigueFusion offers an end-to-end architecture that operates directly on non-fatigued joint angle sequences and control parameters, allowing seamless integration into any motion synthesis pipeline, without relying on fatigue input data. Overall, our framework can be employed for various fatigue-driven synthesis tasks, such as fatigue profile transfer and fusion, while it also provides a solution for accurate rendering of the human fatigue state in both animation and simulation pipelines.

Iliana Loi, Konstantinos Moustakas

Department of Electrical and Computer Engineering, University of Patras, Patras, 26504 Greece

Human Motion Generation

Ask PDF

Feb 17, 2026

Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models

Assessing human muscle fatigue is critical for optimizing performance and safety in physical human-robot interaction(pHRI). This work presents a data-driven framework to estimate fatigue in dynamic, cyclic pHRI using arm-mounted surface electromyography(sEMG). Subject-specific machine-learning regression models(Random Forest, XGBoost, and Linear Regression predict the fraction of cycles to fatigue(FCF) from three frequency-domain and one time-domain EMG features, and are benchmarked against a convolutional neural network(CNN) that ingests spectrograms of filtered EMG. Framing fatigue estimation as regression (rather than classification) captures continuous progression toward fatigue, supporting earlier detection, timely intervention, and adaptive robot control. In experiments with ten participants, a collaborative robot under admittance control guided repetitive lateral (left-right) end-effector motions until muscular fatigue. Average FCF RMSE across participants was 20.8+/-4.3% for the CNN, 23.3+/-3.8% for Random Forest, 24.8+/-4.5% for XGBoost, and 26.9+/-6.1% for Linear Regression. To probe cross-task generalization, one participant additionally performed unseen vertical (up-down) and circular repetitions; models trained only on lateral data were tested directly and largely retained accuracy, indicating robustness to changes in movement direction, arm kinematics, and muscle recruitment, while Linear Regression deteriorated. Overall, the study shows that both feature-based ML and spectrogram-based DL can estimate remaining work capacity during repetitive pHRI, with the CNN delivering the lowest error and the tree-based models close behind. The reported transfer to new motion patterns suggests potential for practical fatigue monitoring without retraining for every task, improving operator protection and enabling fatigue-aware shared autonomy, for safer fatigue-adaptive pHRI control.

Feras Kiki, Pouya P. Niaz, Alireza Madani +1

F. Kiki, P. P. Niaz, A. Madani, and C. Basdogan (corresponding author) are with the Robotics and Mechatronics Laboratory (RML) and the KUIS AI Center, Koc University, Sariyer, Istanbul 34450, Turkey.

Human-Robot Collaboration

Ask PDF

Abstract

Virtual Reality

Explore similar work

MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning

FatigueFusion: Latent Space Fusion for Fatigue-Driven Motion Synthesis

Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models