Designing Fatigue-Aware VR Interfaces via Biomechanical Models

Harshitha Voleti; Charalambos Poullis

cs.HCMar 27, 2026

Designing Fatigue-Aware VR Interfaces via Biomechanical Models

Authors: Harshitha Voleti, Charalambos Poullis

Organizations: Immersive & Creative Technologies Lab, Concordia University, Canada

Abstract

Prolonged mid-air interaction in virtual reality (VR) causes arm fatigue and discomfort, negatively affecting user experience. Incorporating ergonomic considerations into VR user interface (UI) design typically requires extensive human-in-the-loop evaluation. Although biomechanical models have been used to simulate human behavior in HCI tasks, their application as surrogate users for ergonomic VR UI design remains underexplored. We propose a hierarchical reinforcement learning framework that leverages biomechanical user models to evaluate and optimize VR interfaces for mid-air interaction. A motion agent is trained to perform button-press tasks in VR under sequential conditions, using realistic movement strategies and estimating muscle-level effort via a validated three-compartment control with recovery (3CC-r) fatigue model. The simulated fatigue output serves as feedback for a UI agent that optimizes UI element layout via reinforcement learning (RL) to minimize fatigue. We compare the RL-optimized layout against a manually-designed centered baseline and a Bayesian optimized baseline. Results show that fatigue trends from the biomechanical model align with human user data. Moreover, the RL-optimized layout using simulated fatigue feedback produced significantly lower perceived fatigue in a follow-up human study. We further demonstrate the framework's extensibility via a simulated case study on longer sequential tasks with non-uniform interaction frequencies. To our knowledge, this is the first work using simulated biomechanical muscle fatigue as a direct optimization signal for VR UI layout design. Our findings highlight the potential of biomechanical user models as effective surrogate tools for ergonomic VR interface design, enabling efficient early-stage iteration with less reliance on extensive human participation.

Figures & tables

Figure 1. Overview of the simulation-driven UI optimization framework. A high-level UI agent proposes discrete interface layouts by selecting positions on a predefined grid. Each candidate layout is evaluated by a pre-trained biomechanical motion agent that executes the interaction task in simulation. Muscle-level fatigue is estimated using the 3CC-r model and aggregated to form a reward signal, which enables iterative optimization of UI layouts without labeled data or hand-crafted ergonomic heuristics. Block diagram illustrating a closed-loop simulation-driven UI optimization process. A high-level UI agent selects three discrete grid positions to place interface elements. These positions are passed to a virtual reality environment, where a pre-trained biomechanical motion agent executes the interaction task. The motion agent estimates muscle-level fatigue using a biomechanical model. Aggregated fatigue values are returned to the UI agent as feedback, forming a loop that supports iterative refinement of interface layouts.

Figure 2. Overview of the proposed hierarchical framework . A high-level UI agent proposes discrete interface layouts and evaluates them using cumulative fatigue feedback from a simulated user. Each candidate layout is instantiated in the Unity-based VR application through the SIM2VR task module. Rendered observations are provided to the simulated user’s perception module, which is fed into a learned control policy that outputs muscle activation signals. A low-level motion agent controls a biomechanical model to execute the interaction sequence under the proposed layout. Muscle activations are logged during execution and converted into fatigue estimates. The cumulative fatigue over the completed sequence is returned to the UI agent as a reward signal, together with the resulting layout state, to guide subsequent layout optimization. Block diagram illustrating a hierarchical VR interface optimization pipeline. A high-level UI agent proposes candidate button layouts. Each layout is evaluated by a simulated user implemented through the SIM2VR pipeline. Within this pipeline, a low-level motion agent receives rendered observations from a Unity-based VR application, processes them through a perception module and learned control policy, and outputs muscle control signals applied to a biomechanical arm model. During task execution, muscle activations are logged and converted into fatigue estimates. Aggregated fatigue values are returned to the UI agent, forming a closed feedback loop for iterative layout optimization.

Figure 3. Overview of the motion agent used to simulate interaction. The motion agent generates muscle control signals using an RL policy and interacts with the VR application, which updates the scene and returns rendered observations, reward, and state information. Muscle activations produced during task execution are logged and later converted into fatigue estimates, which are used to evaluate interface layouts. Diagram illustrating the motion agent control loop used to simulate human interaction. The motion agent receives RGB-D images and proprioceptive state information from a Unity-based VR application. These observations are processed by a learned policy that outputs continuous muscle control signals applied to a biomechanical arm model. As the arm moves and presses UI elements, muscle activations are logged and later converted into fatigue estimates. The environment updates the scene and returns new observations, allowing the loop to repeat over the interaction sequence.

Figure 4. Distance-based shaping term used for reward design.

Figure 5. Overview of the UI agent responsible for interface layout optimization. The agent proposes discrete grid-based button layouts, which are instantiated in the VR environment and evaluated by the motion agent. Aggregated fatigue is returned as a reward signal, along with the next state of the UI agent defined by the grid coordinates of all button positions. Diagram showing the UI agent workflow. The UI agent selects grid-based positions for multiple UI buttons and instantiates the resulting layout in the virtual environment. A simulated user then executes the interaction sequence after which the UI agent receives aggregated fatigue feedback along with the next state, defined by the x,y coordinates of the current button positions. This feedback is used to evaluate and refine future layout proposals.

Panel of Figure 6 — Figure 6. UI configurations evaluated in the study. (a) RL-based layout optimized using biomechanical fatigue feedback, (b) Bayesian optimization (BO) layout minimizing accumulated effort, and (c) static layout with centrally placed buttons. Three UI layout visualizations are shown. The left image shows the RL-based layout, with three buttons placed close together near the right edge of the grid. The middle image shows the Bayesian-optimized layout, where buttons are spread across different grid cells. The right image shows the static baseline layout, with three buttons placed centrally on the grid.

User	Fatigue Type	UI	Mean	Std.	Comparison	Wilcoxon Signed Rank
						Z	p
Simulated User	Accumulated effort ( $F$ )	UI_Static	30.4513	4.0708	$UI_{Static}<UI_{BO}$	-2.910	$<0.01$
		UI_BO	33.0482	1.3801	$UI_{RL}<UI_{BO}$	-4.782	$<0.001$
		UI_RL	22.8955	0.7652	$UI_{RL}<UI_{Static}$	-4.782	$<0.001$
User Study	Borg CR10	UI_Static	3.639	2.2016	$UI_{Static}<UI_{BO}$	-2.875	$<0.01$
		UI_BO	5.94	1.662	$UI_{RL}<UI_{BO}$	-3.598	$<0.001$

Table 1. Descriptive statistics and Wilcoxon signed-rank post-hoc comparisons of fatigue across UI conditions.

Figure 7. NASA-TLX workload ratings across the three UI configurations. Bars show mean scores for each NASA-TLX subscale, aggregated across participants. Lower values indicate lower perceived workload. Bar chart showing mean NASA-TLX subscale scores for three UI configurations: Static, Bayesian-optimized, and RL-based. Subscales include Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, and Frustration.

Subscale	Comparison	Z	p-value
Overall Scores	$UI_{RL}\text{\textendash}UI_{Static}$	-2.172	$<0.05$
	$UI_{RL}\text{\textendash}UI_{BO}$	-2.461	$<0.05$
	$UI_{BO}\text{\textendash}UI_{Static}$	-2.344	$<0.05$
Effort	$UI_{RL}\text{\textendash}UI_{Static}$	-1.998	$<0.05$
	$UI_{RL}\text{\textendash}UI_{BO}$	-2.594	$<0.01$
	$UI_{BO}\text{\textendash}UI_{Static}$	-1.827	n.s

Table 2. Post-hoc Wilcoxon signed-rank test results for NASA-TLX subscales.

Figure 8. Sequential five-button interaction task. All five buttons are displayed simultaneously, and the agent performs selections in the order specified by the visual marker. Image showing the motion agent interacting with the 5-button task. A small spherical marker indicates the current target button. The simulated controller moves toward the marked button and presses it, after which the marker advances to the next button in the predefined sequence. The image also illustrates the low-resolution rendering (120×80 pixels) that represents the input observed by the motion agent.

Virtual Reality

Virtual reality studies immersive, interactive environments and the technologies and human experiences that make them useful.

See all 40 papers

MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning
Feb 16, 2026Ankit Bhattarai, Hannah Selder, Florian Fischer +2Human-Computer Interaction Reinforcement Learning
Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig University Leipzig, Germany
FatigueFusion: Latent Space Fusion for Fatigue-Driven Motion Synthesis
Apr 11, 2026Iliana Loi, Konstantinos MoustakasHuman Motion Generation
Department of Electrical and Computer Engineering, University of Patras, Patras, 26504 Greece
Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models
Feb 17, 2026Feras Kiki, Pouya P. Niaz, Alireza Madani +1Human-Robot Collaboration
F. Kiki, P. P. Niaz, A. Madani, and C. Basdogan (corresponding author) are with the Robotics and Mechatronics Laboratory (RML) and the KUIS AI Center, Koc University, Sariyer, Istanbul 34450, Turkey.

Feb 16, 2026J/K move · Enter open · S save

MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning

Ankit Bhattarai, Hannah Selder, Florian Fischer, Arthur Fleig+1

Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI) Dresden/Leipzig, Leipzig University Leipzig, Germany

Human-Computer Interaction Reinforcement Learning Human Behavior Simulation Musculoskeletal Modeling

Open paper AskPDF arXiv

Reinforcement learning (RL)-based biomechanical simulations have the potential to revolutionise HCI research and interaction design, but currently lack usability and interpretability. Using the Human Action Cycle as a design lens, we identify key limitations of biomechanical RL frameworks and develop MyoInteract, a novel framework for fast prototyping of biomechanical HCI tasks. MyoInteract allows designers to setup tasks, user models, and training parameters from an easy-to-use GUI within minutes. It trains and evaluates muscle-actuated simulated users within minutes, reducing training times by up to 98%. A workshop study with 12 interaction designers revealed that MyoInteract allowed novices in biomechanical RL to successfully setup, train, and assess goal-directed user movements within a single session. By transforming biomechanical RL from a days-long expert task into an accessible hour-long workflow, this work significantly lowers barriers to entry and accelerates iteration cycles in HCI biomechanics research.

Abstract

Virtual Reality

Explore similar work

MyoInteract: A Framework for Fast Prototyping of Biomechanical HCI Tasks using Reinforcement Learning

FatigueFusion: Latent Space Fusion for Fatigue-Driven Motion Synthesis

Estimating Human Muscular Fatigue in Dynamic Collaborative Robotic Tasks with Learning-Based Models