A Benchmarking Framework for Context-aware XR Interfaces
Authors: Hyunsung Cho, Sarah Yewon Yun, Nancy Ruonan Sun, Ben Lafreniere, Mark Parent, Kashyap Todi, Tanya R. Jonker, Hrvoje Benko, Sherry Tongshuang Wu, David Lindlbauer
Organizations: Carnegie Mellon University Pittsburgh, Pennsylvania, USA · Reality Labs Research, Meta Toronto, Ontario, Canada · Reality Labs Research, Meta Redmond, Washington, United States
Everyday Extended Reality (XR) systems aim to provide context-aware access to the right functionalities at the right time and place, with minimal manual reconfiguration as users switch context. Yet these interfaces are hard to evaluate: current prototyping and user-study workflows offer no systematic, repeatable way to compare adaptation methods across users and scenarios. We present ContextXR, a novel benchmarking framework for context-aware XR interfaces. ContextXR represents an XR application as a connected graph of functional facets, each a semantically coherent group of related capabilities that together support a shared user intent. On this representation, we build MineXR++, a dataset augmenting prior XR interface data with facet-level annotations, and formulate three canonical tasks of context-aware suggestion: context factor analysis, initial facet suggestion, and next facet suggestion. Our evaluation protocol scores suggestion methods by a simulated interaction metric, the navigation and search cost of reaching the desired functionality. Through experiments benchmarking global popularity, relational retrieval, and LLM-based methods, we demonstrate that ContextXR enables the systematic, reproducible evaluation of context-aware XR interfaces.
Figures & tables
Figure 1. We propose ContextXR , a benchmarking framework for developing and evaluating context-aware functionality suggestion in XR . We contribute (1) a representation of digital tools and information as functional facets , (2) MineXR++ , an augmented dataset that serves as ground truth, (3) a problem formulation of the canonical tasks of context-aware suggestion, and (4) an evaluation protocol built on a simulated interaction metric . Jointly, our approach enables principled evaluation of context-aware XR systems.
Figure 2. Top: Snippet of app–facet–capability structure. Bottom: Visualized example of facets and their component capabilities. A navigational capability (e. g., open_cook_recipe ) leads to another facet (e. g., Cook Recipe ).
Figure 3. Object schema relating functional facets to their app, capabilities, and context of use (user, task, environment). This schema defines the structure of MineXR++ and is traversed by the relational retrieval method in Section 4 .
Figure 4. Left: distribution of the dominant context factor across facets (user-, task-, environment-dominant, mixed, or no lift). Right: one representative facet per pattern, showing lift by context factor.
Table 5
Figure 5. Example of initial facet suggestion for a specific participant and context (“P04”, “relaxing”, “living room”). By comparing ground-truth facets in MineXR++ with suggestions from multiple methods, ContextXR makes differences in functional coverage explicit for analysis.