GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development

William Huang; David Li; Xiang 'Anthony' Chen; Xun Qian; Nels Numan; Zhongyi Zhou; Hyunsung Cho; Salman Muin Kayser Chishti

cs.AIOct 21, 2026

GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development

Authors: William Huang, David Li, Xiang 'Anthony' Chen, Xun Qian, Nels Numan, Zhongyi Zhou, Hyunsung Cho, Salman Muin Kayser Chishti,

Organizations: University of California, Los Angeles · Google XR Labs · Carnegie Mellon University · University of Oxford ·

Abstract

Coding agents are rapidly advancing from single-file generation to long-horizon repository-scale engineering, yet the evaluation of their capabilities to author interactive extended reality (XR) applications remains an open challenge. Unlike traditional software, XR applications operate within dynamic 3D physical environments where runtime robustness depends jointly on continuous user motion, spatial input, and environmental sensing. We introduce GenXR-Bench, the first benchmark for the end-to-end evaluation of coding agents in XR software development. GenXR-Bench contains 112 expert-authored tasks spanning 24 feature implementations, 21 bug repairs, and 67 application generation tasks. Held-out tests combine scripted and adaptive interactions with programmatic and visual assertions to evaluate core functionality and runtime behavior. GenXR-Bench's testing suite achieves 96.5% accuracy against blinded expert judgments of 124 candidate applications executed on consumer XR hardware. Evaluating 10 models on GenXR-Bench, we find that the strongest configuration achieves only 50.9% overall Pass@1. Failure analysis identifies recurring misunderstandings in asynchronous sensor lifecycles, physical grounding, and embodied interaction semantics. GenXR-Bench establishes a reproducible testbed for evaluating coding agents in the interactive and physically situated conditions of spatial computing.

Figures & tables

Figure 1: We present GenXR-Bench, a benchmark to evaluate an agent’s ability to understand and create XR apps. All visualizations shown were captured on device.

Domain	Benchmark	Primary tasks	ER	IT	VT	Evaluation
SWE	SWE-bench	Repository repair	✗	✗	✗	Repository tests
	( Jimenez et al., 2024 )		✗	✗	✗
	SWE-bench Multimodal	Repository repair	✗	✓	✓	Repository tests; UI-event checks; pixel-level visual tests
	( Yang et al., 2025 )		✗	✓	✓
Web	Interaction2Code	Interactive page reconstruction	✗	✓	✓	Interaction tests; visual metrics; human assessment
	( Xiao et al., 2025 )		✗	✓	✓

Table 1: Comparison with representative coding benchmarks. ER : embodied reasoning; IT : interaction-based testing; VT : visual testing. ER refers to evaluated capabilities of the benchmark (e.g., applications related to physical user interactions or spatial computing) while IT and VT refer to evaluation methods.

Figure 2: Authoring workflow of GenXR-Bench. Human icon indicates human-authored and robot icon indicates LLM-authored. Both icons indicate LLM generated with human guidance.

Figure 3: Composition of GenXR-Bench. a) Distribution of task type and difficulty. The donut’s inner ring shows task type, while the outer ring shows easy (E), medium (M), and hard (H) task counts within each type. b) Distribution of XR capabilities evaluated across tasks. Each task can have multiple capability labels.

Figure 4: Example of a human-written test for the bumble-calm task, which asks coding agents to implement a simple calming game where users hold a palm up to summon a bee.

App Source	$\uparrow$ Accuracy $\uparrow$	$\uparrow$ Precision $\uparrow$	$\uparrow$ Recall $\uparrow$	$\uparrow$ Fleiss’ $\kappa$ $\uparrow$	$\uparrow$ Flakiness $\downarrow$
gemini-3.5-flash	$0.962\pm 0.009$	$0.958\pm 0.072$	$0.778\pm 0.000$	$0.948$	$0.016$
gemini-3.7-flash	$0.968\pm 0.032$	$1.000\pm 0.000$	$0.923\pm 0.077$	$0.909$	$0.065$
Overall	$\mathbf{0.965\pm 0.020}$	$\mathbf{0.989\pm 0.019}$	$\mathbf{0.886\pm 0.057}$	$\mathbf{0.929}$	$\mathbf{0.040}$

Table 2: Auto-verifier alignment with expert human ground truth and execution stability across three repeated runs (124 applications, 372 runs). Classification metrics report the sample mean and standard deviation across runs (

\text{Mean}\pm\text{STD}

). App Source : model used to generate the applications; Accuracy : proportion of classifications that match human ground truth; Precision : proportion of predicted passes that are human-labeled passes; Recall : proportion of human-labeled passes identified by the verifier; Fleiss’

\kappa

: agreement across repeated runs ( Fleiss, 1971 ) ; Flakiness : proportion of applications whose verifier outputs differ across runs. The overall results are shown in bold.

Model	Pass@1 (%)				Design	Efficiency
Model	$\uparrow$ Overall $\uparrow$	$\uparrow$ Repair $\uparrow$	$\uparrow$ Feature $\uparrow$	$\uparrow$ App $\uparrow$	$\uparrow$ Score $\uparrow$	T/E	Turns	Tokens (M)	$\downarrow$ Cost ( $)$ \downarrow$
claude-opus-5	50.9	85.7	29.2	47.8	3.86	0.44	0 74.1	0 6.17	5.74
deepseek-v4.1-flash	46.4	95.2	20.8	40.3	3.81	0.22	114.9	14.77	0.71
gpt-5.6-sol	44.6	95.2	29.2	34.3	4.03	0.81	0 26.4	0 1.74	2.95
gemini-3.8-flash	42.9	90.5	25.0	34.3	3.71	0.89	127.0	11.91	1.73
gemini-3.7-flash	41.1	90.5	29.2	29.9	3.89	0.96	0 69.6	0 4.14	0.84

Table 3: Evaluation results across models. Pass@1 reports task success overall and by task category. All metrics are means across applicable runs. Design Score uses a 1–5 Likert scale. T/E is the test-to-edit ratio, computed as the ratio turns spent authoring and using unit tests and runtime tests over turns spent editing

(\text{Unit}+\text{Runtime})/\text{Edit}

. Turns , Cost , and Tokens report the mean interaction turns, dollar cost, and token usage per task, respectively.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Example of the set-of-marks visual representation and scene context provided to adaptive actors.

Category	Function	Description
Viewer	navigateTo(target)	Moves the viewer toward a target while respecting the room navigation mesh.
	teleportTo(target)	Places the viewer near a target, with optional target-facing and ground-snapping behavior.
	move(motion)	Translates the viewer relative to its current view direction.
	rotate(rotation)	Rotates the viewer by relative pitch, yaw, and roll angles.
Hand	moveHand(hand, motion)	Translates one hand relative to the viewer and supports direct or ray-based dragging.
	teleportHand(hand, target)	Places a selected hand or hand anchor directly at a target.

Appendix

Table 4: Actions supported by the XR Blocks DevTools evaluation interface.

Domain	Subdomain	Examples
Healthcare	Specialized Medical Education	Trauma Triage and Emergency Care
		Surgical and Invasive Trajectory Planning
		Laparoscopic and Endoscopic Navigation
		Physical Diagnosis and Auscultation
		Pathology and Geriatric Training (Empathy-Building)
		Intensive Care Unit (ICU) Portability & Monitoring

Appendix

Table 5: XR application creation taxonomy.

Category	Heuristic	Description
Task alignment	Prompt alignment	Fidelity of the implementation to the user’s prompt specification, required interactions, and requested feature behaviors.
Visuals	Visual quality	Aesthetic execution, lighting, materials, mesh rendering, and visual polish of 3D virtual elements.
	Spatial anchoring	The virtual content remains rigidly attached to its designated physical or spatial location without drifting, sliding, jittering, or clipping during camera movements or interactions.
	Scale consistency	Virtual objects and UI elements are rendered at realistic, proportional scales relative to each other and the surrounding virtual environment.
Controls	Input responsiveness	The system accurately translates user inputs, pointer actions, and pinch or ray-casting controls into intuitive spatial manipulation without desynchronization.
Realism	Physically realistic behavior	The virtual entities, physics rules, collision bounds, and interactive state transitions behave logically and consistently within the context of the physical environment.

Appendix

Table 6: XR usability heuristics.

Figure 6: An example of an SDK feature task. Task instructions have been lightly edited for visualization purposes.

Figure 7: An example of an SDK repair task. Task instructions have been lightly edited for visualization purposes.

Figure 8: An example of an application-generation task. Task instructions have been lightly edited for visualization purposes.

Heuristic	Human–Human			VLM–Human
	$\uparrow$ $A_{\text{exact}}$ (%) $\uparrow$	$\uparrow$ $A_{\pm 1}$ (%) $\uparrow$	$\uparrow$ $\text{r}_{3}$ $\uparrow$	$\uparrow$ $A_{\text{exact}}$ (%) $\uparrow$	$\uparrow$ $A_{\pm 1}$ (%) $\uparrow$	$\uparrow$ $\mathrm{MAE}$ $\downarrow$
Prompt Alignment	27.90%	63.80%	0.64	32.30%	66.90%	1.20
Visual Quality	26.30%	70.20%	0.64	42.90%	82.00%	0.83
Spatial Anchoring	27.90%	57.40%	0.50	32.30%	66.20%	1.22
Scale Consistency	29.50%	75.30%	0.58	39.10%	81.20%	0.92
Input Responsiveness	32.00%	62.60%	0.66	32.30%	60.90%	1.28

Appendix

Table 7: Comparison of agreement between human raters and VLM judge alignment across XR design heuristics (

N=124

applications). Ratings use a 1–5 Likert scale; agreement values are percentages.

A_{\text{exact}}

: exact score agreement;

A_{\pm 1}

: agreement within one point;

\text{r}_{3}

: reliability of the three-rater mean score (

\text{ICC}(2,3)

); MAE : mean absolute error between model ratings and the human consensus.

Model	$\uparrow$ Input $\uparrow$	$\uparrow$ Depth $\uparrow$	$\uparrow$ Media $\uparrow$	$\uparrow$ Percept $\uparrow$	$\uparrow$ UI $\uparrow$	$\uparrow$ Mean $\uparrow$
Model	(10)	(8)	(7)	(12)	(8)	(45)
claude-opus-5 [medium]	70.0	62.5	42.9	58.3	37.5	55.6
deepseek-v4.1-flash [xhigh]	70.0	50.0	57.1	50.0	50.0	55.6
gpt-5.6-sol [xhigh]	80.0	62.5	42.9	58.3	50.0	60.0
gemini-3.8-flash [high]	70.0	50.0	71.4	41.7	50.0	55.6
gemini-3.7-flash [high]	60.0	50.0	57.1	66.7	50.0	57.8

Appendix

Table 8: Evaluation results by SDK technical subsystem across all 45 tasks. Pass@1 values are percentages; task counts appear in parentheses. Input : input and gestures (10 tasks); Depth : depth and rendering pipeline (8 tasks); Media : camera and audio streams (7 tasks); Percept : spatial perception, meshing, and reference tracking (12 tasks); UI : UI overlays, sliders, reticles, and 3D viewer widgets (8 tasks); Mean : overall SDK mean (45 tasks). The highest value in each column is shown in bold.

Model	$\uparrow$ DL $\uparrow$	$\uparrow$ Des $\uparrow$	$\uparrow$ Edu $\uparrow$	$\uparrow$ Ent $\uparrow$	$\uparrow$ Health $\uparrow$	$\uparrow$ Mean $\uparrow$
Model	(6)	(11)	(20)	(21)	(9)	(67)
claude-opus-5 [medium]	50.0	45.5	45.0	42.9	66.7	47.8
deepseek-v4.1-flash [xhigh]	50.0	45.5	30.0	42.9	44.4	40.3
gpt-5.6-sol [xhigh]	16.7	36.4	35.0	38.1	33.3	34.3
gemini-3.8-flash [high]	33.3	27.3	30.0	42.9	33.3	34.3
deepseek-v4-pro-0813 [xhigh]	16.7	45.5	30.0	33.3	44.4	34.3

Appendix

Table 9: Evaluation results by application domain. Pass@1 values are percentages; task counts appear in parentheses. DL : daily living; Des : design; Edu : education; Ent : entertainment; Health : healthcare; Mean : overall application mean. The highest value in each column is shown in bold.

Model	$\uparrow$ Align $\uparrow$	$\uparrow$ Visual $\uparrow$	$\uparrow$ Anchor $\uparrow$	$\uparrow$ Scale $\uparrow$	$\uparrow$ Input $\uparrow$	$\uparrow$ Physics $\uparrow$
claude-opus-5 [medium]	3.42	3.84	4.29	4.16	3.56	3.44
deepseek-v4.1-flash [xhigh]	3.50	3.86	4.14	4.11	3.53	3.39
gpt-5.6-sol [xhigh]	3.53	4.19	4.44	4.41	3.59	3.59
gemini-3.8-flash [high]	3.24	3.73	4.27	4.08	3.35	3.18
gemini-3.7-flash [high]	3.48	3.87	4.37	4.23	3.60	3.44
deepseek-v4-pro-0813 [xhigh]	3.00	3.48	3.87	3.96	3.13	3.09

Appendix

Table 10: Evaluation results across the 10 XR design heuristics. Scores are reported on a 1–5 Likert scale. Align : prompt alignment; Visual : visual quality; Anchor : spatial anchoring; Scale : scale consistency; Input : input responsiveness; Physics : physical realism; Feed. : interaction feedback; Error : spatial error prevention; Context : contextual placement; Clutter : clutter control; Mean : overall mean design score. The highest value in each column is shown in bold.

Model	Calls	Actions			Testing
Model	Calls	Expl.	Edit	E/E	Unit	Runt.	T/E
claude-opus-5 [medium]	0 58.9	0 55.2	23.1	2.39	0 7.30	2.83	0.44
deepseek-v4.1-flash [xhigh]	109.3	101.3	47.6	2.13	0 9.70	0.68	0.22
gpt-5.6-sol [xhigh]	0 58.9	0 43.6	14.2	3.07	10.75	0.75	0.81
gemini-3.8-flash [high]	107.7	0 68.0	12.7	5.36	11.22	0.12	0.89
gemini-3.7-flash [high]	0 68.6	0 45.6	0 8.9	5.11	0 8.58	0.00	0.96

Appendix

Table 11: Development tool usage across models. Counts are means per task. Calls : total tool calls; Expl. : code reads and searches; Edit : code edits; E/E : exploration-to-edit ratio (

\text{Expl.}/\text{Edit}

); Unit : unit test executions using Vitest or npm test ; Runt. : runtime test executions using the DevTools simulator or Chromium; T/E : test-to-edit ratio (

(\text{Unit}+\text{Runt.})/\text{Edit}

Failure Type	$\uparrow$ Failures (%) $\downarrow$
Spawned virtual objects that clipped into real walls, floors, or furniture	19.8
Lost room-mesh or plane updates as the user walked through the space	10.7
Assumed that the room was prescanned instead of waiting for the user to scan it	0 9.3
Failed to align an object flat against a real-world floor or table	0 4.6
Ignored vertical walls and ceilings by assuming that only flat floors exist	0 4.6
Floated virtual objects in front of real walls because occlusion masks were missing	0 4.4

Appendix

Table 12: Common environmental and perception failure patterns.

Failure Type	$\uparrow$ Failures (%) $\downarrow$
Confused left- and right-hand roles	34.5
Treated physical touch as a flat-screen click without push or depth	20.3
Attached UI directly to the face or head, causing severe motion sickness	16.4
Placed items outside a comfortable reach or field of view	12.8
Made laser pointer targets too small or obstructed, causing ray-casting misses	11.5
Displayed flat text labels or UI backward or edge-on, making them invisible	0 7.7

Appendix

Table 13: Common human body, gesture, and tracking failure patterns.

Failure Type	$\uparrow$ Failures (%) $\downarrow$
Lost object positions inside moving parent groups because of world-space confusion	64.4
Started the app before 3D models or the headset finished loading because of a race condition	44.9
Used the wrong real-world scale because of a centimeter-to-meter conversion error	30.6
Implemented broken reset or undo operations that left the scene stuck after the first action	26.6
Failed to snap objects because of mismatched local anchor coordinates	0 8.3
Caused physics instability or allowed items to fall through the floor	0 5.7

Appendix

Table 14: Common spatial architecture and runtime lifecycle failure patterns.

Figure 9: Examples of successfully generated XR apps spanning healthcare, education, design, daily living, and entertainment. The apps integrate diverse XR capabilities, including multimodal interaction, spatial user interfaces, hand and head tracking, physical-environment sensing, spatial anchoring, object manipulation and placement, and spatial audio. Representative examples include virtual painting, cell and chemical visualization, 3D volume measurement, dart throwing and fruit slicing, and growing a virtual garden with the detected physical environment.

Figure 10: Examples of generated apps for the connect-the-dots task. The task requires a connect-the-dots game in which dots appear around the room and the user must select and drag them to complete a shape. Only claude-opus-5 completed the task where gemini-3.8-flash failed to effectively place the dots on the mesh while gpt-5.6-sol and deepseek-v4.1-flash all failed to make the dots interactable. All models placed the dots in a small controlled area.

Figure 11: Examples of generated apps for the lascaux-torch task. The task requires an educational app in which users can explore cave paintings in their room. The gpt-5.6-sol app failed to start due to hallucinating a reticle configuration that prevented it from loading. The other apps were functional but created virtual walls instead of rendering cave paintings on the physical environment.

Figure 12: Examples of generated apps for the sight-magnifier task. The task requires a daily-living app in which users can form a box with their fingers and see a magnified view of its contents. Models struggled with hand-gesture recognition: they either failed to detect the gesture or aligned the magnified area poorly with the user’s view plane; only claude-opus-5 and deepseek-v4.1-flash effectively identified the hand framing gesture but only claude-opus-5 properly configured the magnification panel.

Figure 13: Examples of generated apps for the walking-clearance-visualization task. The task requires a simple app that visualizes walking clearance to help building owners identify accessibility challenges. Blue and orange boxes are simulated physical objects where we should expect they change the walking clearance visualization. Models struggled with depth tracking and often failed to calculate or render clearance accurately at runtime like the failures seen in gpt-5.6-sol and gemini-3.8-flash while claude-opus-5 and deepseek-v4.1-flash successfully implemented the visualization.

Figure 14: Examples of generated apps for the palm-controller task. The task requires a simple app where when the user flips their hand over, two buttons spawn that, when clicked, create different shapes in the palm. While all models shown were successful, the buttons were often placed deep into the hand due to using the hand root as a coordinate root which affected the overall design of the app.

Code Generation Benchmarks

Code generation benchmarks measure how well AI models produce code across programming tasks, languages, and real-world settings.

See all 152 papers

Jul 3, 2026

GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.

Brian La, Sejoon Chang, Ben Kim +5

Nitrode · Nexon Intelligence Labs · Dartmouth College +3

Coding Agents Software Engineering Benchmarks Code Generation

Ask PDF

Sep 28, 2026

RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

Coding agents are beginning to move beyond purely digital tasks to tackle physical-world challenges, particularly in robotics. Existing robotics benchmarks, however, primarily focus on the performance of individual artifacts, such as policies or controllers, offering limited coverage of coding agents' broader engineering capabilities. Real-world robotics extends beyond control: agents must build, integrate, diagnose, and improve heterogeneous artifacts under resource constraints and reason from multimodal feedback. To evaluate these broader capabilities, we introduce RLE-Bench, a benchmark of robot-learning tasks spanning four representative robotics development workflows: interactive control, policy learning, perception and estimation, and mechanical design. We use diverse task-specific metrics to evaluate the artifacts submitted by the coding agents, from the success rate the agents achieved to the policy agents trained, the harness agent built, and the mechanical structures the agent designed. We aggregate these metrics into an overall RLE Index and report workflow-specific capability profiles, enabling systematic comparison of coding agents' capabilities across multiple capability dimensions. Beyond performance ranks, we also conduct in-depth case studies examining agent behavior on representative tasks, highlighting both current capabilities and limitations, and pointing to the opportunities robotics tasks have to offer for future agent training.

Haitong Ma, Chenxiao Gao, Rushi Qiang +2

Harvard University · Georgia Institute of Technology

AI Agent Benchmarks Robot Learning

Ask PDF

Jun 16, 2026

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

Game generation is an emerging application of coding agents, requiring models to transform natural-language specifications into playable interactive systems. Unlike traditional coding tasks, game generation takes place within a game engine, where scripts, scenes, assets, rendering, and runtime interactions must jointly produce coherent gameplay. We formalize end-to-end game generation as the problem of producing a complete game artifact that realizes a specification through observable player-game interaction in a target environment. We argue that evaluating this setting requires three desiderata: Engine Grounding, Artifact Completeness, and Interactive Verification. We propose an interaction-grounded evaluation framework that assesses executable gameplay through replayed demonstrations and rubric-guided multimodal judging. We instantiate this framework as GameCraft-Bench, a benchmark comprising 140 Godot tasks across 15 game families. Evaluations of frontier coding agents show that end-to-end game generation remains highly challenging: the strongest agent achieves only 41.46%, and most agents score below 40%. Further analysis reveals that while agents often implement recognizable mechanics, they struggle to deliver complete games with sufficient content, functional visual feedback, and coherent presentation. See https://tongxuluo.github.io/gamecraft-bench-website for demos, code, and data.

Tongxu Luo, Rongsheng Wang, Jiaxi Bi +22

1The Chinese University of Hong Kong, Shenzhen · 2Shenzhen Loop Area Institute · 4USTB +5

AI Agent Benchmarks Agentic Code Generation Code Generation Evaluation

Ask PDF

Abstract

Code Generation Benchmarks

Explore similar work

GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments

RLE-Bench: A Qualifying Exam for Coding Agents as Robot Learning Engineers

GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?