ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality

Dawar Khan; Alexandre Kouyoumdjian; Xinyu Liu; Omar Mena; Dominik Engel; Ivan Viola

cs.CVApr 6, 2026

ClickAIXR: On-Device Multimodal Vision-Language Interaction with Real-World Objects in Extended Reality

Authors: Dawar Khan, Alexandre Kouyoumdjian, Xinyu Liu, Omar Mena, Dominik Engel, Ivan Viola

Organizations: King Abdullah University of Science and Technology (KAUST), Saudi Arabia.

Abstract

We present ClickAIXR, a novel on-device framework for multimodal vision-language interaction with objects in extended reality (XR). Unlike prior systems that rely on cloud-based AI (e.g., ChatGPT) or gaze-based selection (e.g., GazePointAR), ClickAIXR integrates an on-device vision-language model (VLM) with a controller-based object selection paradigm, enabling users to precisely click on real-world objects in XR. Once selected, the object image is processed locally by the VLM to answer natural language questions through both text and speech. This object-centered interaction reduces ambiguity inherent in gaze- or voice-only interfaces and improves transparency by performing all inference on-device, addressing concerns around privacy and latency. We implemented ClickAIXR in the Magic Leap SDK (C API) with ONNX-based local VLM inference. We conducted a user study comparing ClickAIXR with Gemini 2.5 Flash and ChatGPT 5, evaluating usability, trust, and user satisfaction. Results show that latency is moderate and user experience is acceptable. Our findings demonstrate the potential of click-based object selection combined with on-device AI to advance trustworthy, privacy-preserving XR interactions. The source code and supplementary materials are available at: nanovis.org/ClickAIXR.html

Figures & tables

Figure 2: Overview of the ClickAIXR pipeline. Users choose between (i) dwell mode , where a fixed-size GCW follows gaze and, after a brief dwell, auto-captures an ROI for image captioning or a spoken/text query; or (ii) GCW select-and-ask , where the user places the border-only GCW on the target, adjusts width/height/depth with the controller, and confirms with a trigger. After confirmation, a microphone icon appears; the spoken question is converted to text via on-device ASR, then fused with the cropped image and processed by the on-device VLM (encoder–decoder, tokenizer). The answer is returned to the XR UI as text and to the user as audio via TTS on ML2.

Figure 3: Examples of images used for the latency test: top row from COCO [ LinCoco2014 ] , bottom row from the Book Covers dataset [ iwana2016judging ] .

#	SUS item	Scale: 1 = Strongly Disagree (SD) — 5 = Strongly Agree (SA)
		SD SA 1 2 3 4 5
1	I think that I would like to use this system frequently.	SD SA
2	I found the system unnecessarily complex.	SD SA
3	I thought the system was easy to use.	SD SA
4	I think that I would need the support of a technical person to be able to use this system.	SD SA
5	I found the various functions in this system were well integrated.	SD SA

Table I: SUS [ brooke1996sus ] questionnaire items with 5-point Likert responses (1=Strongly Disagree, 5=Strongly Agree).

Figure 4: User study overview and in-headset views on Magic Leap 2. Top: participants interacting with the object table using ClickAIXR . Bottom: representative object layouts and direct in-headset views; the border-only rectangle is the gaze-locked clipping window (GCW), which participants are positioning and resizing before capture.

Metric	Mean	Std	Median	Min	Max
Inference Time (Books) [s]	5.36	0.80	5.03	4.36	7.20
Inference Time (COCO) [s]	5.48	1.02	5.29	4.37	8.96
TG Speed (Books) [tokens/s]	3.36	0.06	3.38	3.21	3.45
TG Speed (COCO) [tokens/s]	3.36	0.04	3.37	3.22	3.44

Table II : On-device VLM performance on the Book Covers dataset [ iwana2016judging ] and the COCO dataset [ LinCoco2014 ] . Inference times are reported in seconds; TG speed is tokens per second.

Figure 5: Some of the objects we placed in the room used for the study. Their close proximity to one another often leads to overlap once photographed from a particular angle, which requires disambiguation.

Method	$n$	Mean	SD	Median	Q25	Q75	SEM	$\pm$ 95% CI
Gemini	12	81.88	11.24	83.75	73.75	90.63	3.24	6.36
ChatGPT	12	76.67	15.79	76.25	69.38	90.00	4.56	8.93
ClickAIXR	12	60.00	17.06	61.25	50.00	70.63	4.92	9.65

Table III : Summary of SUS results (0–100). Mean

\pm

SD, median, quartiles, SEM, and

\pm

95% CI for each method (

n=12

Figure 6: Mean SUS scores (0–100) for ChatGPT, Gemini, and ClickAIXR. Bars show mean values; error bars indicate $\pm$ 95% confidence intervals (CI). Results: Gemini = $81.9\pm 11.2$ (SD), CI $\pm 6.36$ ; ChatGPT = $76.7\pm 15.8$ (SD), CI $\pm 8.93$ ; ClickAIXR = $60.0\pm 17.1$ (SD), CI $\pm 9.65$ .

Figure 7: Mean self-assessment ratings (1–5 Likert scale) for reliability (left) and error rate (right) across ChatGPT, Gemini, and ClickAIXR. Bars show mean responses; error bars indicate $\pm$ 95% confidence intervals (CI). Participants rated ChatGPT and Gemini higher on perceived reliability, while ClickAIXR received lower reliability scores and higher perceived error rates.

Figure 8: Mean ranks by method (1 = best), with error bars showing $\pm$ 95% confidence intervals (CI). Gemini achieved the best average rank, followed by ChatGPT, while ClickAIXR was ranked lowest on average.

Human-Object Interaction

Human-object interaction studies how people perceive, manipulate, and coordinate with objects in images, videos, and embodied settings.

See all 112 papers

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence
Jun 10, 2026Dingyu Yao, Junhao Zhou, Chenxu Yang +17Human-AI Interaction Vision-Language Agents
Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation
Apr 30, 2026Yajvan Ravan, Adam Rashid, Alan Yu +8Robotic Data Collection Robotics Simulation
MIT CSAIL · 2FortyFive Labs · 3Caltech +3
SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions
Mar 19, 2026Vasco Xu, Brian Chen, Eric J. Gonzalez +4Gesture Recognition Multimodal Sensor Fusion
Vasco Xu and Henry Hoffmann are with the University of Chicago. · Brian Chen is with Northwestern University. · The remaining co-authors are with Google. +1

Jun 10, 2026J/K move · Enter open · S save

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Dingyu Yao, Junhao Zhou, Chenxu Yang, Chuanyu Qin+16

Human-AI Interaction Vision-Language Agents

Open paper AskPDF arXiv

Many moments in the real world do not wait for a user to ask. A fire starts on a security monitor, an expression flickers across a video call, or a product a viewer wants flashes by in a livestream. Yet today's large models remain mostly turn-based by design: they answer only when addressed, and even video-call apps that appear interactive still operate as question-answer systems, reacting only when polled or prompted. We argue for a different paradigm: a model that is present in the world like a person. It continuously watches what is happening now, decides on its own whether to speak or stay silent, interacts in real time, and delegates to a background model when the problem is hard. To advance interaction models and their adoption across domains, we make two fully open-sourced contributions. First, we release JoyAI-VL-Interaction, an 8B-scale, vision-first VL-interaction model. The model makes the response decision internally, choosing each second to stay silent, respond, or delegate to a background model, and it excels at vision-triggered responsiveness and time awareness. We pair it with a transferable training recipe, from which capabilities we never trained for emerge, such as guiding a shopper through changing app screens or improvising a lecture from a slide deck. Second, we release a complete, deployable system built around that model. The system streams any ongoing video into the model, making it genuinely present in the world. All other components are pluggable, including ASR/TTS modules, memory, visualization UI, and a background brain that can connect to any API or agent. Across six real-world scenarios, human raters prefer JoyAI-VL-Interaction over the in-app video-call assistants of Doubao and Gemini by a wide margin. To our knowledge, this is the first open, vision-driven interaction model released together with its training recipe, data, and complete deployable system.

Abstract

Human-Object Interaction

Explore similar work

JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

Lucid-XR: An Extended-Reality Data Engine for Robotic Manipulation

SurfaceXR: Fusing Smartwatch IMUs and Egocentric Hand Pose for Seamless Surface Interactions