As autonomous vehicles and Extended Reality (XR) headsets enable novel in-car interactions, seamlessly querying physical landmarks, known as Outside-the-Vehicle Referencing (OVR), remains challenging due to ego-motion and referential ambiguity. We present a robust, multimodal OVR framework fusing user gaze and natural language to identify Points of Interest (POIs). To address the scarcity of dynamic vehicular data, we developed a VR-based pipeline synchronizing 360-degree transit videos with vehicle GNSS telemetry. Through a user study (N=46) mapping passenger head orientation into a 3D geospatial Digital Twin, we captured authentic gaze-speech behaviors. We subsequently trained a lightweight Transformer network, leveraging LLMs to dynamically align continuous spatial gaze vectors with discrete verbal context. Experimental results demonstrate high accuracy and low computational overhead, achieving an 83.33% Top-1 accuracy (87.72% Top-2) and an average inference time of 24.3 milliseconds. This real-time paradigm effectively resolves referential ambiguity, enabling context-aware spatial retrieval for passengers within the vehicle.
Figures & tables
Figure 1. End-to-end overview of our proposed Outside-the-Vehicle Referencing (OVR) framework. Top: The spatial mapping pipeline. In-vehicle passenger head kinematics (left) are projected into a synchronized 3D Digital Twin (middle) to identify candidate buildings via continuous ray-casting (right). Bottom: The multimodal inference architecture, which fuses spatial features from the intersected buildings with discrete semantic speech features via a Transformer network to predict the final target Point of Interest (POI).
Figure 2. Data acquisition hardware setup inside the physical vehicle.
Figure 3. Spatial correspondence between the visual stimulus and our Digital Twin approach. (a) Participant’s Field of View (FoV) during the 360-degree VR transit video. (b) A 2D viewport rendering of the underlying 3D reconstructed scene. Logged vehicle GNSS dictates the Unity virtual camera’s global position within Cesium, while logged HMD data (representing the participant’s head pose) drives its orientation. This precise alignment replicates the user’s view within the 3D space for accurate ray-casting and semantic building extraction.
Figure 4. Overview of the 3D spatial ray-casting strategy within the Digital Twin. The user’s dynamic field of view is sampled using (a) a uniform grid of rays to (b) continuously detect intersections with candidate 3D OpenStreetMap building colliders.
Figure 5. Violin plots of subjective evaluation scores for (a) immersion and (b) cognitive workload.
Figure 6. End-to-End Multimodal Inference Pipeline: From heterogeneous spatial and semantic inputs to final POI aggregation.
Figure 7. Evolution of Rank-1 Accuracy and F1-Score on the validation set. The proposed model achieves asymptotic stability, demonstrating robust generalization and an optimal precision-recall balance.
Model Configuration
Rank-1 Acc (%)
F1-Score
Our Full Architecture (Trainable IDs, GELU, Sum Agg.)
83.33
0.813
Unimodal Comparison
Vision-Only (Spatial & Geometric Features)
76.32
0.780
Hyperparameter Ablations
Fixed Random ID Embeddings
77.56
0.795
Leaky ReLU Activation (replacing GELU)
78.25
0.762
Table 1. Ablation study comparing our proposed architecture against unimodal, hyperparameter and structural variants. The proposed model consistently outperforms modified configurations.
Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) · MMLab, CUHK · School of Transportation Science and Engineering, Harbin Institute of Technology +7
National Key Laboratory of Science and Technology on Multi-spectral Information Processing, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology, Wuhan 430074, China