AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Models

Kuan-Lin Chen; Tzu-Ti Wei; Chao-Chi Liao; Yu-Chee Tseng; Jen-Jee Chen

cs.CVAug 16, 2026

AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Models

Authors: Kuan-Lin Chen, Tzu-Ti Wei, Chao-Chi Liao, Yu-Chee Tseng, Jen-Jee Chen

Organizations: Institute of Computer Science and Engineering & College of Artificial Intelligence National Yang Ming Chiao Tung University Hsinchu, Taiwan

Abstract

This study investigates the challenge of ambiguity faced by Vision-Language Models (VLMs) in understanding spatial semantics. Spatial cognition, shaped by cognitive psychology, spatial science, and cultural context, often assigns directionality to objects. However, natural language descriptions of spatial relations frequently omit explicit reference frames, leading to semantic ambiguity and potentially serious errors for embodied AI robots. Existing VLMs, due to insufficient training on reference frames and object orientations, often produce inconsistent responses. To address this issue, we construct a new dataset, AlloEgo-View, comprising (image, query, view-specific answer) triplets that capture key object relations from both allocentric and egocentric perspectives. The view-specific descriptions follow a structured spatial representation that annotate detailed scene descriptions, reference and target objects, their orientations, reference frames, and view types. Building on AlloEgo-View, we develop AlloEgo-VLM, a framework to disambiguate allocentric and egocentric reference frames, even under ambiguous queries, and to be easily integrated into existing VLMs via supervised fine-tuning. Furthermore, we deploy our framework onto an embodied robotic platform within NVIDIA Isaac Sim to validate its real-world feasibility in open-ended object searching tasks. Experiments highlight the limitations of current VLMs in handling view-specific queries and demonstrate the strong disambiguation ability of AlloEgo-VLM.

Figures & tables

Figure 1: Under an ambiguous instruction, general VLMs suffer from high view ambiguity rate (VAR). AlloEgo-VLM resolves this by decoupling spatial semantics into explicit egocentric and allocentric perspectives, ensuring precise view-specific accuracy (VSA).

Dataset	Task Description	View Type	MCAF	RFF	Scale
CLEVR [ Johnson et al.(2017)Johnson, Hariharan, van der Maaten, Fei-Fei, Zitnick, and Girshick ]	Visual reasoning (synthetic QA)	Ego	$\triangle$	$\checkmark$	$\sim$ 700K
GQA [ Hudson and Manning(2019) ]	VQA + scene graph reasoning	Ego	$\triangle$	$\checkmark$	$\sim$ 22M
SpatialSense [ Yang et al.(2019)Yang, Russakovsky, and Deng ]	Pairwise spatial relation classification	Ego	$\checkmark$	$\checkmark$	$\sim$ 11K
SPAR [ Zhang et al.(2025)Zhang, Chen, Zhou, Xu, Huang, Mei, Chen, Yuan, Cai, Huang, Quan, Xu, and Zhang ]	Spatial perception and reasoning	Ego	$\triangle$	$\checkmark$	$\sim$ 7M
Thinking in Space [ Yang et al.(2024)Yang, Yang, Gupta, Fei-Fei, and Xie ]	Video reasoning and memory	Allo, Ego	$\times$	$\times$	$\sim$ 5K
Perspective Aware [ Lee et al.(2025)Lee, Je, Park, Uy, Guibas, and Sung ]	Abstract perspective change	Allo, Ego	$\checkmark$	$\times$	-

Table 1: Comparison of spatial understanding datasets. MCAF denotes “multi-choice answer-free queries” (e.g., [ Yang et al.(2024)Yang, Yang, Gupta, Fei-Fei, and Xie ] requires verbose queries with predefined options). RFF denotes “reference-frame-free queries” (e.g., [ Li et al.(2025)Li, Li, Wang, Yan, Zhang, Chen, Hou, Jiang, Zhang, Shen, Lu, and Zhuang ] requires explicit reference-frame specification). AlloEgo-View supports natural queries such as “Where is the refrigerator?” (

\checkmark

= yes,

\times

= no,

\triangle

= partial).

Figure 2: AlloEgo-View data curation pipeline. Each triplet consists of an image, a natural but potentially ambiguous query, and an answer that provides detailed egocentric and allocentric view descriptions.

Panel of Figure 3 — Figure 3: Structured spatial answer generation: (a) overall workflow and (b) standardized answer format.

Figure 5: Multi-stage training pipeline of AlloEgo-Adapter and progressive expansion of AlloEgo-View.

Figure 6: Spatial Query Classifier for filtering and categorizing spatial-related user queries.

Method	Params	VAR $\downarrow$	VSA $\uparrow$
Qwen 2.5-VL	7B	94.568%	50%
Llama 3.2-V	11B	100%	–
Gemma 3-V	4B	99.753%	100%
GPT-4o	–	99.259%	66.666%
GPT-4o mini	–	99.012%	75%

Table 2: Perspective tests of strong VLMs.

Method	Params	AD_RO↑	AD_TO↑	Ego↑	Allo↑
VS-Format (Qwen 2.5-VL)	7B	4.08	3.62	4.51	2.65
VS-Textbook (Qwen 2.5-VL)	7B	5.22	4.61	5.39	4.16
AlloEgo-VLM (Qwen 2.5-VL)	7B	7.94	8.04	8.19	6.25
VS-Format (Llama 3.2-V)	11B	3.82	3.53	4.36	3.74
VS-Textbook (Llama 3.2-V)	11B	4.95	4.56	5.03	3.38
AlloEgo-VLM (Llama 3.2-V)	11B	7.92	8.17	8.28	6.72

Table 3: Comparison of disambiguation capability.

Method	Para.	Size	Inf.	Acc.
Spatial Classifier	6M	0.25GB	$\approx$ 2ms	100%
Qwen 2.5-VL	7B	7GB	$\approx$ 260ms	99.53%
Llama 3.2-V	11B	11GB	$\approx$ 700ms	99.71%
Gemma 3-V	4B	4GB	$\approx$ 240ms	91.24%

Table 4: Capability for recognizing spatial queries.

Panel of Figure 7 — Figure 7: 3D PCA projection of features encoded by the Spatial Query Classifier.

Figure 8: Progressive training of AlloEgo-Adapter on Qwen 2.5-VL-7B . Line: performance; bars: dataset size.

	AQ (implicit allocentric SR)			EQ (explicit egocentric SR)
Scenario ¡reference obj¿	Qwen-For	Qwen-Txt	AlloEgo	Qwen-For	Qwen-Txt	AlloEgo
Case 1: Table (Non-directional)	1.0	1.0	1.0	1.0	0.9	1.0
Case 2: Sofa (Facing Observer)	0.0	0.1	1.0	0.9	1.0	1.0
Case 3: Chair (Facing Observer)	0.2	0.2	1.0	0.8	0.9	0.9
Case 4: Chair (Facing Left)	1.0	0.7	0.8	0.4	0.6	0.8
Case 5: Human (Facing Observer)	0.2	0.1	0.4	0.3	0.5	0.5

Table 5: Navigation Success Rate (SR) for embodied search in Isaac Sim experiments. AQ (Ambiguous Queries) evaluate implicit allocentric transformations with unstated reference frames, while EQ (Explicit Queries) serve as an egocentric control benchmark for standard camera-space grounding using the phrase “from your perspective” .

Figure 9: Example multi-object scenarios in Isaac Sim for embodied object search tasks. Boxes are positioned in front of and behind the chair (left), while carts are placed on both sides of the human avatar (right).

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Grid set $grids$	Description $d$
$[2]$ / $[4]$ / $[5]$ / $[6]$ / $[8]$	Top / Left / Center / Right / Bottom
$[1]$ / $[3]$	Top-left / Top-right
$[7]$ / $[9]$	Bottom-left / Bottom-right
$[1,2]$ , $[1,4]$ / $[2,3]$ , $[3,6]$	Top-left / Top-right
$[4,7]$ , $[7,8]$ / $[6,9]$ , $[8,9]$	Bottom-left / Bottom-right
$[2,5]$ / $[5,8]$	Upper-center / Lower-center

Appendix

Table 6: Mapping rules between grid sets and spatial descriptions

\bar{D}_{r}\leftarrow\frac{1}{n}\sum_{i=1}^{n}d_{ri}

Appendix

Algorithm 2 Depth-Based Relative Position Description

Panel of Figure 10 — Figure 10: Examples of emergent reasoning in AlloEgo-VLM (with Qwen2.5-VL as the backbone). Behaviors after and before training stage 3 are shown.

Visual Spatial Reasoning

Visual spatial reasoning studies how AI systems infer geometry, object relations, viewpoints, and physical change from images and video.

See all 251 papers

LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning
Sep 28, 2026Bang Xiao, Wenqi Jia, Ozgur Kara +5Visual Spatial Reasoning Visual Reasoning
University of Illinois Urbana-Champaign · Zhiyuan College, Shanghai Jiao Tong University · University of California, Merced +2
Learning Multi-View Spatial Reasoning from Cross-View Relations
Mar 30, 2026Suchae Jeong, Jaehwi Song, Haeone Lee +9Multimodal Spatial Reasoning Visual Spatial Reasoning
KAIST · Config · Hanyang University +2
Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models
Sep 7, 2026Mimo Shirasaka, Haochen Zhang, Yonatan BiskVision-Language Grounding Visually Grounded Reasoning
The University of Tokyo Tokyo, Japan · Carnegie Mellon University Pittsburgh, PA, USA

Sep 28, 2026J/K move · Enter open · S save

LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

Bang Xiao, Wenqi Jia, Ozgur Kara, Tiancheng Shen+4

University of Illinois Urbana-Champaign · Zhiyuan College, Shanghai Jiao Tong University · University of California, Merced+2

Visual Spatial Reasoning Visual Reasoning VLM Reasoning Spatial Reasoning

Open paper AskPDF arXiv

Perspective taking is a fundamental component of spatial intelligence, requiring models interpret spatial relations from a specified viewpoint, such as that of another entity or an imagined observer. Despite the increasing spatial reasoning capabilities of Vision-Language Models (VLMs), they still struggle with perspective taking, often defaulting to the camera viewpoint when a query requires reasoning from a different perspective. We introduce Learning Reference Coordinate Frames for Perspective Taking (LeRF), a framework that trains VLMs to construct and use explicit reference frames for viewpoint-dependent reasoning. Given an image and a query, LeRF decides whether a coordinate frame is necessary. If so, it grounds the reference entity and predicts the frame's origin and entity-centered reference frame. A lightweight renderer overlays the frame onto the image, enabling subsequent reasoning over these visual cues without external perception models or explicit 3D reconstruction. To learn this process, we first perform supervised fine-tuning to teach selective tool invocation and reference coordinate frame prediction, followed by reinforcement learning on spatial VQA pairs to improve frame-guided reasoning. Across diverse perspective-taking benchmarks, LeRF consistently improves over its backbone and achieves strong performance against existing open-source methods. Further evaluations also show improved reference-frame grounding and orientation estimation, supporting the effectiveness of learned reference frames for viewpoint-dependent reasoning.

Abstract

Visual Spatial Reasoning

Explore similar work

LeRF: Learning Reference Coordinate Frames for Perspective Taking Reasoning

Learning Multi-View Spatial Reasoning from Cross-View Relations

Contextual Observer Grounding: Evaluating Situated Spatial Reasoning in Vision-Language Models