Visuospatial Perspective Taking in Multimodal Language Models

Jonathan Prunty; Seraphina Zhang; Patrick Quinn; Jianxun Lian; Xing Xie; Lucy Cheke

cs.CLMar 4, 2026

Visuospatial Perspective Taking in Multimodal Language Models

Authors: Jonathan Prunty, Seraphina Zhang, Patrick Quinn, Jianxun Lian, Xing Xie, Lucy Cheke

Organizations: Leverhulme Centre for the Future of Intelligence, University of Cambridge · Department of Psychology, University of Cambridge · Microsoft Research Asia

Abstract

As multimodal language models (MLMs) are increasingly used in social and collaborative settings, it is crucial to evaluate their perspective-taking abilities. Existing benchmarks largely rely on text-based vignettes or static scene understanding, leaving visuospatial perspective-taking (VPT) underexplored. We adapt two evaluation tasks from human studies: the Director Task, assessing VPT in a referential communication paradigm, and the Rotating Figure Task, probing perspective-taking across angular disparities. Across tasks, MLMs show pronounced deficits in Level 2 VPT, which requires inhibiting one's own perspective to adopt another's. These results expose critical limitations in current MLMs' ability to represent and reason about alternative perspectives, with implications for their use in collaborative contexts.

Figures & tables

Figure 1: Example stimuli from the Rotating Figure Task (visual field cone shown in yellow for illustration). Left: the figure can see the symbol in front of them (Test 1). Centre: the symbol p appears as a d from the figure’s perspective, and is located on their right (Test 2). Right: If asked about the symbol on the figure’s right appears to them, the correct response would be 6 , requiring both visual and spatial perspective taking (Test 3). A line-of-sight arrow was added to scaffold model’s ability to perceive the viewing direction of figures from a top-down perspective.

Figure 2: Mean MLM accuracy in Test conditions of the Figure Rotation Task , binned by figure rotation angle (12 bins) and aggregated across visual and spatial questions in Tests 1–2. 0 ° indicates a shared perspective and 180 ° an opposite perspective; 90 ° and − 90 ° correspond to right- and left-facing figures. The dashed line indicates chance.

Condition	Sub-type	Description
None	–	Only one object matches the director’s description (e.g., “red book”)
Size	–	Multiple objects match; target specified by size (e.g., “the largest red book”)
Spatial	Shared	Multiple matching objects; target specified by vertical term (“topmost” or “bottommost”)
Spatial	Different	Multiple matching objects; target specified by horizontal term (“leftmost” or “rightmost”)

Table 2: Relative adjective conditions in the Director Task

Figure 3: Example stimulus from the Director Task . If the director says “Please select the rightmost blue, non-striped item of clothing from my point of view”, he would be referring to the item in C3 . The answer is not D1 or D4 , as the director specified the rightmost item from his perspective. The answer is also not A1 as the item is occluded from the director’s view. All other items do not fit the object specification.

Figure 4: MLM mean accuracy in the Director Task on visual (occluded vs. no occluded alternatives) and spatial (horizontal vs. vertical adjectives) VPT trials. The data are plotted separately for Image and ASCII tasks, and include only spatial relative adjective trials from the director’s perspective. Error bars are 95 percent confidence intervals.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Prompt	Content
General	You are a helpful assistant attempting to submit the correct answer. You have several functions available to help with finding the answer. Each message may perform one function call. You will see the result of the function right after sending the message. If you need to perform multiple actions, you can always send more messages with subsequent function calls. Do some reasoning before your actions, describing what function calls you are going to use and how they fit into your plan. When you have completed the task and have an answer, call the submit() function to report it.
Rotating Figure Task	You are participating in a visual perspective-taking experiment. You will be shown images containing a person (or arrow) and possibly a number or symbol on the ground nearby. Your task is to answer questions about what you can see in the image, considering both your own perspective as the viewer and the perspective of the person/figure in the image. Pay careful attention to: the orientation and position of figures in the image; whether numbers appear as ‘‘6’’ or ‘‘9’’ depending on viewing angle; spatial relationships (left/right, front/behind) from different perspectives; what is visible vs. hidden from different viewpoints. Answer exactly as requested in the prompt.
Director Task	There is a shelf in front of you with a grid of items on it. The user is standing on the opposite side of the shelves and is asking you to pick up a specific item for them that they can see. Some of the cells in the grid are blocked from the user’s view, hiding anything that might be there, indicated by the dark background in the cell. You will receive an image of the grid from your point of view. Submit your answer in the form <column><row>, for example ‘C2’.

Appendix

Table 4: System prompts

Stimulus Set	Description
Control 1	Assessed symbol identification and left–right spatial discrimination. Non-rotated symbols were presented either to the left or right of the figure. Figure orientation was constrained to 70°–110° to ensure that symbols were positioned unambiguously on the left or right side of the image.
Control 2	Assessed line-of-sight judgments. Participants were required to identify either the colour or the spatial location of the wall directly in front of the figure. Figure orientation varied across the full 360° range. Symbols were masked in this condition.
Test 1	Assessed Level 1 visual perspective taking (VPT), specifically whether a symbol was visible to the figure. Symbols were positioned either in front of or behind the figure, defined relative to a 60° field-of-view (FOV) cone extending from the figure. Symbols were oriented to appear either upright or inverted from the figure’s viewpoint, with a small amount of rotational jitter applied. Figure orientation varied across the full 360°.
Test 2	Assessed Level 2 VPT, requiring judgments about how a symbol appeared or where it was located from the figure’s perspective. Trial structure was identical to Test 1, except that symbol placement was restricted to either the front-left or front-right of the FOV.
Test 3	Assessed combined visual and spatial Level 2 VPT. Trial structure was identical to Test 2, except that two symbols were presented simultaneously: one in the front-left region and one in the front-right region of the FOV cone. The second symbol always differed in identity from the first (e.g., 6 vs m).

Appendix

Table 5: Description of stimulus sets for the Rotating Figure Task

Stimulus Set	Context Prompt	Visual Prompt	Spatial Prompt
control_1	There is a person standing in the centre of the image, and a number or letter on the floor nearby.	What number or letter can you see in the image? Respond with a single number or letter:	Is the number or letter on the left or right side of the image? Respond with a single word: LEFT or RIGHT
control_2	There is a person standing in the centre of the image. There are coloured walls along the border.	What colour is the wall directly in front of the person? Respond with a single word: RED, GREEN, BLUE or BLACK	Which side of the image is directly in front of the person? Respond with a single word: LEFT, RIGHT, TOP or BOTTOM
test_1	There is a person standing in the centre of the image, and a number or letter on the floor nearby.	Can the person see the number or letter? Respond with either: CAN SEE or CANNOT SEE	Is the number or letter in front of or behind the person? Respond with a single word: FRONT or BEHIND
test_2	There is a person standing in the centre of the image, and a number or letter on the floor nearby.	What number or letter can the person see? Respond with a single number or letter:	Is the number or letter on the person’s left or right? Respond with a single word: LEFT or RIGHT
test_3	There is a person standing in the centre of the image, and 2 numbers or letters on the floor nearby.	What number or letter can the person see on their {side} side? Respond with a single number or letter:

Appendix

Table 6: Stimulus set prompts for the Rotating Figure Task

	Control 1				Control 2
	Accuracy		Invalid		Accuracy		Accuracy (CR)
Model	Visual	Spatial	Visual	Spatial	Visual	Spatial	Visual	Spatial
o3	0.824	0.853	0.048	0.106	0.716	0.752	0.801	0.843
o4-mini	0.886	0.978	0.037	0.002	0.698	0.715	0.774	0.798
gpt-4o	0.912	1.000	0.048	0.000	0.594	0.696	0.663	0.787
gpt-4o-mini	0.901	0.979	0.041	0.000	0.403	0.608	0.435	0.679

Appendix

Table 7: Mean accuracy in control conditions

Figure 6: Mean MLM accuracy in Control 2 of the Figure Rotation Task . Accuracy is aggregated into 24 bins by figure rotation angle. At 0 ° , the figure faces forward (shared perspective); at 180 ° , it faces backward (opposite perspective). Angles wrap around such that 90 ° corresponds to the figure facing right and − 90 ° (= 270 ° ) corresponds to the figure facing left. The dashed line represents chance performance.

Figure 7: Mean MLM accuracy in Test 1 and 2 of the Figure Rotation Task , by Question Type (Visual vs. Spatial). Accuracy is aggregated into 12 bins by figure rotation angle. At 0 ° , the figure faces forward (shared perspective); at 180 ° , it faces backward (opposite perspective). Angles wrap around such that 90 ° corresponds to the figure facing right and − 90 ° (= 270 ° ) corresponds to the figure facing left. The dashed line represents chance performance.

Question	Model	0–45°	45–90°	90–135°	135–180°	$\chi^{2}$	df	p
	GPT-4o-mini	0.401	0.601	0.682	0.745	203.59	3	$<$ .001
	GPT-4o	0.699	0.672	0.735	0.779	23.57	3	$<$ .001
	o4-mini	0.986	0.812	0.830	0.978	145.93	3	$<$ .001
Visual	o3	0.771	0.792	0.771	0.751	3.53	3	0.317
	GPT-4o-mini	0.569	0.549	0.517	0.525	5.14	3	0.162
	GPT-4o	0.788	0.720	0.697	0.750	18.26	3	$<$ .001

Appendix

Table 8: Mean accuracy in Test 1 by Angular Disparity

Question	Model	0–45°	45–90°	90–135°	135–180°	$\chi^{2}$	df	p
	GPT-4o-mini	0.819	0.511	0.209	0.074	787.19	3	$<$ .001
	GPT-4o	0.315	0.281	0.340	0.539	131.16	3	$<$ .001
	o4-mini	0.448	0.281	0.323	0.572	160.43	3	$<$ .001
Visual	o3	0.597	0.368	0.331	0.721	304.35	3	$<$ .001
	GPT-4o-mini	0.715	0.532	0.430	0.250	322.21	3	$<$ .001
	GPT-4o	0.779	0.593	0.444	0.191	493.58	3	$<$ .001

Appendix

Table 9: Mean accuracy in Test 2 by Angular Disparity

Model	0–45°	45–90°	90–135°	135–180°	$\chi^{2}$	df	p
GPT-4o-mini	0.746	0.364	0.121	0.018	680.52	3	$<$ .001
GPT-4o	0.746	0.418	0.174	0.023	634.71	3	$<$ .001
o4-mini	0.796	0.472	0.253	0.276	493.27	3	$<$ .001
o3	0.766	0.517	0.269	0.311	420.33	3	$<$ .001
Values are estimated marginal probabilities of correct responses.
Simple main effects test angular disparity within each Model.

Appendix

Table 10: Mean accuracy in Test 3 by Angular Disparity

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	-0.403	0.073	-5.487	$<$ 0.001***
Model $\times$ GPT-4o	1.246	0.108	11.582	$<$ 0.001***
Model $\times$ o4-mini	4.629	0.309	14.996	$<$ 0.001***
Model $\times$ o3	1.617	0.113	14.285	$<$ 0.001***
Question Type $\times$ Spatial	0.683	0.103	6.597	$<$ 0.001***

Appendix

Table 11: Mixed effects logistic regression for Test 1: AI Model

\times

Angular Disparity

\times

Question Type

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	1.510	0.096	15.787	$<$ 0.001***
Model $\times$ GPT-4o	-2.287	0.125	-18.318	$<$ 0.001***
Model $\times$ o4-mini	-1.721	0.121	-14.176	$<$ 0.001***
Model $\times$ o3	-1.118	0.121	-9.219	$<$ 0.001***
Question Type $\times$ Spatial	-0.588	0.126	-4.678	$<$ 0.001***

Appendix

Table 12: Mixed effects logistic regression for Test 2: AI Model

\times

Angular Disparity

\times

Question Type

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	1.076	0.084	12.860	$<$ 0.001***
Model $\times$ GPT-4o	0.000	0.118	0.000	1.000
Model $\times$ o4-mini	0.287	0.123	2.332	0.020*
Model $\times$ o3	0.108	0.120	0.902	0.367
Angular Disparity $\times$ 67.5°	-1.636	0.112	-14.588	$<$ 0.001***

Appendix

Table 13: Mixed effects logistic regression for Test 3: AI Model

\times

Angular Disparity

Figure 8: MLM mean accuracy in the Director Task for Visual Perspective conditions (Visual-shared and Visual-different) across Task Formats (ASCII and Image). Spatial-different and participant POV trials were excluded. Error bars are 95 percent confidence intervals.

Figure 9: MLM mean accuracy in the Director Task on trials with spatial adjectives. Spatial perspective taking condition levels are Spatial-shared (Vertical:’topmost’/’bottommost’) and Spatial-different (Horizontal: ’leftmost’/’rightmost’). Perspective Reversal condition levels are Participant point-of-view (POV; ’from your point of view’) and Director POV (’from my point of view’). The participant is required to adopt the director’s spatial perspective on spatial-different trials from the director’s POV only (rightmost grouping). Visual-different perspective-taking trials are not included. Error bars are 95 percent confidence intervals.

Contrast	Context	Estimated marginal means		Estimate	SE	z	p
A. Perspective $\times$ Format
Shared vs. Different	ASCII	Shared: 0.951	Different: 0.420	26.881	7.766	11.393	$<$ 0.001***
Shared vs. Different	Image	Shared: 0.743	Different: 0.292	6.999	1.934	7.041	$<$ 0.001***
B. Format $\times$ Perspective
ASCII vs. Image	Visual-shared	ASCII: 0.951	Image: 0.743	6.730	1.952	6.574	$<$ 0.001***
ASCII vs. Image	Visual-different	ASCII: 0.420	Image: 0.292	1.752	0.482	2.038	0.042*

Appendix

Table 14: Pairwise comparisons with estimated marginal means: AI Model

\times

Visual PT

\times

Task Format

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	2.427	0.214	11.352	$<$ 0.001***
Model: GPT-4o	-1.455	0.109	-13.394	$<$ 0.001***
Model: o4-mini	0.585	0.139	4.222	$<$ 0.001***
Model: o3	3.031	0.346	8.752	$<$ 0.001***
Format: Image	-2.748	0.292	-9.400	$<$ 0.001***

Appendix

Table 15: Mixed effects logistic regression: AI Model

\times

Visual PT

\times

Task Format

Contrast	Context	Estimated marginal means		Estimate	SE	z	p
A. GPT-4o-mini
Shared vs. Different	None	Shared: 0.739	Different: 0.135	18.178	6.020	8.757	$<$ 0.001***
Shared vs. Different	Size	Shared: 0.715	Different: 0.282	6.403	2.090	5.688	$<$ 0.001***
Shared vs. Different	Spatial	Shared: 0.630	Different: 0.198	6.889	2.248	5.915	$<$ 0.001***
B. GPT-4o
Shared vs. Different	None	Shared: 0.416	Different: 0.166	3.594	1.178	3.903	$<$ 0.001***

Appendix

Table 16: Pairwise comparisons with estimated marginal means: AI Model

\times

Visual PT

\times

Relative Adjective

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	1.618	0.249	6.501	$<$ 0.001***
Model: GPT-4o	-1.380	0.103	-13.341	$<$ 0.001***
Model: o4-mini	1.106	0.121	9.103	$<$ 0.001***
Model: o3	2.055	0.158	13.034	$<$ 0.001***
Rel. Adj: Size	-0.119	0.327	-0.365	0.715

Appendix

Table 17: Mixed effects logistic regression: AI Model

\times

Visual PT

\times

Relative Adjective

Contrast	Context	Estimated marginal means		Estimate	SE	z	p
A. GPT-4o-mini
Shared vs. Different	Participant POV	Shared: 0.637	Different: 0.526	1.586	0.346	2.112	0.035*
Shared vs. Different	Director POV	Shared: 0.630	Different: 0.008	224.268	88.320	13.745	$<$ 0.001***
B. GPT-4o
Shared vs. Different	Participant POV	Shared: 0.861	Different: 0.568	4.724	1.067	6.873	$<$ 0.001***
Shared vs. Different	Director POV	Shared: 0.623	Different: 0.292	4.023	0.873	6.411	$<$ 0.001***

Appendix

Table 18: Pairwise comparisons with estimated marginal means: AI Model

\times

Spatial PT

\times

Perspective Reversal

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	1.310	0.173	7.566	$<$ 0.001***
Model: GPT-4o	1.261	0.115	10.942	$<$ 0.001***
Model: o4-mini	2.705	0.159	17.053	$<$ 0.001***
Model: o3	4.041	0.265	15.260	$<$ 0.001***
Spatial PT: Different	-0.461	0.218	-2.112	0.035*

Appendix

Table 19: Mixed effects logistic regression: AI Model

\times

Spatial PT

\times

Perspective Reversal

Contrast	Context	Estimated marginal means		Estimate	SE	z	p
A. Visual $\times$ Spatial
Visual-shared vs. Visual-different	Spatial-shared	Shared: 0.856	Different: 0.292	14.390	3.710	10.344	$<$ 0.001***
Visual-shared vs. Visual-different	Spatial-different	Shared: 0.285	Different: 0.087	4.183	1.148	5.212	$<$ 0.001***
B. Spatial $\times$ Visual
Spatial-shared vs. Spatial-different	Visual-shared	Shared: 0.856	Different: 0.285	14.914	4.073	9.894	$<$ 0.001***
Spatial-shared vs. Spatial-different	Visual-different	Shared: 0.292	Different: 0.087	4.335	1.124	5.656	$<$ 0.001***

Appendix

Table 20: Pairwise comparisons with estimated marginal means: AI Model

\times

Visual PT

\times

Spatial PT

Term	Est./( $\chi^{2}$ )	SE/(df)	z	p
A. Model Coefficients
Intercept	0.909	0.210	4.325	$<$ 0.001***
Model: GPT-4o	-0.028	0.097	-0.292	0.770
Model: o4-mini	2.199	0.138	15.918	$<$ 0.001***
Model: o3	2.841	0.172	16.537	$<$ 0.001***
Visual PT: Different	-1.927	0.270	-7.135	$<$ 0.001***

Appendix

Table 21: Mixed effects logistic regression: AI Model

\times

Visual PT

\times

Spatial PT

Panel of Figure 10 — Figure 10: Relative position of each error made by the models for questions using "leftmost" or "rightmost" selectors split by model and then by perspective.

Panel of Figure 11 — Figure 11: Relative position of each error made by the models for questions using "topmost" or "bottommost" selectors split by model and then by perspective.

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,091 papers

Egocentric Bias in Vision-Language Models
Feb 10, 2026Maijunxian Wang, Yijiang Li, Bingyang Wang +6VLM Evaluation Visual Spatial Reasoning
Cognitive Science Program, University of California, Berkeley · Department of Electrical and Computer Engineering, University of California San Diego · School of Computer Science, Georgia Institute of Technology & Emory University +6
Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models
Jun 2, 2026Mahtab Bigverdi, Linjie Li, Weikai Huang +8Multimodal Understanding Spatial Reasoning
University of Washington · 2Allen Institute for AI · 3Microsoft +1
Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Apr 17, 2026Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India

Feb 10, 2026J/K move · Enter open · S save

Egocentric Bias in Vision-Language Models

Maijunxian Wang, Yijiang Li, Bingyang Wang, Tianwei Zhao+5

Cognitive Science Program, University of California, Berkeley · Department of Electrical and Computer Engineering, University of California San Diego · School of Computer Science, Georgia Institute of Technology & Emory University+6

VLM Evaluation Visual Spatial Reasoning Visual Reasoning VLM Reasoning

Open paper AskPDF arXiv

Visual perspective taking--inferring how the world appears from another's viewpoint--is foundational to social cognition. We introduce FlipSet, a diagnostic benchmark for Level-2 visual perspective taking (L2 VPT) in vision-language models. The task requires simulating 180-degree rotations of 2D character strings from another agent's perspective, isolating spatial transformation from 3D scene complexity. Evaluating 103 VLMs reveals systematic egocentric bias: the vast majority perform below chance, with roughly three-quarters of errors reproducing the camera viewpoint. Control experiments expose a compositional deficit--models achieve high theory-of-mind accuracy and above-chance mental rotation in isolation, yet fail catastrophically when integration is required. This dissociation indicates that current VLMs lack the mechanisms needed to bind social awareness to spatial operations, suggesting fundamental limitations in model-based spatial reasoning. FlipSet provides a cognitively grounded testbed for diagnosing perspective-taking capabilities in multimodal systems.

Abstract

Multimodal Large Language Models

Explore similar work

Egocentric Bias in Vision-Language Models

Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs