SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera

Shriram Damodaran; Soumyaratna Debnath; Cheston Tan; Lin Wang

cs.CVSep 27, 2026

SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera

Authors: Shriram Damodaran, Soumyaratna Debnath, Cheston Tan, Lin Wang

Organizations: IHPC, A*STAR, Singapore

Abstract

Omnidirectional or 360 cameras provide embodied AI agents with a holistic, wide field-of-view (FoV) view of their surroundings, motivating the use of Multi-modal Large Language Models (MLLMs) for omnidirectional spatial reasoning. However, most MLLMs are trained on conventional 2D perspective images and struggle with the severe distortions and wrap-around discontinuities induced by spherical geometry. Enabling them to generalize to non-Euclidean 3D spaces without retraining therefore remains challenging. We propose SphMind, a training-free, plug-and-play framework that decouples semantic perception from geometric reasoning. Rather than requiring MLLMs to learn spherical geometry internally, SphMind preserves their semantic capabilities while handling geometry externally. We introduce a Spherical Harmonics-based Spatial Graph (SHSG) that models spatial relationships through equivariant transformations on the sphere, together with Inference-Time Geometric Grounding (IGG), a model-agnostic closed-loop optimization process that aligns MLLM representations with spherical geometric constraints during inference. Experiments on three benchmarks show that SphMind achieves over 21.4% average improvement in directional reasoning on MP3D and Stanford2D-3D, outperforms prompt-engineering baselines by 8.7% on the real-world ODI-Bench, and improves rotational invariance by 5.9% under panorama rotations, without additional training or dataset-specific tuning. In-the-wild evaluations further show that SphMind resolves directional reasoning queries that baseline vision-language models fail to answer correctly.

Figures & tables

Figure 1 : ( a ) Given ‘in-the-wild’ real-world 360 capture, existing VLMs fail at spherical reasoning, leading to incorrect spatial predictions. Our SphMind can enable VLMs to generalize to non-Euclidean 3D spaces without retraining. ( b ) Accuracy comparison of VLMs vs. SphMind across four categories; bars show per-category gains, shaded line shows min–max range across models. Project Page: https://empactlab.github.io/SphMind-NeurIPS-2026/ .

Figure 2 : Geodesic distances resolve ERP distortion. VLMs misjudge depth from pixel size and wrap-around, ranking the chair closer than the monitor. SphMind uses geodesic priors on $\mathbb{S}^{2}$ to correctly identify the board as nearer.

Figure 3 : Overview of SphMind . Given an ERP image and query, SHSG lifts detections to $\mathbb{S}^{2}$ and builds an SH-encoded scene graph. Wigner-D scoring produces a candidate answer set $\mathbf{Y}$ . IGG then steers the frozen VLM hidden state to the final grounded answer $y^{*}$ .

Figure 4 : SHSG directional reasoning on MP3D. Query: “What is to the left of the window?” (a) ERP image with candidates: $\bullet$ door, $\bullet$ board, $\bullet$ chair, $\boldsymbol{+}$ window anchor. (b) SH spherical views: RGB scene on $\mathbb{S}^{2}$ (left); SH field centred at anchor (center); after Wigner-D rotation the peak shifts left, aligning with $\bullet$ chair (right).

Figure 5 : IGG pipeline. Given the VLM hidden state $\mathbf{h}_{0}$ , ranked evidence $\mathbf{S}_{d}$ , IGG performs $T$ norm-preserving Riemannian updates to steer $\mathbf{h}$ toward the geometrically consistent answer $y^{*}$

Figure 6 : (A) Token-space PCA: IGG shifts $\mathbf{h}_{0}\rightarrow\mathbf{h}^{\prime}$ toward the correct token ( chair ) while preserving semantic structure. (B) Panorama candidates: $\bigstar$ marks the board; $\bullet$ indicates the correctly localized chair below it on $\mathbb{S}^{2}$ . (C) Cosine similarity: confirms semantic consistency, valid tangent update, and correct gradient alignment, showing correction of the initial model bias through geometric grounding.

Model	Type	Category	Matterport3D						Stanford2D
			Scene 1	Scene 2	Scene 3	Scene 4	Scene 5	Scene 6	Scene 1	Scene 2	Scene 3	Scene 4
Qwen2.5-7B	Baseline	Overall	40	42.9	36.2	24.3	41.7	38.6	35.7	39.8	31.5	37.2
		Presence	62.5	67.8	62.5	53.5	72.2	64.5	100	70.3	58.4	66.9
		Counting	20	10	14.2	13	11.1	20	8.3	17.6	9.8	15.2
		Distance	40	40	27	42.2	44.8	41.7	50	45.6	34.9	43.8
		Direction	35.3	41.2	28.6	22.5	32.1	20.7	14.3	32.8	24.7	30.4

Table 1 : Performance across datasets and scenes.

Table 2 : Comparison across panoramic reasoning benchmarks. SphMind consistently improves performance across diverse settings, including (a) object-directional inference (ODI-Bench), (b) viewpoint generalization (OpenView), and (c) rotation-consistent spatial grounding (RCS).

Config	Ov.	Dir.	Dist.	$\Delta$
Full	58.4	53.1	64.1	—
Component removal
No grad	50.5	46.6	54.9	$-7.9$
w/o $C(y_{k})$	49.8	45.6	54.0	$-8.6$
SHSG only	45.1	42.5	48.5	$-13.3$
Rand $C(y_{k})$	37.7	34.0	43.3	$-20.7$

Table 3 : Ablation on MP3D (Qwen2.5-7B).

\Delta

= change vs Full.

Figure 7 : t-SNE embedding of the SHSG spherical field at 450 samples on the sphere colored by direction.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 8 : SH inner product decay vs. angular separation for $L\in\{2,7,12\}$ . $L=7$ (blue) decays sharply near $0^{\circ}$ and oscillates minimally in the mid-range, whereas $L=12$ (green) rings more severely despite a sharper onset. The dashed black curve shows the ideal $\cos(\delta)$ reference. ReLU gating in $S_{\mathrm{SH}}$ makes ringing beyond $90^{\circ}$ irrelevant, justifying $L_{\max}=7$ as the operating point.

Figure 9 : SHSG corrects directional reasoning via SO(3)-equivariant scoring. For “What is in front of the board?” , SphMind places objects on the sphere: $+$ SH center; $\times$ suppressed candidates (low Wigner- $D$ $S_{\mathrm{SH}}$ ); $\bigstar$ top candidate (chair) aligned with the rotated anchor lobe; $\bigstar$ anchor.

Figure 10 : ERP pixel-space vs. geodesic $S^{2}$ scoring on a real MP3D scene. (a) Cartesian ERP: iso-distance rings (pink dotted) are distorted ellipses; the ERP straight-line path (red dashed) is geometrically misleading. (b) Mollweide $S^{2}$ : geodesic rings (teal dotted) are metric circles; marker area $\propto S_{\mathrm{SH}}$ score; Bed scores $0.82$ (correct), Cabinet $0.31$ , all others $\leq 0.18$ . The green geodesic arc connects the anchor (purple star) to the top-ranked candidate along the true great-circle path.

Figure 11 : Gram matrix of Wigner-D score vectors across all six query directions (MP3D). Diagonal $=1.00$ ; all off-diagonal Pearson correlations lie in $[-0.13,\,-0.05]$ , confirming near-orthogonality of the six directional subspaces induced by $S_{\mathrm{SH}}$ . The $-0.13$ entries at antipodal pairs (Right/Left, Front/Behind) reflect residual spectral leakage at $\delta\approx\pi$ , which the bandwidth-derived cone gate $\Delta_{d}\approx 15.2^{\circ}$ is designed to suppress.

Figure 12 : Effect of IGG on candidate probability distribution. Query: “What is in front of the board?” Heatmap shows $P(y_{k}|\mathbf{h})$ projected onto candidate bounding box regions. Before IGG (left): probability mass concentrates on the geometrically incorrect candidate ( $\oplus$ , wrong answer). After IGG (right): the Riemannian update shifts probability mass to the geometrically correct candidate ( $\bigstar$ , chair), consistent with SHSG’s directional score ranking. This shift occurs without modifying any model weights. See Table 3 (SHSG-only vs. Full) for the quantitative contribution of IGG beyond candidate filtering alone.

Parameter	Value	Role
$\beta_{\mathrm{geo}}$	$0.3$	Weight of geometric violation term
$\beta_{\mathrm{KL}}$	$0.7$	Weight of prior-preservation term
$\tau$	$2.0$	Softmax temperature for $P_{\tau}$ ; raised to maintain gradients across all candidates
$\varepsilon_{0}$	$0.15$	Base guidance scale
$T$	$2$	Max correction steps
$\delta$	$0.005$	Early stopping threshold on $\|\Delta\mathcal{E}\|$

Appendix

Table 4 : IGG hyperparameters and their roles.

Figure 13 : SO(3)-equivariance of the SH directional field. Qualitative results to show directional reasoning using SH scores and equivariance properties. Top: ERP panorama with SH field overlay; the anchor (blue) and query arrow rotate consistently with the scene. Bottom: Pure SH field on $\mathbb{S}^{2}$ (equirectangular); the peak tracks the rotation exactly ( $<\!1^{\circ}$ error), a direct consequence of Wigner- $D$ equivariance (Eq. 2 ).

Configuration	Distance Acc.	$\Delta$
Full ( $\alpha{=}0.6$ )	64.1	—
$\alpha{=}0.4$	63.2	$-0.9$
$\alpha{=}0.8$	63.7	$-0.4$
$\alpha{=}1.0$ (elev. only)	61.3	$-2.8$
$\alpha{=}0.0$ (density only)	60.8	$-3.3$
Linear decay vs exp. kernel	63.9	$-0.2$

Appendix

Table 5: Distance accuracy sensitivity on MP3D (Qwen2.5-7B). Neither signal alone matches their combination;

\alpha

is robust within

[0.4,0.8]

Method	Dir.	Full $\uparrow$	$90^{\circ}$ $\uparrow$	$180^{\circ}$ $\uparrow$	$270^{\circ}$ $\uparrow$	Unk% $\downarrow$
ERP-Pixel	front	0.040	0.441	0.198	0.453	16.5
	left	0.029	0.370	0.189	0.415	30.6
	right	0.044	0.448	0.218	0.451	26.1
	behind	0.058	0.490	0.244	0.483	55.1
	All	0.043	0.437	0.212	0.451	33.4
SphMind	front	0.250	0.654	0.589	0.648	8.2

Appendix

Table 6 : Full per-direction RCS breakdown on Stanford 2D-3D-S. Full RCS per direction computed as

p_{90}\times p_{180}\times p_{270}

per direction subset. Unk% = fraction of questions returning unknown averaged across all rotation variants.

Figure 14 : SO(3)-equivariance of the SH directional field. Each column shows the same scene under azimuthal rotation ( $0^{\circ}$ , $+90^{\circ}$ , $+270^{\circ}$ ). Top: ERP panorama with SH field overlay; the anchor (blue) and query arrow rotate consistently with the scene. Bottom: Pure SH field on $\mathbb{S}^{2}$ (equirectangular); the peak tracks the rotation exactly ( $<\!1^{\circ}$ error), a direct consequence of Wigner- $D$ equivariance (Eq. 2 ).

Pair type	Direction GA	Distance GA	Overall GA
Matched (same query, rotated scene)	$0.31\pm 0.08$	$0.22\pm 0.09$	$0.27\pm 0.09$
Unmatched (different query)	$0.01\pm 0.02$	$0.01\pm 0.02$	$0.01\pm 0.02$
Gap ( $\Delta$ )	$+0.30$	$+0.21$	$+0.26$

Appendix

Table 7: Gradient Alignment (GA) between IGG correction vectors

g_{\psi}=\nabla_{h}\mathcal{L}|_{\psi}

for matched query pairs (same scene and query direction, different ERP rotation

\psi\in\{90^{\circ},180^{\circ},270^{\circ}\}

) versus unmatched pairs (different query groups). Unmatched GA

\approx 0

is expected by isotropy of random vectors in

\mathbb{R}^{4096}

(

\sigma\approx 1/\sqrt{d}\approx 0.016

). The significant gap confirms that IGG correction vectors are geometrically structured rather than arbitrary.

Parameter	Value	Role
$\beta_{\mathrm{geo}}$	$0.30$	Weight of geometric violation term in IGG energy; controls how strongly $\mathcal{C}(y_{k})$ shapes the gradient
$\beta_{\mathrm{KL}}$	$0.70$	Weight of KL prior-preservation anchor; KL-dominant setting ensures geometry provides a structured correction rather than overriding the VLM distribution
$\tau$	$2.0$	Softmax temperature for $P_{\tau}$ ; raised above default to maintain non-negligible gradients across all candidates
$\varepsilon_{0}$	$0.15$	Base Riemannian step size
$T$	$2$	Maximum Riemannian correction steps; early stopping via $\delta$ typically terminates at $T{=}2$
$\delta$	$0.005$	Early stopping threshold on $\|\Delta\mathcal{E}\|$ ; prevents unnecessary updates when energy has converged

Appendix

Table 8 : IGG and SHSG hyperparameters.

$\beta_{\mathrm{geo}}$ / $\beta_{\mathrm{KL}}$	Ov.	Dir.	Dist.	$\Delta$
0.2/0.8	57.2	51.4	63.1	$-1.2$
0.3/0.7	58.4	53.1	64.1	—
0.4/0.6	57.9	52.3	63.7	$-0.5$
0.5/0.5	57.4	51.8	63.2	$-1.0$
0.6/0.4	56.6	50.7	62.4	$-1.8$

Appendix

Table 9 : Sensitivity to energy weights

\beta_{\mathrm{geo}}

\beta_{\mathrm{KL}}

(MP3D, Qwen2.5-7B).

Explore similar work

Jun 3, 2026cs.LG

From Symbolic to Geometric: Enabling Spatial Reasoning in Large Language Models

Recent large language models (LLMs) often appear to exhibit spatial reasoning ability; however, this capability is largely \emph{symbolic}, arising from pattern matching over spatial language rather than true \emph{geometric} reasoning over space. Because LLMs operate on discrete tokens, they lack native support for continuous spatial representations, explicit geometric computation, and structured spatial operators. To address this limitation, we introduce the \emph{Spatial Language Model (SLM)}, the first multimodal LLM that treats location information as a first-class modality and enables geometric spatial reasoning within the model's inference process. SLM directly operates on learned spatial representations rather than textual descriptions of spatial relations. To support effective training, we construct a \emph{Spatial Instruction Dataset} that aligns spatial representations, atomic geometric operations, and natural language instructions. We further propose a new benchmark named \emph{SpatialEval}, which is designed to evaluate spatial reasoning across attributes, distance, topology, and relative-position tasks. Extensive experiments show that SLM significantly outperforms existing LLM-based approaches that rely on symbolic reasoning via prompt engineering or textual abstraction, demonstrating the benefits of integrating geometric spatial representations for robust spatial reasoning. Our instruction dataset, evaluation benchmark, model training codes, and models' checkpoints can be found at: \hyperlink{https://github.com/chuchen2017/SLM}{https://github.com/chuchen2017/SLM}.

Chen Chu, Bita Azarijoo, Li Xiong +2

University of Southern California Los Angeles, CA, USA · Emory University Atlanta, GA, USA · Novateur Research Solutions Ashburn, Virginia, USA

Spatial Reasoning LLM Reasoning Strategies Multimodal Large Language Models Symbol-

Ask PDF

Nov 10, 2025cs.CV

SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language tasks, but continue to struggle with spatial reasoning. Existing spatial MLLMs rely on large-scale datasets, explicit 3D inputs, architecture-specific modifications, or sparse Reinforcement Learning (RL) methods that provide insufficient guidance for spatially-grounded reasoning. We introduce SpatialThinker. To our knowledge, it is the first MLLM unifying Scene Graph Generation (SGG) and visual reasoning in a single pass via online RL. The model simulates human-like spatial perception by constructing a mental scene graph of task-relevant objects and relations, and reasoning toward an answer via dense spatial rewards. Our contributions are threefold: (1) SGG-grounded reasoning: integrating SGG directly within the reasoning chain rather than as a disjoint preprocessing step; (2) STVQA-7K: a high-quality spatial VQA training dataset via a scalable synthesis pipeline; and (3) a dense spatial reward design that enforces structured grounding during RL and generalizes to improve broad visual perception. SpatialThinker-7B achieves 3.6

\times

larger gains over SFT and

1.7\times

better in- and out-of-distribution generalization than sparse RL. Trained on only 7K samples, SpatialThinker-7B matches GPT-5 and outperforms GPT-4o, while SpatialThinker-30B surpasses both GPT-5 and Claude 4 Sonnet on average across 14 spatial and real-world benchmarks, demonstrating that structured spatial grounding with reward-aligned reasoning enables robust spatial understanding with limited data.

Hunar Batra, Haoqin Tu, Hardy Chen +3

University of Oxford · University of California, Santa Cruz

Spatial Reasoning Spatial Grounding Multimodal Large Language Models Geospatial Reasoning Scene Understanding

Ask PDF

Aug 5, 2026cs.LG

Disentangling 3D Modeling from Spatial Reasoning

In this work, we explore an alternative paradigm for spatial reasoning by explicitly disentangling 3D perception from reasoning, rather than jointly acquiring implicit 3D perception and reasoning through large-scale training. Our key observation is that modern perception models excel at estimating continuous 3D geometry, whereas large language models (LLMs) are particularly effective at compositional and symbolic reasoning. Motivated by these complementary strengths, we propose the Disentangled Spatial Reasoner (DiSR), a simple yet effective framework that reconstructs the physical world into structured 3D evidence using off-the-shelf expert perception models and fine-tunes an LLM with LoRA to perform reasoning solely over this explicit geometric evidence. Without large-scale 3D VQA training or complex tool-use policies, DiSR achieves competitive performance on popular spatial reasoning benchmarks. Beyond its strong performance, DiSR offers improved interpretability, modularity, and computational efficiency, demonstrating that explicit separation of perception and reasoning is a scalable and effective alternative paradigm to end-to-end modeling for spatial intelligence.

Haoze Sun, Jiequan Cui, Qingshan Xu +1

HFUT · USTC

3D Spatial Reasoning Spatial Reasoning 3D Representation

Ask PDF