Geospatial Vector Geometry as a Native Spatial Modality for Vision-Language Models

Huang Fei; Boyi Chen; Hao Xue; Yang Yue; Yehong Xu; Xiaofang Zhou

cs.AIOct 12, 2026

Geospatial Vector Geometry as a Native Spatial Modality for Vision-Language Models

Authors: Huang Fei, Boyi Chen, Hao Xue, Yang Yue, Yehong Xu, Xiaofang Zhou

Organizations: The Hong Kong University of Science and Technology (Guangzhou), China · The Hong Kong University of Science and Technology, Hong Kong SAR, China

Abstract

Geospatial reasoning links VLMs to real-world action and decision making. Reliable reasoning requires accurate spatial information and effective spatial representations. However, current VLMs obtain this information mainly from language and vision. At larger geographic scales, limited visual observations make it difficult to establish accurate spatial references and recover precise spatial relations. This leads to a bottleneck for VLM spatial perception and reasoning. This paper proposes that geospatial vector geometry can become a native spatial modality within a VLM and introduces GeoVec-VLM. First, we develop a structured geometry representation that preserves the spatial structure and arrangement of vector objects, building an explicit spatial basis. Second, we introduce a geometry interaction strategy that brings this basis into multimodal computation, where it interacts with visual content and language references according to the queried spatial relations. Together, these designs allow perception and reasoning to operate on shared geometric references. Across five spatial reasoning tasks, GeoVec-VLM improves mean accuracy by about 11% relative to the strongest baseline, maintains its gains under geographic transfer, and shows greater geometric consistency. Our results show that geospatial vector geometry offers a promising internal spatial basis for VLM spatial reasoning. https://github.com/TXFR/GeoVector-VLM

Figures & tables

Figure 1: Motivation and overview of GeoVec-VLM. Current VLMs infer spatial relations from raster images and language, where metric geometry and spatial relations are only implicitly represented. GeoVec-VLM introduces geospatial vector data as a native spatial modality, preserves its geometric structure during representation, and integrates it with visual and language information for spatial perception and reasoning.

Figure 2: Architecture of GeoVec-VLM. The geometry encoder preserves within-object structure and scene-level metric structure in geometry tokens. Inside the VLM, geometry tokens interact with visual and spatial-query tokens, are organized into object states, and form query-conditioned role and relation states used by the language generation path. Structured decoders provide spatial supervision during training.

Model	Interface	Direction $\uparrow$	Proximity $\uparrow$	Topology $\uparrow$	Area † $\uparrow$	Distance † $\uparrow$
Qwen3-VL-8B	image	0.312	0.553	0.539	0.528	0.286
	+ GeoJSON	0.322	0.566	0.508	0.602	0.336
DeepSeek-V4-Flash	image	0.635	0.562	0.548	0.739	0.417
	+ GeoJSON	0.609	0.569	0.631	0.744	0.475
Kimi K2.5	image	0.668	0.714	0.652	0.756	0.591
	+ GeoJSON	0.655	0.711	0.693	0.711	0.605

Table 1: Spatial reasoning on geographically disjoint test regions from China. † Area and Distance are evaluated through language generation without corresponding structured decoders or downstream task supervision. SpatialRGPT ∗ ( Cheng et al., 2024 ) and SR-3D ∗ ( Cheng et al., 2026a ) are fine-tuned from released checkpoints on our dataset. Masks denote ground-truth region masks.

Figure 3: Task accuracy can conceal geometrically inconsistent spatial behavior. (a) Mean reciprocal consistency over direction, comparative proximity, and topology. Because repeated errors can remain reciprocally consistent, panels (b–d) examine task-specific geometric constraints. (b) Categorical direction predictions are compared with the directions implied by predicted horizontal and vertical displacements. Polar histograms show the corresponding answer distributions. (c) Proximity is evaluated under candidate-order reversal, which preserves the physically closer parcel. Black ticks indicate hard-sample accuracy. (d) Point–parcel class-conditional recall and parcel–parcel prediction frequencies reveal query-dependent topology bias.

Geometry Representation	Direction $\uparrow$	Proximity $\uparrow$	Topology $\uparrow$	Area † $\uparrow$	Distance † $\uparrow$
Geo2Vec ∗ ( Chu and Shahabi, 2026 )	0.203	0.511	0.526	0.567	0.246
Poly2Vec ∗ ( Siampou et al., 2025 )	0.233	0.503	0.525	0.566	0.248
Serialized GeoJSON	0.249	0.517	0.569	0.613	0.332
Pool GeoVec-VLM	0.704	0.756	0.748	0.603	0.568
GeoVec-VLM	0.804	0.807	0.811	0.778	0.659

Table 2: Retaining structured vector representations through multimodal interaction improves spatial reasoning. All variants use Qwen3-VL-8B with the same training split and task setting. Geo2Vec ∗ and Poly2Vec ∗ are our VLM adaptations of the open-source geometry encoders under this controlled setting. Pool GeoVec-VLM pools projected geometry tokens to a single embedding.

Figure 4: Vector–visual alignment supports cross-modal grounding. (a) CSG requires mapping image-only query points to their corresponding vector parcels. (b–c) Grounding signals weaken when vector–visual interaction is disabled, and further deteriorate after shuffling vector geometry. (d) With aligned inputs, visual support and decision-relevant attribution focus on the corresponding parcel. The final panel shows positive Gradient $\times$ Activation for the gold grounding margin.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Reference-point and path feature construction. Left : boundary simplification and the geometric features extracted for reference-point and path elements. Right : the features are packed into a shared 18-dimensional feature representation, where reference-point and path features occupy complementary channels and inactive channels are zero-padded. The complete feature definitions are summarized in Table 4 .

RP feature	Geometric interpretation	Path feature	Geometric interpretation
$(x_{\mathrm{obj}},y_{\mathrm{obj}})$	Position normalized by the bounding box	$(\Delta x,\Delta y)/1000$	Path endpoint displacement
$(x_{\mathrm{tile}},y_{\mathrm{tile}})/1000$	Offset from the parcel’s H3-cell center	$\log(1+l^{\mathrm{arc}})$	Length of the original boundary path
$\sigma_{q}=\frac{d_{R}(p_{0},p_{q})}{\operatorname{length}(Ring)}$	Normalized position along the ring	$\log(1+l^{\mathrm{chord}})$	Chord length between path endpoints
x $(\sin\theta_{\mathrm{in}},\cos\theta_{\mathrm{in}})$	Incoming direction	$\frac{l^{\mathrm{arc}}}{l^{\mathrm{chord}}}$	How curvy compared to the straight chord
$(\sin\theta_{\mathrm{out}},\cos\theta_{\mathrm{out}})$	Outgoing direction	$\operatorname{atan2}(\Delta y,\Delta x)$	Chord bearing
$(\sin\Delta\theta,\cos\Delta\theta)$	Signed local turning angle	$n$	Number of intermediate original vertices removed by simplification

Appendix

Table 4: Geometric features of reference-point and path elements.

Figure 7: Vector encoder architecture and geometry pretraining. The object encoder processes alternating reference-point and path elements with separate attention streams and endpoint-conditioned cross-stream interaction. The resulting element sequences are then concatenated into a spatial context window and processed by the scene encoder with 2D positional encoding.

Setting	Object level	Scene level
Architecture
Input	18-D element features	128-D object element states
Projection	Linear $18\!\rightarrow\!128$	Spatial $4\!\rightarrow\!128\!\rightarrow\!128$
Encoder	$d=128$ ; 4 heads; FFN $128\!\rightarrow\!512\!\rightarrow\!128$	$d=128$ ; 4 heads; FFN $128\!\rightarrow\!512\!\rightarrow\!128$
Cross-stream MLP	Path: $512\!\rightarrow\!128\!\rightarrow\!128$ RP: $384\!\rightarrow\!128\!\rightarrow\!128$	–
Context limit	256 tokens / parcel	384 tokens / parcel; 16 parcels; 1024 tokens / scene

Appendix

Table 5: Configuration of vector geometry encoder pretraining.

Setting	Value
Stage-1 warmup	12,000 instances; 4 $\times$ Ada6000 GPUs; global batchsize 100; 240 updates
Stage-2 training	8 $\times$ Ada6000 GPUs; global batchsize 1,024; 680 updates
Vector–visual interaction layers	8, 16
Object formation layer	24
Spatial reasoning layers	28, 32, 35
Language–spatial fusion layers	28, 32, 35

Appendix

Table 6: Training configuration for GeoVec-VLM.

Figure 8: Geographic coverage and region-disjoint evaluation design. (a) Geographic coverage of the China, U.S., and Singapore datasets. (b) U.S. metropolitan regions and the six regions used for zero-shot transfer evaluation. (c) Region-disjoint training, validation, and test split in China.

Figure 9: Geometry distribution shifts between training and transfer geographies. Empirical cumulative distributions of (a) projected polygon area, (b) retained reference-point count, and (c) geometry-token sequence length for L4 polygons in the China training regions and the U.S. and Singapore transfer geographies. Transfer distributions are computed over the polygons in the reported evaluation regions.

Figure 10: Representative examples from the benchmark. Each panel shows the visual input, one prompt realization, and the target answer.

Figure 11: Post-hoc explanations for representative comparative proximity and parcel–parcel topology predictions. Explanations are elicited only after the constrained task answers have been collected and are used for qualitative analysis.

Region	Model	Direction $\uparrow$	Proximity $\uparrow$	Topology $\uparrow$	Area † $\uparrow$	Distance † $\uparrow$
U.S.	DeepSeek-V4-Flash	0.652	0.591	0.548	0.758	0.408
	Kimi K2.5	0.695	0.751	0.599	0.781	0.590
	Gemini 2.5 Flash	0.672	0.708	0.578	0.773	0.569
	Qwen 3.7 Plus	0.744	0.738	0.618	0.779	0.692
	GPT-5.1	0.654	0.747	0.664	0.738	0.579
	GeoVec-VLM-8B	0.801	0.799	0.824	0.794	0.657

Appendix

Table 9: Spatial reasoning performance under geography transfer. Baselines use image + GeoJSON; GeoVec-VLM uses native vectors. † Area and Distance receive no corresponding downstream task supervision.

Intervention	Disrupted information	Preserved information
-SceneRef	Object-level H3/centroid placement and RP tile/context coordinates in the shared scene frame.	Within-object relative contour, path attributes, hierarchical identifiers, and object identity.
-RPGeom	Object-local RP coordinates, bearing/turn features, and pointwise tile/local/context coordinates, which are collapsed to the object center.	Object centers and cross-object placement, arc phase, path attributes, and hierarchical identifiers.
-HierID	Explicit part, ring, ring-role, local-structure, and phase identifiers.	Token type, sequence order, RP/path alternation, attention masks, and boundary-neighbor connectivity.

Appendix

Table 10: Definition of the targeted geometry interventions. Each intervention disrupts one source of geometric information while preserving the remaining representation.

Geography	Intervention	Dir. $\Delta$	Prox. $\Delta$	Parcel Topo. $\Delta$	Point Topo. $\Delta$	Area $\Delta$	Dist. $\Delta$
China	$-$ SceneRef	-63.94	-30.06	-20.06	-0.67	-0.61	-23.61
	$-$ RPGeom	+1.39	-4.61	-17.50	-0.56	-4.28	+0.06
	$-$ HierID	-13.06	-8.72	-8.72	-7.17	-3.83	-1.94
U.S.	$-$ SceneRef	-61.39	-28.11	-33.72	-1.11	-4.00	-26.72
	$-$ RPGeom	-2.72	-4.11	-17.17	-0.33	-7.17	+0.39
	$-$ HierID	-10.78	-22.59	-10.78	-12.94	-8.39	-6.17

Appendix

Table 11: Effects of targeted geometry-token interventions across evaluation regions. Values are percentage-point changes in final-answer accuracy relative to the intact GeoVec-VLM. Larger negative changes indicate greater reliance on the disrupted spatial information

Figure 12: Spatial information enters and structures the language path. (a) Vocabulary-head readout of the answer-prefix state around language–spatial fusion. With multimodal inputs, top-1 accuracy rises sharply after the language states read the spatial memory; removing this fusion suppresses the transition. (b) Pairwise distances between direction-conditioned language states increase with physical angular separation. (c) Proximity-state variation tracks the physical distance advantage between candidates.

Figure 13: Total input-token consumption and spatial reasoning accuracy across evaluation sets. Token counts include the complete model input, including visual inputs, language instructions, and geometry when present. For API models, counts are those reported by the corresponding model interface. China and U.S. each contain 10,800 evaluated questions, while Singapore contains 3,600. Geo2Vec and Poly2Vec are evaluated only in the controlled China setting, consistent with Table 2 .

Benchmark	Scope	$N$	Qwen	GeoVec	$\Delta$ (pp)
IFEval	Overall	541	84.47%	85.03%	+0.55
MMLU	Overall	1140	77.54%	77.28%	-0.26
	STEM	360	73.06%	72.78%	-0.28
	Humanities	260	80.77%	80.38%	-0.38
	Other	280	76.79%	76.79%	+0.00
	Social sciences	240	81.67%	81.25%	-0.42

Appendix

Table 12: General capability after GeoVec adaptation. GeoVec retains the trained language parameters while disabling the GeoVec-specific spatial pathway.

Vision-Language Models

Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.

See all 736 papers

Mar 27, 2026

Make Geometry Matter for Spatial Reasoning

Empowered by large-scale training, vision-language models (VLMs) achieve strong image and video understanding, yet their ability to perform spatial reasoning in both static scenes and dynamic videos remains limited. Recent advances try to handle this limitation by injecting geometry tokens from pretrained 3D foundation models into VLMs. Nevertheless, we observe that naive token fusion followed by standard fine-tuning in this line of work often leaves such geometric cues underutilized for spatial reasoning, as VLMs tend to rely heavily on 2D visual cues. In this paper, we propose GeoSR, a framework designed to make geometry matter by encouraging VLMs to actively reason with geometry tokens. GeoSR introduces two key components: (1) Geometry-Unleashing Masking, which strategically masks portions of 2D vision tokens during training to weaken non-geometric shortcuts and force the model to consult geometry tokens for spatial reasoning; and (2) Geometry-Guided Fusion, a gated routing mechanism that adaptively amplifies geometry token contributions in regions where geometric evidence is critical. Together, these designs unleash the potential of geometry tokens for spatial reasoning tasks. Extensive experiments on both static and dynamic spatial reasoning benchmarks demonstrate that GeoSR consistently outperforms prior methods and establishes new state-of-the-art performance by effectively leveraging geometric information. The project page is available at https://suhzhang.github.io/GeoSR/.

Shihua Zhang, Qiuhong Shen, Shizun Wang +2

National University of Singapore, Singapore

Visual Spatial Reasoning 3D Spatial Reasoning Vision-Language Models 3D VLMs

Ask PDF

May 15, 2026

GeoWorld-VLM: Geometry from World Models for Vision-Language Models

Modern Vision-Language Models (VLMs) achieve strong semantic recognition, yet remain brittle on elementary spatial relations such as left of, on, behind, and between. One cause of this failure arises before language reasoning begins: the visual pathway may compress or discard critical 3D structural cues during feature extraction, so the language model receives image representations that are already insufficient for reliable spatial judgment. We introduce GeoWorld-VLM, a VLM-side distillation framework that transfers geometric structure from frozen camera-conditioned video world models into VLMs. GeoWorld-VLM fine-tunes only the image encoder and multimodal projector, aligning post-projector image features with intermediate world-model representations while leaving the main backbone frozen. Given images, a prompt, and a sampled camera trajectory, the world-model teacher converts static visual input into a synthetic multi-view spatial signal. Training combines spatial answer supervision, teacher-student feature alignment, and a preservation anchor to the original VLM. Since the language model remains frozen, GeoWorld-VLM preserves the original model's linguistic capabilities while attributing spatial improvements to the enhanced visual pathway. To evaluate the effectiveness and generality of the proposed method, we apply GeoWorld-VLM to two distinct VLM architectures and observe consistent improvements across both backbones. GeoWorld-VLM improves performance by approximately 4 percent on both the What'sUp and VSR benchmarks, suggesting that world-model-guided visual alignment generalizes across model structures and spatial reasoning datasets.

Renjie Gu, Kaichen Zhou, Yan Luo +1

Harvard AI and Robotics Lab · Kempner Institute for the Study of Natural and Artificial Intelligence · Harvard University

Visual Spatial Reasoning VLM Adaptation VLM Distillation Visually Grounded Reasoning Vision-Language Alignment Knowledge Distillation

Ask PDF

Sep 28, 2026

Geometric Encoding for Spatial Reasoning in Vision-Language Models

Vision-Language Models (VLMs) are far more reliable at recognizing what appears in a video than at reasoning about its spatial and temporal properties, such as metric distances, object dimensions, and consistent object identities across frames. We present Geometric Code, a perception-to-geometry pipeline that computes explicit spatial structure from video and supplies it to VLMs as context to augment reasoning. A perception layer segments and classifies objects and recovers depth, camera pose, and intrinsics from monocular RGB video. A deterministic geometric engine then back-projects, merges, and cleans these outputs into a spatial code, including per-object positions, dimensions, counts, inter-object distances, appearance order, and room geometry. The code is serialized into VLMs' prompts, either alongside the video or replacing it entirely. Specifically, there is no component trained or fine-tuned in our approach. On VSI-Bench, augmenting 2B and 4B open models with the spatial code improves average accuracy by +4.1 points over the frames-only baseline, with the largest gains on numeric estimation tasks such as absolute distance (+24.1 points). The results suggest that explicitly computed geometry, delivered through the language channel, recovers spatial competence that small VLMs cannot extract from pixels alone.

Antonio Jun, Haoshui Yu, Zhengyi Lu +2

School of Arts and Sciences New York University New York City, United States · Dept. of Engineering and Computer Science Oakland University Rochester, United States

Visual Spatial Reasoning Representation Geometry in VLMs VLM Reasoning

Ask PDF

Abstract

Vision-Language Models

Explore similar work

Make Geometry Matter for Spatial Reasoning

GeoWorld-VLM: Geometry from World Models for Vision-Language Models

Geometric Encoding for Spatial Reasoning in Vision-Language Models