3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

Yiping Chen; Jinpeng Li; Wenyu Ke; Yang Luo; Jie Ouyang; Zhongjie He; Li Liu; Hongchao Fan

cs.CVMar 24, 2026

3DCity-LLM: Empowering Multi-modality Large Language Models for 3D City-scale Perception and Understanding

Authors: Yiping Chen, Jinpeng Li, Wenyu Ke, Yang Luo, Jie Ouyang, Zhongjie He, Li Liu, Hongchao Fan,

Organizations: School of Geospatial Engineering and Science, Sun Yat-sen University, Zhuhai, 519082, China. · College of Electronic Science, National University of Defense Technology, Changsha, 410000, China. · Department of Civil and Environmental Engineering, Norwegian University of Science and Technology, Trondheim, Norway. · National Geomatics Center of China, Beijing, 100830, China.

Abstract

While multi-modality large language models excel in object-centric or indoor scenarios, scaling them to 3D city-scale environments remains a formidable challenge. To bridge this gap, we propose 3DCity-LLM, a unified framework designed for 3D city-scale vision-language perception and understanding. 3DCity-LLM employs a coarse-to-fine feature encoding strategy comprising three parallel branches for target object, inter-object relationship, and global scene. To facilitate large-scale training, we introduce 3DCity-LLM-1.2M dataset that comprises approximately 1.2 million high-quality samples across seven representative task categories, ranging from fine-grained object analysis to multi-faceted scene planning. This strictly quality-controlled dataset integrates explicit 3D numerical information and diverse user-oriented simulations, enriching the question-answering diversity and realism of urban scenarios. Furthermore, we apply a multi-dimensional protocol based on text-similarity metrics and LLM-based semantic assessment to ensure faithful and comprehensive evaluations for all methods. Extensive experiments on two benchmarks demonstrate that 3DCity-LLM significantly outperforms existing state-of-the-art methods, offering a promising and meaningful direction for advancing spatial reasoning and urban intelligence. The source code and dataset are available at https://github.com/SYSU-3DSTAILab/3D-City-LLM.

Figures & tables

Dataset	Source	Modality	N.I.	C.S.	Sample	Task
RSVQA ( Lobry et al., 2020 )	Sentinel 2, HRO	Text, 2D	$\checkmark$	$\times$	1.1M	QA
VQA-TextRS ( Al Rahhal et al., 2022 )	Google Earth, PatternNet, NWPU	Text, 2D	$\times$	$\times$	6.2k	QA
RSGPT ( Hu et al., 2025 )	DOTA	Text, 2D	$\times$	$\times$	3.5k	QA
VRSBench ( Li et al., 2024 )	DOTA, DIOR	Text, 2D	$\checkmark$	$\times$	205k	Multiple Tasks
EarthVQA ( Wang et al., 2024a )	LoveDA	Text, 2D	$\times$	$\times$	209k	Multiple Tasks
UrBench ( Zhou et al., 2025 )	Google Street View, Cityscapes, VIGOR	Text, 2D	$\checkmark$	$\checkmark$	11.6k	Multiple Tasks

Table 1: Comparison of representative city-scale vision-language datasets. “N.I.” means the numerical information (3D coordinates, distance, angle, etc.). ”C.S.” means the contextual simulation for different user groups

Dataset	Area	Sample	Word		Token
Dataset	Area	Sample	Que. Len.	Ans. Len.	Que. Len.	Ans. Len.
NuScenes-QA ( Qian et al., 2024 )	Roadside	460k	14.32	1.04	15.59	1.05
City-3DQA ( Sun et al., 2024 )	Urban-scale	450k	12.56	1.80	14.80	1.81
SVM-City ( Sun et al., 2025 )	Urban-scale	567k	-	-	-	-
3DCity-LLM-1.2M (ours)	Urban-scale	1.2M	13.49	39.47	17.29	49.44

Table 2: Statistical overview of the existing 3D large-scale vision-language datasets designed for city scenes. We report the number of samples, average question length ( Que. Len. ) and answer length ( Ans. Len. ) across different datasets. ’Roadside’ denotes vehicle-mounted perspective data, ’Urban-scale’ refers to city-level data reconstructed from aerial LiDAR or photogrammetry. The SVM-City dataset is currently unavailable for public access

Task Category	Sample (Proportion)	Word		Token
Task Category	Sample (Proportion)	Que. Len.	Ans. Len.	Que. Len.	Ans. Len.
Object Caption	350k (28.3%)	10.32	60.46	12.79	73.89
Object Localization	94k (7.6%)	16.48	50.64	18.34	73.35
Object Analysis	470k (37.9%)	11.25	23.42	14.07	28.63
Relationship Computation	56k (4.6%)	25.45	40.62	53.83	65.21
Scene Caption	160k (12.9%)	9.73	32.40	11.33	38.56

Table 3: Statistical overview of 3DCity-LLM-1.2M dataset. We report the number of samples, average question length ( Que. Len. ) and answer length ( Ans. Len. ) across different task categories

Figure 1: The statistical information and representative cases of 3DCity-LLM-1.2M dataset. We provide two representative cases for each task type. The specific 3D numerical information is highlighted with underscores

Figure 2: Automated data generation pipeline of 3DCity-LLM-1.2M dataset, leveraging instance-level masks and landmark annotations from the SensatUrban, UrbanBIS, and City-BIS datasets. We first construct city scene attributes for each environments as input for VLM, then prompt the VLM to generate diverse and high-quality QA pairs grounded in the scene with multiple instructions. In total, we built 1.2M samples spanning object caption, object localization, object analysis, relationship computation, scene caption, scene analysis and scene planning tasks

Figure 3: Model architecture of 3DCity-LLM. 3DCity-LLM receives target object, its neighboring objects, city scene and text query as multi-modality inputs, then identifies task type based on the text query and activates the corresponding feature encoding branches before producing the final answers

Figure 4: The coarse-to-fine feature encoding in 3DCity-LLM. (a) Object Encoding (b) Relationship Encoding (c) Scene Encoding

Method	Object-level Tasks					Relationship-level Tasks					Scene-level Tasks
Method	B-4	ROU.	MET.	Log.	Rel.	B-4	ROU.	MET.	Log.	Rel.	B-4	ROU.	MET.	Log.	Rel.
Training-free but payment-required LVLMs ( $>$ 70B)
ChatGPT-5.1	17.72	29.01	33.95	7.92	6.50	16.64	30.11	38.42	6.08	4.95	11.44	24.26	35.63	7.27	6.04
Qwen3-VL Max	15.23	27.12	34.65	7.05	6.39	18.99	32.70	40.69	6.30	5.37	9.32	21.63	32.37	7.40	6.12
DeepSeek-R1	15.60	24.20	37.44	7.48	6.17	14.09	27.04	37.91	5.75	4.28	10.42	20.95	31.90	6.94	5.87
Gemini 3	14.94	25.02	33.87	7.45	6.47	18.06	29.50	39.52	5.62	5.07	10.60	21.91	31.33	7.43	5.90

Table 4: Overall performance comparison on 3DCity-LLM-1.2M dataset. ”B-4”, ”ROU.”, ”MET.”, ”Log.” and ”Rel.” are abbreviations for ”BLEU-4”, ”ROUGE-L”, ”METEOR”, ”Logicality” and ”Reliability”, respectively. The score range for ”Logicality” and ”Reliability” metrics is from 0 to 10. Object-level tasks consist of object caption, object localization and object analysis. Scene-level tasks consist of scene caption, scene analysis and scene planning

Method	Sentence-wise			City-wise
Method	Single-hop	Multi-hop	All	Single-hop	Multi-hop	All
ScanQA ( Azuma et al., 2022 )	76.42	28.31	49.28	64.84	27.03	47.33
3D-VisTA ( Zhu et al., 2023 )	79.23	44.67	59.63	71.28	43.87	56.74
Sg-CityU ( Sun et al., 2024 )	80.95	50.75	63.94	78.46	50.50	63.76
City-VLM ( Sun et al., 2025 )	81.74	56.80	67.30	78.84	52.26	64.70
3DCity-LLM (Ours)	82.41	57.75	68.55	79.10	51.23	64.49

Table 5: Overall performance comparison on City-3DQA dataset under sentence-wise and city-wise evaluation settings

Method	Object Caption					Object Localization					Object Analysis
Method	B-4	ROU.	MET.	Log.	Rel.	B-4	ROU.	MET.	Log.	Rel.	B-4	ROU.	MET.	Log.	Rel.
LLaVA	11.98	22.01	29.52	5.32	3.92	14.75	28.19	31.10	5.98	3.96	32.57	45.01	52.29	7.84	6.46
Chat 3D	9.75	18.55	25.74	5.10	2.98	11.91	23.97	26.41	3.64	3.07	28.99	40.07	45.24	7.79	5.75
Chat Scene	12.29	22.70	30.95	5.62	4.20	19.01	32.36	35.50	5.91	4.10	36.74	50.96	56.62	8.01	6.84
City-VLM	11.75	20.49	31.97	5.85	4.01	20.07	33.23	37.20	6.12	4.34	35.57	52.02	54.36	7.56	6.43
3DCity-LLM (Ours)	13.80	23.24	32.25	5.89	4.37	19.36	30.14	39.41	6.28	4.52	39.22	52.69	57.02	8.09	6.95

Table 6: Performance comparison on object-level tasks in 3DCity-LLM-1.2M dataset. ”B-4”, ”ROU.”, ”MET.”, ”Log.” and ”Rel.” are abbreviations for ”BLEU-4”, ”ROUGE-L”, ”METEOR”, ”Logicality” and ”Reliability”, respectively. The score range for ”Logicality” and ”Reliability” metrics is from 0 to 10

Method	Scene Caption					Scene Analysis					Scene Planning
Method	B-4	ROU.	MET.	Log.	Rel.	B-4	ROU.	MET.	Log.	Rel.	B-4	ROU.	MET.	Log.	Rel.
LLaVA	20.74	26.19	30.37	7.14	6.03	14.82	24.20	32.93	6.96	5.04	15.57	28.41	38.56	6.91	6.06
Chat 3D	18.46	24.81	26.39	6.82	4.19	12.09	20.07	29.83	6.58	4.73	12.84	24.48	33.20	6.76	5.09
Chat Scene	23.44	28.90	32.98	6.78	6.27	14.30	23.39	34.09	7.09	5.50	14.97	26.25	37.69	7.72	5.85
City-VLM	22.51	30.20	31.54	7.12	5.94	15.79	24.94	34.27	7.24	5.44	16.54	29.32	40.15	7.54	6.30
3DCity-LLM (Ours)	25.01	31.75	35.08	7.65	6.50	17.42	28.54	36.60	7.30	5.76	16.92	28.87	41.85	7.51	6.12

Table 7: Performance comparison on scene-level tasks in 3DCity-LLM-1.2M dataset. ”B-4”, ”ROU.”, ”MET.”, ”Log.” and ”Rel.” are abbreviations for ”BLEU-4”, ”ROUGE-L”, ”METEOR”, ”Logicality” and ”Reliability”, respectively. The score range for ”Logicality” and ”Reliability” metrics is from 0 to 10

Figure 5: Qualitative results on the object-level tasks from 3DCity-LLM-1.2M dataset. (a) Object Caption, (b) Object Analysis, (c) Object Localization

Figure 6: Qualitative results on the relationship-level tasks from 3DCity-LLM-1.2M dataset. 3DCity-LLM model can extract object information in urban scenes and calculate distance and orientation based on their coordinates

Feature Encoding			Object-level Tasks
Local View	Shape	Landmark	B-4	ROU.	MET.	Log.	Rel.
$\checkmark$			29.61	42.66	45.94	7.25	5.64
$\checkmark$	$\checkmark$		29.82	42.47	46.50	7.20	5.72
$\checkmark$	$\checkmark$	$\checkmark$	30.64	42.81	48.89	7.33	6.02

Table 8: Ablation study of different feature encoding for object-level tasks

Feature Encoding		Relationship-level Tasks
Geometry	Landmark	B-4	ROU.	MET.	Log.	Rel.
$\checkmark$		18.24	33.87	41.26	5.01	4.04
$\checkmark$	$\checkmark$	20.98	34.61	42.09	5.13	4.36

Table 9: Ablation study of different feature encoding for relationship-level tasks

Figure 7: Qualitative results on the scene-level tasks from 3DCity-LLM-1.2M dataset. (a-b) Scene Caption, (c-d) Scene Analysis, (e-f) Scene Planning

Feature Encoding		Scene-level Tasks
Global View	Landmark	B-4	ROU.	MET.	Log.	Rel.
$\checkmark$		19.72	29.03	35.82	7.47	5.93
$\checkmark$	$\checkmark$	20.11	29.87	37.84	7.50	6.16

Table 10: Ablation study of different feature encoding for scene-level tasks

Figure 8: Qualitative examples of diverse queries for the same object (top) or scene (bottom). Queries with varying linguistic styles and word lengths are employed to evaluate the 3DCity-LLM’s generalization ability on text modality

Figure 9: Qualitative examples of the same queries for diverse object (top) or scene (bottom). Different objects and scenes are employed to evaluate the 3DCity-LLM’s generalization ability on 3D modality

Figure 10: The representative examples of LLM-based evaluation process from three LLM evaluators (ChatGPT-5, Qwen3-VL Plus and DeepSeek-V3). Each evaluator assigns logicality and reliability metrics with explicit scoring justification for evidence usage, specificity, and spatial-reasoning completeness

Task Category	Logicality				Reliability
Task Category	ChatGPT-5	Qwen3-VL Plus	DeepSeek-V3	Average	ChatGPT-5	Qwen3-VL Plus	DeepSeek-V3	Average
Object Caption	5.87	5.74	6.08	5.89	4.75	3.88	4.50	4.37
Object Localization	6.56	6.65	5.62	6.28	4.06	5.79	3.72	4.52
Object Analysis	8.14	8.28	7.85	8.09	7.06	6.98	6.81	6.95
Relationship Computation	4.59	6.46	4.34	5.13	4.40	5.58	3.12	4.36
Scene Caption	7.61	7.39	7.95	7.65	6.59	6.05	6.87	6.50

Table 11: Comparison of logicality and reliability metrics provided by three independent LLM evaluators (ChatGPT-5, Qwen3-VL Plus, DeepSeek-V3) for all seven task categories in 3DCity-LLM-1.2M dataset. By averaging the logicality and reliability metrics from different evaluators, the final assessment is more stable and less affected by evaluator-specific biases

Figure 11: Correlation heatmap of LLM metrics across the object caption, object localization, object analysis, relationship computation, scene caption, scene analysis, and scene planning tasks. The LLM-based metrics show moderately strong but not absolute inter-evaluator correlation with the most values in 0.5 to 0.8

Task Category	B-4	ROU.	MET.	Log.	Rel.	Time (s)
Object-level Task	30.64	42.81	48.89	7.33	6.02	7.40
Relationship-level Task	20.98	34.61	42.09	5.13	4.36	13.69
Scene-level Task	20.11	29.87	37.84	7.50	6.16	14.54

Table 12: Efficiency and performance of 3DCity-LLM. We report BLEU-4, ROUGE-L, METEOR, logicality, reliability, and average inference time

3D VLMs

Three-dimensional vision-language models connect language with spatially grounded visual data for 3D understanding, reasoning, and action.

See all 213 papers

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes
Sep 14, 2026Anubhav Khanal, Prabigya Acharya, Roshni Poudel +5Spatial Reasoning Benchmarks 3D Spatial Reasoning
Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal · INSAIT, Sofia University, Bulgaria · State University of New York at Stony Brook, Korea
Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM
Mar 29, 2026Haifeng Huang, Yilun Chen, Zehan Wang +23D VLMs 3D Scene Understanding
School of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China. · Shanghai AI Laboratory, Shanghai 200032, China.
CityLLM: A framework for natural-language querying of semantic 3D city models
Jul 16, 2026Rabindra Lamsal, Sisi Zlatanova, Johnson Xuesong ShenGeospatial QA Question Answering
GRID Lab, School of Built Environment, UNSW Sydney, NSW 2052, Australia · School of Civil and Environmental Engineering, UNSW Sydney, NSW 2052, Australia

Sep 14, 2026J/K move · Enter open · S save

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Anubhav Khanal, Prabigya Acharya, Roshni Poudel, Sujan Kapali+4

Nepal Applied Mathematics and Informatics Institute for research (NAAMII), Nepal · INSAIT, Sofia University, Bulgaria · State University of New York at Stony Brook, Korea

Spatial Reasoning Benchmarks 3D Spatial Reasoning 3D Scene Understanding 3D VLMs

Open paper AskPDF arXiv

Vision-language models excel at 2D image understanding but remain limited in 3D spatial reasoning. Progress is hindered by limitations in current benchmarks. First, 3D datasets often rely on point clouds that capture geometry but discard rich visual features like texture, text, and materials. Second, annotations treat objects in isolation while ignoring real-world hierarchical organization (scenes, rooms, functional areas, object groups). Third, evaluation tasks focus narrowly on basic recognition rather than multi-step spatial reasoning. In this context, we introduce SceneBench, a benchmark of 966 photorealistic 3D scenes reconstructed with Gaussian Splatting and densely annotated with hierarchical semantics spanning scenes, rooms, functional areas, object groups, and individual objects. These annotations are produced through a human-in-the-loop pipeline combining vision-language models with roughly 1,500 human-hours of iterative refinement and verification, producing over 183K annotated nodes with textual descriptions and 3D bounding boxes. Building on this representation, we define three evaluation tasks: Existence-Based Questions probing object attributes, Spatial Intelligence Questions covering counting, size comparison, distance, and directional relations, and Grounded Question-Reasoning-Answer (QRA) triplets requiring multi-step reasoning across semantic levels. Experiments with state-of-the-art vision-language models show that while models perform well on basic recognition tasks (e.g., up to 85% accuracy for detection), performance drops substantially on hierarchical and compositional reasoning (e.g., down to 60% for counting), revealing limitations not captured by existing benchmarks. SceneBench provides a realistic testbed for developing and evaluating models capable of fine-grained spatial reasoning in photorealistic 3D environments.

Abstract

3D VLMs

Explore similar work

SceneBench: A Hierarchical Benchmark for Vision-Language Understanding of 3D Scenes

Chat-Scene++: Exploiting Context-Rich Object Identification for 3D LLM

CityLLM: A framework for natural-language querying of semantic 3D city models