Seeing the City or Recognizing the Place? What Street-View Imagery Adds Beyond Existing Urban Data in VLM-Based Urban Sensing

Kaizhen Tan

cs.CVSep 2, 2026

Seeing the City or Recognizing the Place? What Street-View Imagery Adds Beyond Existing Urban Data in VLM-Based Urban Sensing

Authors: Kaizhen Tan

Organizations: New York University

Abstract

Street-view imagery is increasingly analysed with vision-language models (VLMs) to infer urban attributes, but predictive accuracy alone does not show how much a photograph contributes beyond data already available for the same place. Using three VLMs, we compare image-based predictions with existing urban data for seven attributes drawn from five public resources. Each urban unit is evaluated with imagery, with location or text context, and against non-image predictions from nearby observations or public records. We also replace images and add conflicting records to test which source the predictions follow. Nearby observations or public records matched or exceeded image-only predictions for road damage, curb ramps, population, and house price. Images were more informative for building type, building function, and the floor count of low-rise buildings. For floor count, the advantage of images over nearby OpenStreetMap labels increased by 5.7 percentage points per doubling of distance to the nearest labelled building and declined for tall buildings whose rooflines often fell outside the frame. When images and records disagreed, predictions usually moved towards the supplied record. Released OpenFACADES floor annotations, generated with OpenStreetMap floor values as input, showed the same dependence: their agreement with the reference increased with building height, whereas that of image-only reruns decreased. The value of street-view imagery therefore depends on whether an attribute is visible and how well the place is already covered by existing data. Because machine-derived labels are often reused as references, these comparisons also bear on how urban datasets are documented and evaluated.

Figures & tables

Attribute	Resource	Potential image cue	Existing urban data
Road damage	SVRDD ( Ren et al.,, 2024 )	surface condition	nearby road observations
Curb ramp	Project Sidewalk ( Saha et al.,, 2019 )	corner geometry	municipal asset register
Floor count	OpenFACADES ( Liang et al.,, 2025 )	visible elevation and roofline	nearby OSM floor tags
Building type	OpenFACADES ( Liang et al.,, 2025 )	facade form and use cues	local class prevalence
Building function	BuildingSense ( Su et al.,, 2026 )	signage and entrances	nearby labels and place text
Population	CityLens ( Liu et al.,, 2026 )	indirect neighbourhood cues	neighbouring statistics

Table 1 : Urban attributes and the image and non-image sources considered in the study.

Source	Information supplied	Role in the study
Task context	city, coordinates, or released text	VLM without the image
Nearby observations	labels from other urban units	spatial alternative
Public record	register or official statistics	administrative alternative

Table 2 : Non-image sources used in the comparisons.

Figure 1 : Study design for comparing street-view imagery with existing urban data. The figure links the information available for an urban unit to paired source conditions, non-image alternatives, and the comparisons used to identify image and record influence.

Figure 2 : Study resources, urban units, and geographic coverage. (a) Evaluated urban units in the five public resources. Symbol area is proportional to the number of units in each city, and the photographs show example units from OpenFACADES (Mapillary, CC BY-SA) and SVRDD. (b) Unit of analysis, evaluated units, coverage, and, in green, the existing urban source used as the non-image comparator for each resource. (c) Floor-count buildings in the seven metropolitan areas, shaded by reference height, over the OpenStreetMap reference pool in grey; the strip under each map gives the count per height stratum. Basemap: Esri World Light Gray Canvas.

Attribute	Metric	Non-image source: score	Image only			Image gain range
			GPT	Gemini	Qwen
Road damage	Balanced accuracy	Nearby frames: 72.3	50.4	54.7	52.6	$+0.9$ to $+4.2$
Curb ramp	Accuracy	SDOT register: 78.0	76.0	60.7	52.7	$+0.3$ to $+18.3$
Floors, 1–4	Within 1 floor	Nearby tags: 59.3	92.0	88.7	86.0	$+56.0$ to $+90.0$
Floors, 5–14	Within 1 floor	Nearby tags: 43.2–43.9	40.3	64.2	54.7	$+10.0$ to $+34.0$
Floors, 15+	Within 1 floor	Nearby tags: 17.4–17.7	3.4	21.5	18.4	$+2.0$ to $+16.0$

Table 3 : Image-only scores, strongest tested non-image sources, and within-model image gains. Accuracy scores are percentages; MAE uses the released CityLens score scale. GPT, Gemini, and Qwen denote the three models specified in Section 4.3.

Figure 3 : Image contribution across the evaluated urban attributes. (a) Change when the target image is added to the same VLM’s task context. (b) Difference between the image-only prediction and the strongest tested non-image source. (c) CityLens MAE reduction from the location-only prompt, which includes centroid coordinates, to the released image prompt. (d) CityLens MAE reduction from the median of neighbouring official statistics to image-only predictions without the city name. Road damage is scored with balanced accuracy. Positive values favour the image condition. Points identify models and horizontal lines show paired uncertainty intervals. Green text names the existing urban source used in (b) and (d).

Figure 4 : Spatial variation in the image contribution to floor-count prediction. (a) Evaluated buildings in Berlin, shaded by the distance to the nearest labelled building in the OpenStreetMap reference pool (grey). The inset shows one target with its 50 m exclusion ring and its five nearest labelled neighbours over OpenStreetMap building footprints. (b) Image advantage by building-height group and model. (c) Adjusted image advantage as the nearest labelled building becomes more distant; the band estimates use the colours of (a). (d) The distance relationship after omitting each metropolitan area in turn. Basemap: Esri World Light Gray Canvas; footprints and streets from OpenStreetMap contributors.

Figure 5 : Predictions when image and record sources disagree. (a) Change after replacing the target photograph while retaining its task context; road damage is scored with balanced accuracy. (b) Share of predictions that follow a conflicting record. (c) Record influence on floor-count predictions by building height. Points identify models and lines show paired uncertainty intervals.

Figure 6 : Influence of OpenStreetMap reference values on the released OpenFACADES annotations. (a) The released workflow supplied the image and a floor-count record to the annotating model. (b) Mean absolute error by building height for image-only reruns and released annotations on matched buildings; vertical lines show bootstrap 95% intervals, and $n$ is the number of buildings in each height band. (c,d) Floor-band confusion matrices using all available image-only predictions and released annotations, respectively; cells give the percentage of each reference band, and $n$ is the number of predictions or annotations in the band.

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Figure S1 : Selection and reporting in the reviewed street-view building-attribute literature. (a) Records followed through title, abstract, and full-text screening; column height is on a log scale. (b) Reporting practices among the 33 retained studies, one tile per study; filled tiles mark studies that report the practice.

Attribute	Sources considered	Main comparison source	Urban setting
Road damage	prevalence, nearby frames	nearby frames	gaps in a dense survey
Curb ramp	prevalence, nearby labels, SDOT	SDOT register	municipal inventory
Floor count	prevalence, nearby OSM	nearby OSM tags	incomplete building inventory
Building type	prevalence, nearby OSM	modal OSM class	local class distribution
Building function	prevalence, nearby labels	nearby labels	sampled local labels
Population	prevalence, official neighbours	official neighbours	area statistics

Appendix

Table S1 : Existing urban sources considered for each task.

Figure S2 : How prediction sources vary across tasks. (a) Performance from prevalence, existing urban data, task context, and target images. (b) Relationship between non-image predictability and sensitivity to image replacement. Values are interpreted within each task because the metrics and class structures differ. Both panels use hit rates; for road damage this is the mean per-class accuracy, under which always predicting no damage scores 75.3%.

Exclusion radius	50 m	100 m	200 m	400 m
Balanced accuracy	72.3 [69.1, 75.3]	69.4 [66.3, 72.5]	66.3 [63.0, 69.5]	64.3 [61.2, 67.4]
Macro accuracy	82.6 [80.6, 84.6]	80.3 [78.3, 82.3]	77.8 [75.7, 80.0]	76.5 [74.4, 78.5]

Appendix

Table S2 : SVRDD buffered spatial predictor at four exclusion radii. Both metrics average over six damage classes; brackets give bootstrap 95% intervals.

Predictor	Macro acc.	Macro-F1	Balanced acc.	Positive recall
Prevalence benchmark	75.3	0.0	50.0	0.0
Buffered spatial	82.6	59.6	72.3	54.9
GPT image only	74.0	7.9	50.4	4.8
Gemini image only	73.9	27.5	54.7	23.7
Qwen image only	73.3	23.6	52.6	19.7

Appendix

Table S3 : SVRDD results under class-sensitive metrics at the 50 m exclusion radius. Values are percentages averaged over the six damage classes.

Model	Current	Source-era	Difference	Changed
GPT-4o-mini	80.0%	74.0%	$-6.0$ [ $-17.0$ , $+5.0$ ]	30.0%
Gemini 2.5 Flash Lite	63.0%	68.0%	$+5.0$ [ $-5.0$ , $+16.0$ ]	31.0%
Qwen3.5 Flash	50.0%	54.0%	$+4.0$ [ $+1.0$ , $+8.0$ ]	4.0%

Appendix

Table S4 : Matched Project Sidewalk temporal validation on 100 clusters (50 per class). The interval is a paired stratified-bootstrap 95% interval for the source-era-minus-current balanced-accuracy difference.

Task	Donor separation
Road damage	another district, with at least one different damage label
Curb ramp	opposite reference class
Floor count	another building, usually differing by more than one floor
Building type	different reference class
Building function	independently sampled building
Population	area in another city

Appendix

Table S5 : Reference-label separation between target and donor images.

Predictor	Accuracy	Macro-F1	Balanced accuracy
Modal class	58.8	12.3	16.7
Spatial predictor	27.8	18.7	25.2
GPT image only	72.2	64.1	70.8
Gemini image only	77.3	74.2	76.1
Qwen image only	76.3	64.3	63.1

Appendix

Table S6 : Building-type results on the floor-stratified sample. Values are percentages.

Term	Estimate	Bootstrap SE	95% interval
Intercept	$-96.07$	$27.31$	$[-148.65,-42.20]$
$\log_{2}$ nearest-label distance (m)	$+5.74$	$2.18$	$[+1.42,+10.03]$
$\log_{2}$ camera-to-building distance (m)	$+9.12$	$4.04$	$[+1.19,+16.89]$
Capture year minus 2022	$+0.94$	$2.47$	$[-3.95,+5.72]$
Height: 1–4 floors	$+43.92$	$6.55$	$[+30.31,+56.31]$
Height: 5–14 floors	$+19.81$	$5.97$	$[+7.98,+31.64]$

Appendix

Table S7 : Full estimates for the adjusted image-advantage model in Equation S1 . Coefficients are percentage points. Bootstrap standard errors and percentile intervals use 4,000 resamples within metropolitan-area-by-height cells. Reference categories are Amsterdam and buildings with 15 or more floors.

Attribute	$n$	Image	Coordinates	Difference
Floors, within 1	450	50.7%	17.3%	$+33.3$
Building type	291	76.6%	46.4%	$+30.2$
Construction decade	93	71.0%	34.4%	$+36.6$
Distance to hospital	450	20.4%	22.4%	$-2.0$

Appendix

Table S8 : Image and coordinate accuracy for three facade attributes and distance to the nearest hospital.

n

counts paired model–building observations across the three models. A construction-decade answer, a year rounded to the decade, counts as correct when it lies within 15 years of the OpenStreetMap start date; for a start date of 1968, answers of 1960, 1970, and 1980 are correct.

Figure S3 : Image and coordinate performance for three facade attributes and distance to the nearest hospital. Bars and points are coloured by attribute, with rose marking the non-visual attribute; horizontal lines show paired uncertainty intervals.

Predictor	Accuracy	Macro-F1	Balanced accuracy
Spatial (50 m)	56.0%	6.5%	8.4%
GPT image only	63.0%	23.9%	27.6%
Gemini image only	59.0%	24.2%	24.0%
Qwen image only	63.0%	25.4%	31.1%

Appendix

Table S9 : BuildingSense results under overall and class-balanced metrics. Macro metrics average over the 21 reference classes present; invalid predictions remain errors.

Model	Nearby place added	Address shuffle	Image added, no nearby place
GPT	$+23.1\ [10.8,35.4]$	$+4.5\ [-1.0,10.5]$	$+14.1\ [8.1,20.7]$
Gemini	$+21.5\ [7.7,35.4]$	$-1.0\ [-8.0,6.5]$	$+44.4\ [35.6,53.3]$
Qwen	$+18.5\ [6.2,30.8]$	$+3.0\ [-2.5,8.5]$	$+8.9\ [1.5,16.3]$

Appendix

Table S10 : Paired BuildingSense source contrasts in percentage points, with uncertainty intervals.

Figure S4 : Floor-band confusion for (a) image-only reruns, (b) released OpenFACADES annotations, and (c) predictions scored against a supplied conflicting record. Cells give row percentages, normalised within each band of the reference floors in (a) and (b) and of the supplied record in (c);

n

is the number of cases in each row.

Figure S5 : Floor-count predictions under record conflict. (a) Agreement with the conflicting record and the independent reference by building height and model. (b) Agreement when the supplied record is correct or incorrect.

Task	GPT	Gemini	Qwen
Road damage	$+0.8\ [-0.7,2.4]\ (300)$	$+4.6\ [1.9,7.3]\ (300)$	$+4.5\ [1.8,7.3]\ (300)$
Curb ramp	$+46.7\ [39.7,53.7]\ (300)$	$+5.0\ [1.3,8.7]\ (300)$	$+5.4\ [2.7,8.4]\ (299)$
Floor count	$+35.3\ [26.7,44.0]\ (150)$	$+45.6\ [36.2,55.0]\ (149)$	$+37.4\ [28.6,46.3]\ (147)$
Building type	$+58.8\ [46.4,70.1]\ (97)$	$+67.0\ [55.7,78.4]\ (97)$	$+64.9\ [53.6,75.3]\ (97)$
Building function	$+16.5\ [10.0,23.0]\ (200)$	$+18.5\ [11.0,26.0]\ (200)$	$+25.0\ [19.0,31.5]\ (200)$
Population	$-3.3\ [-10.0,3.3]\ (150)$	$+3.4\ [-6.9,13.8]\ (116)$	$+20.5\ [10.3,30.8]\ (146)$

Appendix

Table S11 : Paired image-replacement effects in percentage points. Positive values indicate lower accuracy after replacement. Road damage is scored with balanced accuracy and the CityLens tasks with the share of predictions within one point of the reference. Brackets give uncertainty intervals and parentheses give paired sample sizes.

Task	Model	Image prompt	Statistics input	Difference
Population	GPT	2.76	1.88	$-0.87\ [-1.13,-0.63]\ (150)$
	Gemini	2.18	1.90	$-0.27\ [-0.64,0.07]\ (123)$
	Qwen	1.97	1.98	$+0.01\ [-0.34,0.37]\ (148)$
House price	GPT	2.94	0.25	$-2.69\ [-3.14,-2.28]\ (120)$
	Gemini	4.38	0.41	$-3.97\ [-5.94,-2.00]\ (12)$
	Qwen	1.85	0.24	$-1.61\ [-1.84,-1.38]\ (120)$

Appendix

Table S12 : Released CityLens image prompt compared with the statistics-input condition, in which the VLM receives the location-only prompt and the scores of the five nearest areas but no image. MAE uses the released score scale and is computed on paired areas. The difference is the statistics-input MAE minus the image-prompt MAE, so negative values favour the statistics input. Brackets give uncertainty intervals and parentheses give paired sample sizes.

Explore similar work

AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery
Jul 16, 2026Zahratu Shabrina, Muhammad Asa, Jin Rui +2VLM Evaluation VLM Reasoning
Department of Geography, King’s College London, UK · Resilience Development Initiative, Indonesia · School of Computer Science and Electronic Engineering, University of Surrey, UK +1
Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated
May 30, 2026Rashid MushkaniVLM Evaluation Inter-Rater Reliability
Université de Montréal, Montréal, Québec, Canada · Mila – Québec AI Institute, Montréal, Québec, Canada
You Cannot Photograph the Same Street Twice: Reliability Limits in Vision-Language Measurement of Urban Change
Sep 1, 2026Kaizhen TanVLM Evaluation Vision-Language Models
New York University

Jul 16, 2026, cs.AIJ/K move · Enter open · S save

AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery

Zahratu Shabrina, Muhammad Asa, Jin Rui, Lu Yin+1

Department of Geography, King’s College London, UK · Resilience Development Initiative, Indonesia · School of Computer Science and Electronic Engineering, University of Surrey, UK+1

VLM Evaluation VLM Reasoning Urban Planning

Open paper AskPDF arXiv

This research investigates the potential of Vision-Language Models (VLMs) to infer building typologies: Construction, Current Use, and Storeys from Google Street View (GSV) images. Predictions generated by VLMs are compared with inference by human experts (civil engineers and architects) as a source of manually labelled ground-truth data. We evaluate several state-of-the-art VLMs, including GPT-4o, Claude 3.5 Sonnet, and Gemini 2.0 Flash. By applying different scaling strategies and prompting techniques, we found that Chain-of-Thought prompts provide an overall more stable model performance. We also investigate the reasoning behind VLMs' building-typology predictions by examining the probabilities of keywords appearing in AI explanations. This enabled us to analyse patterns in these reasonings and identify key themes driving both agreements and disagreements between VLM and expert labels. We find that AI tends to focus on visual indicators, whereas human experts place greater emphasis on broader contextual cues and domain knowledge, in addition to visual cues. Overall, VLM can approximate experts' capability in building-typology classification at scale, with an average accuracy of approximately 70%. The study demonstrates the VLM's potential for AI automation in tasks that require pattern recognition and object identification in an urban context. AI have the potential to serve as complementary and collaborative tools for urban analysis, leveraging their strengths in understanding visual patterns. This study contributes to the exploration of the efficiency and scalability of AI visual prediction and provides insights into the reasoning processes that could support automation processes in urban analysis and prediction.

Abstract

Explore similar work

AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

You Cannot Photograph the Same Street Twice: Reliability Limits in Vision-Language Measurement of Urban Change