UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

Yecheng Zhang; Rong Zhao; Zhizhou Sha; Yong Li; Lei Wang; Ce Hou; Wen Ji; Hao Huang

cs.CVFeb 23, 2026

UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

Authors: Yecheng Zhang, Rong Zhao, Zhizhou Sha, Yong Li, Lei Wang, Ce Hou, Wen Ji, Hao Huang,

Organizations: Tsinghua University · University College London · Hong Kong University of Science and Technology · Peking University ·

Abstract

Vision-language models (VLMs) can describe urban scenes in rich detail, yet consistently fail to produce reliable human preference labels in domain-specific tasks such as safety assessment and aesthetic evaluation. The standard fix, fine-tuning or RLHF, requires large-scale annotations and model retraining. We ask a different question: can a frozen VLM be aligned with human preferences without modifying any weights? Our key insight is that VLMs are strong concept extractors but poor decision calibrators. We propose a three-stage post-hoc pipeline that exploits this asymmetry: (i) interpretable evaluation dimensions are automatically mined from consensus exemplars; (ii) an Observer-Debater-Judge chain extracts robust concept scores from the frozen VLM; and (iii) locally-weighted ridge regression on a hybrid manifold calibrates these scores to human ratings. Applied as UrbanAlign on Place Pulse 2.0, the framework reaches 72.2% accuracy (kappa=0.45) across six perception categories, outperforming all baselines by +11.0 pp and zero-shot VLM by +15.5 pp, with full interpretability and zero weight modification.

Figures & tables

Figure 1: Three approaches to urban perception: (a) supervised baselines (e.g., Siamese networks, segmentation regression) require labelled data and lack interpretable intermediates; (b) zero-shot VLM prompting is training-free but produces opaque, uncalibrated judgements; (c) UrbanAlign routes predictions through VLM-discovered semantic dimensions for interpretable, locally calibrated comparison.

Figure 2: UrbanAlign framework. Stage 1 mines semantic dimensions from consensus exemplars; Stage 2 distils dimension scores via Observer–Debater–Judge; Stage 3 calibrates via LWRR on the hybrid differential manifold. $\mathcal{D}_{\mathrm{ref}}\cap\mathcal{D}_{\mathrm{val}}\cap\mathcal{D}_{\mathrm{test}}=\varnothing$ .

	Safety		Beautiful		Lively
Method	Acc. (%)	$\kappa$	Acc. (%)	$\kappa$	Acc. (%)	$\kappa$
ResNet50 Siamese ( Koch et al., 2015 )	54.8 (54.0)	0.16 (0.18)	58.6 (56.0)	0.18 (0.19)	45.7 (42.5)	$-$ 0.07 ( $-$ 0.06)
CLIP Siamese ( Radford et al., 2021 )	66.7 (64.4)	0.34 (0.31)	61.4 (57.3)	0.23 (0.20)	48.2 (44.8)	$-$ 0.01 ( $-$ 0.01)
Seg. Regression ( Chen et al., 2018 )	61.9 (59.8)	0.25 (0.23)	57.1 (53.3)	0.15 (0.13)	51.9 (48.3)	0.04 (0.04)
GPT-4o zero-shot ( OpenAI, 2024 )	61.9 (59.8)	0.23 (0.21)	62.9 (58.7)	0.25 (0.21)	49.4 (46.0)	0.01 (0.01)
BT Logistic ( Bradley and Terry, 1952 ) †	65.5 (63.2)	0.31 (0.29)	58.6 (54.7)	0.17 (0.15)	56.8 (52.9)	0.17 (0.15)

Table 1: Comparison on Place Pulse 2.0 across all six perception categories. All values reported as excl-equal (incl-equal). UrbanAlign outperforms all baselines including same-feature controls (72.2% vs. 61.2% best baseline).

Safety ( $n{=}283$ , avg 64.7%)		Beautiful ( $n{=}125$ , avg 56.7%)		Lively ( $n{=}259$ , avg 53.0%)
Dimension	Power	Dimension	Power	Dimension	Power
Greenery & Landscaping	71.2	Greenery & Nat. Elem.	64.4	Human Presence	59.6
Lighting Adequacy	71.2	Color Coordination	62.2	Building Diversity	57.7
Building Maintenance	69.2	Pedestrian Amenities	55.6	Street Furniture	57.7
Pedestrian Infrastructure	65.4	Street Cleanliness	55.6	Color & Lighting	53.8
Street Activity	57.7	Architectural Coherence	55.6	Commercial Activity	53.8

Table 2: Per-dimension discriminative power (Mode 4). “Power” = fraction (%) of test pairs where the score difference on that single dimension correctly predicts the human winner (50% = chance).

n

= number of test pairs; “avg” = mean power across all dimensions in that category.

Figure 3: Qualitative examples across six perception categories. A denotes the human-selected winner (green border). Centre chart : per-dimension concept scores for image A ( green ) and image B ( blue ) on the left half, and LWRR contribution towards A on the right half; the $y$ -axis lists dimension names and bar length represents each dimension’s contribution to $\hat{\delta}_{q}$ ( Eq. 5 ); positive LWRR bars support A, negative bars support B. The summary line reports the raw score comparison and calibrated LWRR delta; “Corrected” marks cases where calibration overturns an incorrect raw prediction. Right chart : baseline confidence bars ( orange = conf. A, purple = conf. B, grey = equal); UrbanAlign is shown in bold.

	Single-Shot	Multi-Agent	Multi-Agent Gain
Single-Image	50.9 / 0.08	55.9 / 0.18	+5.0
Pairwise	51.9 / 0.10	72.2 / 0.45	+20.3
Pairwise Gain	+1.0	+16.3	—

Table 3:

2{\times}2

factorial averaged across all six categories (accuracy% /

\kappa

, post-LWRR). Per-category results in Table 17 .

	Safety	Beaut.	Lively	Wealthy	Boring	Depress.	Avg
Raw Acc. (%)	68.4	63.9	58.5	62.9	38.9	42.9	55.9
Aligned Acc. (%)	81.6	69.8	69.4	74.0	70.2	68.2	72.2
$\Delta$ (pp)	+13.3	+5.9	+10.9	+11.1	+31.3	+25.3	+16.3
Raw $\kappa$	0.38	0.29	0.17	0.26	$-$ 0.22	$-$ 0.13	0.13
Aligned $\kappa$	0.64	0.38	0.40	0.48	0.41	0.37	0.45
$n$ (test pairs)	260	115	238	107	83	87	—

Table 4: LWRR calibration gain (Mode 4, accuracy % excl. “equal”) across all six categories. LWRR improves all six categories; the largest gain is on boring (+31.3 pp). Average gain: +16.3 pp.

Appendix figures & tables19 assets

Supplementary material from the paper’s appendix.

Appendix

Category	Raw Pairs	$N{\geq}2$	Cons. (%)	Left	Right	Equal
Safety	360,095	1,412	87.6	682	652	78
Beautiful	171,858	618	90.0	325	266	27
Lively	260,428	1,288	85.8	595	613	80
Wealthy	148,739	581	90.9	264	274	43
Boring	124,371	449	90.3	211	206	32
Depressing	129,229	471	91.4	223	230	18

Appendix

Table 5: Dataset filtering pipeline across all six perception categories. Raw pairs are unique image comparisons in Place Pulse 2.0.

N{\geq}2

retains only pairs with at least two independent votes. Consensus (%) is the average majority-vote agreement among retained pairs. All retained pairs are used in the experiments. Label distribution is by majority vote.

Figure 4: Distribution of image pairs across vote thresholds (

{\geq}1

{\geq}5

votes) for all six categories (log-scaled). Our consensus filter retains pairs with

N{\geq}2

votes.

Dimension	Mean $w$	Std $w$	CV
Building Modernity	2.305	2.676	1.16
Infrastructure Condition	$-$ 1.858	2.074	1.12
Vegetation Maintenance	1.722	1.996	1.16
Street Cleanliness	1.355	1.318	0.97
Pavement Integrity	$-$ 1.068	2.387	2.23
Vehicle Quality	$-$ 1.043	1.763	1.69

Appendix

Table 6: LWRR local weight statistics across 117 test pairs ( wealthy category, Mode 4).

Category	Dimension	Mean $w$	Std $w$	CV
Safety ( $n{=}283$ )	Visibility Clarity	$-$ 0.421	1.702	4.04
	Lighting Adequacy	0.291	1.401	4.81
	Building Maintenance	0.280	1.311	4.67
Beautiful ( $n{=}125$ )	Arch. Coherence	1.471	4.825	3.28
	Visual Complexity	1.193	10.960	9.19
	Pedestrian Amenities	0.887	4.901	5.53

Appendix

Table 7: Top-3 LWRR local weights by

|\text{Mean}|

for each perception category (Mode 4). High Std relative to Mean confirms that dimension importance varies substantially across the manifold.

$K_{\max}$	Safety	Beaut.	Lively	Wealthy	Boring	Depress.	Avg
10	59.2	50.7	51.3	55.7	53.6	54.1	54.1
20	66.7	53.8	61.3	57.1	52.8	58.6	58.4
30	70.3	52.2	61.6	49.3	50.7	55.6	56.6
50	66.7	54.4	55.6	52.6	58.6	57.4	57.6

Appendix

Table 8: Full

K_{\max}

sweep (Mode 4, all categories, accuracy % excl. equal).

$\alpha$	Sem. Wt.	Safety	Beaut.	Lively	Wealthy	Boring	Depress.	Avg
0.1	90%	61.3	52.9	57.9	55.6	50.0	57.8	55.9
0.2	80%	64.6	54.5	57.3	56.4	52.9	58.8	57.4
0.3	70%	66.7	53.8	61.3	57.1	52.8	58.6	58.4
0.5	50%	62.0	56.1	49.3	56.8	52.1	56.9	55.5
0.7	30%	62.0	60.3	51.4	54.4	58.0	62.5	58.1

Appendix

Table 9: Full

\alpha

sweep (Mode 4, all categories, accuracy % excl. equal).

$\tau_{\mathrm{kernel}}$	Acc (%)	$\kappa$	$N_{\mathrm{aligned}}$
0.1	54.9	0.09	87
0.3	56.1	0.12	87
0.5	56.8	0.13	87
0.8	57.0	0.13	87
1.0	57.1	0.13	87

Appendix

Table 10: Full

\tau_{\mathrm{kernel}}

sweep (Mode 4, wealthy ).

$\lambda$	Acc (%)	$\kappa$	$N_{\mathrm{aligned}}$
0.01	56.1	0.12	87
0.05	56.1	0.12	87
0.1	56.1	0.12	87
0.5	56.8	0.13	87
1.0	57.1	0.13	87

Appendix

Table 11: Full

\lambda_{\mathrm{ridge}}

sweep (Mode 4, wealthy ).

$\varepsilon$	Safety	Beaut.	Lively	Wealthy	Boring	Depress.	Avg
0.3	66.3	56.5	57.5	72.3	59.7	52.8	60.9
0.5	66.3	56.5	56.4	72.3	59.4	52.8	60.6
0.8	66.3	56.5	57.1	72.3	61.5	54.3	61.3
1.2	66.2	57.4	56.6	72.8	63.5	54.4	61.8
2.0	67.1	59.7	59.1	72.4	63.3	56.2	63.0

Appendix

Table 12: Full

\varepsilon

(EQUAL_EPS) sweep (Mode 4, all categories, accuracy % excl. equal).

Category	Acc. (%)	$\kappa$	Key Parameters
Safety	81.6	0.636	$K{=}30$ , $\alpha{=}0.2$ , sel ${=}0.6$
Beautiful	69.8	0.384	$K{=}10$ , $\alpha{=}0.5$ , sel ${=}0.6$
Lively	69.4	0.404	$K{=}50$ , $\alpha{=}0.7$ , sel ${=}0.6$
Wealthy	74.0	0.480	$K{=}50$ , $\alpha{=}0.7$ , sel ${=}0.6$
Boring	70.2	0.414	$K{=}30$ , $\alpha{=}0.7$ , sel ${=}0.6$
Depressing	68.2	0.370	$K{=}20$ , $\alpha{=}0.2$ , sel ${=}0.6$

Appendix

Table 13: Best per-category configurations from combined random search (Mode 4, accuracy % excl. equal).

Main pipeline (4,819 pairs $\times$ 6 categories)
Component	Calls/pair	Total calls	Tokens (M)	Cost (USD)
Mode 1 (Single-Image Direct)	2	9,638	10.2	$45.59
Mode 2 (Pairwise Direct)	1	4,819	5.1	$22.79
Mode 3 (Single-Image Multi-Agent)	6	28,914	30.5	$136.76
Mode 4 (Pairwise Multi-Agent)	3	14,457	15.3	$68.38
End-to-end dimension optimization (20% subsample, Mode 2, 12 trials)

Appendix

Table 14: API cost breakdown for the full experimental pipeline (4,819 pairs across 6 categories, GPT-4o pricing). Stage 3 LWRR is purely local computation with zero API cost.

Trial	Phase	$\tau$	Safety	Beaut.	Lively	Wealthy	Boring	Depress.	Avg
0	Explore	0.85	58.8	61.5	47.1	55.6	47.1	50.0	53.3
1	Explore	0.88	76.5	46.2	47.1	77.8	35.3	68.8	58.6
2	Explore	0.91	70.6	69.2	35.3	61.1	35.3	56.3	54.6
3	Explore	0.94	64.7	53.8	47.1	50.0	35.3	62.5	52.2
4	Explore	0.97	35.3	53.8	52.9	72.2	47.1	31.3	48.8
5	Explore	1.00	58.8	46.2	52.9	55.6	47.1	56.3	52.8

Appendix

Table 15: E2E dimension optimization: per-category accuracy (%, excluding “equal”) across 12 trials (all six categories, Mode 4). Bold marks the per-category best. Trials 0–5 are explore phase; 6–11 are converge phase.

Category	Trial	Phase	Acc (%)	$\kappa$	Optimised Dimensions
Safety	1	Explore	76.5	0.534	Lighting Adequacy, Pedestrian Infrastructure, Building Maint., Street Activity, Visibility Clarity, Greenery & Landscaping
Beautiful	2	Explore	69.2	0.458	Greenery & Nat. Elements, Street Cleanliness, Arch. Coherence, Color Coordination, Pedestrian Amenities, Visual Complexity
Lively	7	Converge	58.8	0.308	Human Presence, Vegetation & Greenery, Building Diversity, Street Furniture, Commercial Activity, Color & Lighting, Public Art & Signage
Wealthy	1	Explore	77.8	0.579	Façade Quality, Vegetation Maint., Pavement Integrity, Vehicle Quality, Building Modernity, Infrastructure Cond., Street Cleanliness, Lighting Quality
Boring	10	Converge	64.7	0.407	Visual Complexity, Activity Presence, Arch. Diversity, Vegetation Variety, Color Vibrancy, Street Furniture Presence, Spatial Enclosure
Depressing	1	Explore	68.8	0.437	Façade Condition, Street Cleanliness, Greenery Presence, Lighting Quality, Visual Clutter

Appendix

Table 16: Per-category assembled best dimensions from E2E optimization (default calibration hyperparameters:

K{=}20

\alpha{=}0.3

, sel

{=}1.0

). Each category’s optimal dimensions may come from a different trial, enabling independent specialisation. Note: these results differ from Table 1, which uses per-category optimised hyperparameters with original dimensions (72.2% avg); here only the dimension set is optimised (69.3% avg). The two optimisation axes are complementary.

	Mode 1	Mode 2	Mode 3	Mode 4	MA Gain	PW Gain	Mode4 $-$ 1
	(SI+SS)	(PW+SS)	(SI+MA)	(PW+MA)	(SI)	(MA)
Safety	45.2	59.5	61.4	81.6	+16.2	+20.2	+36.4
Beautiful	51.4	57.1	52.9	69.8	+1.4	+16.9	+18.3
Lively	58.0	50.6	51.9	69.4	$-$ 6.2	+17.5	+11.4
Wealthy	55.4	42.2	63.9	74.0	+8.4	+10.1	+18.6
Boring	47.3	51.4	50.0	70.2	+2.7	+20.2	+22.9

Appendix

Table 17:

2{\times}2

factorial design across all six categories (accuracy %, post-LWRR, excl. “equal”). Mode 4 (pairwise + multi-agent) consistently outperforms all other modes. “Synergy” = Mode 4

-

Mode 1

-

(sum of individual effects).

	Mode 1	Mode 2	Mode 3	Mode 4
Raw Acc. (%)	48.7	63.3	58.8	62.9
Aligned Acc. (%)	55.4	42.2	63.9	74.0
$\Delta$ (pp)	+6.7	$-$ 21.1	+5.1	+11.1
Raw $\kappa$	0.208	0.263	0.266	0.256
Aligned $\kappa$	0.159	$-$ 0.037	0.313	0.480
$n$ (test pairs)	117	117	117	107

Appendix

Table 18: LWRR alignment gain (accuracy % excluding “equal”) per mode, wealthy category. Mode 4 shows the largest calibration gain (+11.1 pp).

Dataset	Domain	Scale	Format	Access
Urban & Scene Perception
Place Pulse 2.0 $\star$	Urban 6-dim	1.17M pairs	Pairwise	pulse.media.mit.edu
SPECS $\star$	Urban 10-dim	10 dims + demo.	Pairwise	ual.sg/project/specs
ScenicOrNot $\star$	Scenicness	200K+ locs	Rating	scenic.mysociety.org
Streetscore	Safety	Boston/NYC	Rating	MIT Media Lab
Image Generation Quality

Appendix

Table 19: Public human preference datasets. “Pairwise” indicates the native annotation format uses pairwise comparisons (the same format as UrbanAlign). Datasets marked with

\star

are the most direct candidates for UrbanAlign transfer experiments.

	GPT-4o		Qwen2.5-VL-72B		$\Delta$ (Qwen–GPT)
Category	Raw	Aligned	Raw	Aligned	Raw	Aligned
Safety	68.4	81.6	65.3	81.2	$-$ 3.1	$-$ 0.4
Beautiful	63.9	69.8	63.2	70.2	$-$ 0.7	+0.5
Lively	58.5	69.4	53.2	66.7	$-$ 5.3	$-$ 2.7
Wealthy	62.9	74.0	64.5	77.1	+1.6	+3.1
Boring	38.9	70.2	41.5	69.0	+2.6	$-$ 1.2

Appendix

Table 20: GPT-4o vs. Qwen2.5-VL-72B backbone comparison (Mode 4, excluding equal labels). “Raw” = Stage 2 multi-agent scores only; “Aligned” = after Stage 3 LWRR calibration with per-backbone optimised hyperparameters.

Category	$n$	Acc. (%)	95% CI	$\boldsymbol{\kappa}$ 95% CI
Safety	49	81.8	[69.4, 91.8]	[0.41, 0.83]
Beautiful	43	70.0	[55.8, 83.7]	[0.11, 0.64]
Lively	49	69.4	[55.1, 81.6]	[0.15, 0.64]
Wealthy	50	74.2	[62.0, 86.0]	[0.24, 0.72]
Boring	47	70.6	[57.4, 83.0]	[0.17, 0.66]
Depressing	44	68.5	[54.5, 81.8]	[0.11, 0.64]

Appendix

Table 21: Bootstrap 95% confidence intervals (1,000 iterations, excluding “equal” labels).

Category	Pairs	Human Agr. (%)	Human $\kappa$	UrbanAlign (%)	Gap (pp)
Safety	1,950	81.6	0.674	81.8	$+$ 0.2
Beautiful	736	82.9	0.709	70.0	$-$ 12.9
Lively	1,675	78.4	0.633	69.4	$-$ 9.0
Wealthy	682	85.7	0.746	74.2	$-$ 11.5
Boring	538	85.5	0.744	70.6	$-$ 14.9
Depressing	542	86.7	0.755	68.5	$-$ 18.2

Appendix

Table 22: Human split-half agreement ceiling (100 random splits) compared to UrbanAlign accuracy.

VLM Calibration

Vision-language model calibration makes confidence scores better reflect when multimodal predictions are correct and grounded in visual evidence.

See all 81 papers

May 30, 2026

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

Vision-language models (VLMs) are increasingly used to generate structured descriptions of street-level imagery for tasks such as streetscape auditing, mapping, and public consultation. These uses combine observable attributes with appraisal categories, and the human targets are often distributions of judgments with disagreement and explicit non-response. This paper argues that benchmarking VLMs for urban perception should treat disagreement and abstention as measurement outcomes, report inter-annotator reliability alongside model alignment, and treat the label space and scoring policy as negotiable artifacts when outputs are intended to inform urban governance. We ground the argument in a benchmark of 100 Montreal street scenes annotated along 30 dimensions by 12 participants from seven community organizations, and in a deterministic zero-shot evaluation of seven VLMs. Across dimensions, model agreement with human consensus co-varies with dimension-level human reliability, and for the appraisal dimension Overall Impression models and annotators exhibit distributional mismatch including different rates of Not applicable. We close with actions for benchmark creators, model developers, and institutions to make uncertainty and benchmark assumptions visible in evaluation reports.

Rashid Mushkani

Université de Montréal, Montréal, Québec, Canada · Mila – Québec AI Institute, Montréal, Québec, Canada

VLM Evaluation Inter-Rater Reliability Abstention in VLMs Vision-Language Models

Ask PDF

Jun 2, 2026

A Dataset for Dynamic Human Preferences for Vision Language Models

Given the increased adoption of Vision Language Models (VLMs) in human-interactive settings, it is important that we evaluate how well these models can adapt to real-time preferences for different users. While an increasing number of vision-language benchmarks have recently been introduced, they focus largely on evaluating static capabilities and generally-held preferences learned from extensive training data. This work introduces a new benchmark for evaluating the ability of VLMs to understand dynamic human-preferences, i.e. preferences that are passed in-context at inference time. We provide an automated pipeline for generating this benchmark with variations on image dependence, a dynamic multi-modal human-preference dataset, and evaluations of state-of-the-art models on the novel benchmark.

Hannah Gao, Dylan Hadfield-Menell, Rachel Ma

Massachusetts Institute of Technology · MIT Cambridge, MA

VLM Evaluation Human Preference Evaluation Vision-Language Models

Ask PDF

Mar 19, 2026

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.

Rui Hong, Shuxue Quan

George Mason University, Fairfax, VA, USA · Independent Researcher

Selective Prediction for VLMs Sycophancy in VLMs Vision-Language Models Vision-Language Grounding Visual Evidence Sensitivity in VLMs Hallucination Detection in VLMs+1

Ask PDF

Abstract

VLM Calibration

Explore similar work

Benchmarks for Vision-Language Models in Urban Perception Should Be Reliability-Aware and Negotiated

A Dataset for Dynamic Human Preferences for Vision Language Models

To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs