Dodgersort: Uncertainty-Aware VLM-Guided Human-in-the-Loop Pairwise Ranking

Yujin Park; Haejun Chung; Ikbeom Jang

cs.CVMar 21, 2026

Dodgersort: Uncertainty-Aware VLM-Guided Human-in-the-Loop Pairwise Ranking

Authors: Yujin Park, Haejun Chung, Ikbeom Jang

Organizations: Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea

Abstract

Pairwise comparison labeling is emerging as it yields higher inter-rater reliability than conventional classification labeling, but exhaustive comparisons require quadratic cost. We propose Dodgersort, which leverages CLIP-based hierarchical pre-ordering, a neural ranking head and probabilistic ensemble (Elo, BTL, GP), epistemic--aleatoric uncertainty decomposition, and information-theoretic pair selection. It reduces human comparisons while improving the reliability of the rankings. In visual ranking tasks in medical imaging, historical dating, and aesthetics, Dodgersort achieves a 11--16% annotation reduction while improving inter-rater reliability. Cross-domain ablations across four datasets show that neural adaptation and ensemble uncertainty are key to this gain. In FG-NET with ground-truth ages, the framework extracts 5--20 $\times$ more ranking information per comparison than baselines, yielding Pareto-optimal accuracy--efficiency trade-offs.

Figures & tables

Figure 1: Pipeline of Dodgersort. Phase 1: VLM hierarchical pre-ordering produces coarse ranking. Phase 2: Initialize neural ranking head and probabilistic ensemble (GP, Elo, BTL). Phase 3: Uncertainty-guided MergeSort loop automates confident pairs or queries human, updating ensemble with feedback to produce final ranking $\mathcal{R}$ .

Figure 2: Hierarchical VLM pre-ordering. Left: Prompt structure with $B$ bins. Center: CLIP assigns soft bin probabilities $\{p_{ib}\}$ via image-text cosine similarities. Right: Elo rating trajectories show that proper bin initialization effectively accelerates convergence.

Size	Exhaustive	Sort	EZ-Sort	Ours
$n{=}30$	435	126	90	80
$n{=}50$	1,225	240	144	147
$n{=}100$	4,950	582	475	400

Table 1: Human annotation count averaged across EyePACS, DHCI, TAD66k.

Retina (EyePACS)

Historical (DHCI)

Aesthetics (TAD66k)

Method

ICC

Classification

0.53 (

\pm

0.06)

0.46 (

\pm

0.07)

0.54 (

\pm

0.07)

0.75 (N/A)

0.39 (

\pm

0.06)

0.33 (

\pm

0.05)

0.42 (

\pm

0.06)

0.68 (N/A)

0.46 (

\pm

0.11)

0.38 (

\pm

0.12)

0.50 (

\pm

0.16)

0.64 (N/A)

Sort comparison [ 6 ]

0.72 (

\pm

0.07)

0.56 (

\pm

0.06)

0.72 (

\pm

0.07)

0.89 (N/A)

0.47 (

\pm

0.17)

0.35 (

\pm

0.15)

0.47 (

\pm

0.17)

0.78 (N/A)

0.43 (

\pm

0.01)

0.34 (

\pm

0.01)

0.43 (

\pm

0.01)

0.60 (N/A)

EZ-Sort [ 10 ]

0.85 (

\pm

0.09)

0.76 (

\pm

0.14)

0.85 (

\pm

0.09)

0.94 (N/A)

0.47 (

\pm

0.17)

0.39 (

\pm

0.15)

0.47 (

\pm

0.16)

0.73 (N/A)

0.46 (

\pm

0.02)

0.31 (

\pm

0.02)

0.56 (

\pm

0.06)

0.71 (N/A)

Dodger-Sort (ours)

0.86 (

\pm

0.07)

0.67 (

\pm

0.06)

0.86 (

\pm

0.06)

0.95 (N/A)

0.60 (

\pm

0.05)

0.43 (

\pm

0.04)

0.60 (

\pm

0.05)

0.82 (N/A)

0.47 (

\pm

0.10)

0.32 (

\pm

0.07)

0.57 (

\pm

0.12)

0.72 (N/A)

Table 2: Inter-rater reliability across datasets. Reported: Sp (Spearman), Ke (Kendall), Pe (Pearson), ICC (intraclass correlation). The last three methods (Sort comparison, EZ-Sort, and Dodger-Sort) are pairwise comparison-based approaches.

Size $n$	$\Delta\text{HC}$	$\tau_{\text{EZ}}$	$\tau_{\text{Dodger}}$	EffGain
50	24	0.736	0.8122	1.94 $\times$
100	72	0.671	0.8440	5.94 $\times$
200	50	0.695	0.7935	19.6 $\times$
300	91	0.698	0.7469	12.1 $\times$

Table 3: Accuracy–efficiency on FG-NET (simulated ground truth).

\Delta\text{HC}

: additional human comparisons over EZ-Sort.

\tau_{\text{EZ}}

and

\tau_{\text{Dodger}}

: Kendall’s

\tau

correlation to ground-truth ages. EffGain: relative efficiency of each additional comparison vs. random labeling (EffGain

>1

indicates superior information extraction).

Component Removed	Retina		Face-Age		Historical		Aesthetics
Component Removed	$\Delta\tau$ $\downarrow$	$\Delta\text{HC}$ $\uparrow$	$\Delta\tau$ $\downarrow$	$\Delta\text{HC}$ $\uparrow$	$\Delta\tau$ $\downarrow$	$\Delta\text{HC}$ $\uparrow$	$\Delta\tau$ $\downarrow$	$\Delta\text{HC}$ $\uparrow$
Hierarchical Prompts	$-0.032$	$-15.0$	$-0.030$	$-27.0$	$-0.057$	$-18.0$	$-0.035$	$-2.0$
Neural Ranking Head	$+0.243$	$+388.0$	$+0.214$	$+411.0$	$+0.210$	$+251.0$	$+0.173$	$+394.0$
Full Ensemble	$+0.141$	$+343.5$	$+0.183$	$+360.3$	$+0.057$	$+72.7$	$+0.137$	$+337.3$
No VLM prior (Elo-IG only)	$-0.333$	$-111.0$	$-0.241$	$-61.0$	$-0.236$	$-83.0$	$-0.380$	$-63.0$
Full Dodgersort	All components enabled

Table 4: Cross-domain ablation (n=200). We report

\Delta\tau=\tau_{\text{abl}}-\tau_{\text{full}}

and

\Delta\text{HC}=\text{HC}_{\text{abl}}-\text{HC}_{\text{full}}

(negative

\Delta\tau

/ positive

\Delta\text{HC}

indicate worse than full). Full ensemble removed uses Elo only instead of text+Elo+BTL+GP. No VLM prior (Elo-IG only) skips VLM pre-ordering and selects queries via Elo information gain.

Figure 3: Ablation study across four domains. Full system (green) achieves Pareto-superior trade-offs by balancing accuracy and human comparison cost. Key components (neural ranking head, ensemble, smart selection) each contribute to optimal performance.

Pairwise Ranking

Pairwise ranking infers ordered preferences from comparisons between items, systems, or responses.

See all 62 papers

Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling
Jun 14, 2026Yujin Park, Haejun Chung, Ikbeom JangPairwise Preference Learning Pairwise Comparison
Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea
Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference
Jun 3, 2026Abhishek DivekarLLM-as-a-Judge IR Evaluation
Amazon
Active Learners as Efficient PRP Rerankers
May 14, 2026Jeremías Figueiredo Paschmann, Juan Kaplan, Francisco Nattero +3Pairwise Preference Learning Pairwise Comparison
ELIAS Lab, Departamento de Ingeniería, Universidad de San Andrés

Jun 14, 2026J/K move · Enter open · S save

Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

Yujin Park, Haejun Chung, Ikbeom Jang

Hanyang University, Seoul, Republic of Korea · Hankuk University of Foreign Studies, Yongin, Republic of Korea

Pairwise Preference Learning Pairwise Comparison Pairwise Preference Evaluation Learning to Rank Human-in-the-Loop Annotation Active Learning+1

Open paper AskPDF arXiv

Pairwise comparison is the gold standard for subjective ranking tasks; however, exhaustive annotation requires a massive number of human comparisons (

O(n^2)

). While sorting-based methods have reduced this burden to

O(n\log n)

, they still require expensive human judgment for every single comparison. To further improve annotation efficiency, we propose leveraging a Vision-Language Model (VLM) not as an annotator replacement, but as a question prioritizer to identify which comparisons genuinely require human judgment. The proposed Surprise-Guided MergeSort (SGS) framework achieves this through three integrated components: (1) a bottom-up MergeSort scheduler that structures comparisons and exploits transitivity, (2) a composite Surprise Scorer -- combining position-bias-cancelled VLM confidence, Elo gap, and vote entropy -- to quantify comparison ambiguity, and (3) an adaptive budget allocator that routes high-surprise pairs to humans while automating low-surprise pairs via transitivity inference. Validation was conducted on six diverse benchmarks spanning text similarity (STS-B, BIOSSES, SICKR-STS) and image quality assessment (KonIQ-10k, TID2013, LIVE Challenge). SGS effectively identified and skipped up to 535 non-informative comparisons per session. Consequently, it achieved Kendall's

τ{\times}100

improvements of

+6

+12

over Active Elo under the same total budget. These results demonstrate that combining VLM-guided surprise metrics with algorithmic sorting provides a generally consistent accuracy-efficiency trade-off across diverse domains.

Abstract

Pairwise Ranking

Explore similar work

Surprise-Guided MergeSort: Budget-Efficient Human-in-the-Loop Ranking via Adaptive Comparison Scheduling

Statistically Reliable LLM-Based Ranking Evaluation via Prediction-Powered Inference

Active Learners as Efficient PRP Rerankers