Detecting Fine-Grained Differences between Image Datasets

Leander Girrbach; Alexander von Recum; Kirill Bykov; Zeynep Akata

cs.CVOct 28, 2026

Detecting Fine-Grained Differences between Image Datasets

Authors: Leander Girrbach, Alexander von Recum, Kirill Bykov, Zeynep Akata

Organizations: Technical University of Munich, Munich Center for Machine Learning (MCML) · Helmholtz Munich · Ludwig Maximilian University of Munich

Abstract

Detecting whether and in particular how two data distributions differ is a fundamental statistical problem. For example, we might be interested in which concepts (expressed in natural language) differ between two classes, the outputs of different generative models, or the same model with different prompt specifications. When working with large and complex image datasets, we intuitively think that first selecting concepts from a broad list using fine-grained patch embeddings, and then optimizing the concepts to better fit the data is an effective workflow. However, prior work instead uses MLLMs to propose concepts and image-level embeddings to select them. Therefore, we systematically compare these techniques and find that selecting concepts from broad lists can replace MLLM-proposals at obviously lower cost. MLLM inference can then be spent on optimizing selected concepts instead to achieve substantial improvements. Additionally, by designing a new benchmark that focuses on low-prevalence and multi-concept distinctions, we find that selecting nonredundant concepts instead of only the most distinguishing concepts matters, and we introduce techniques for how to effectively do this. Finally, we show how our methods address important problems like detecting spurious correlations and biases. Overall, our work presents a fundamental perspective shift on how we analyze interpretable distributional differences between datasets.

Figures & tables

Figure 1: How we detect fine-grained differences between datasets. Left: We are given two sets $A$ and $B$ consisting of natural images. Center: We compute multimodal patch embeddings and calculate max-over-patches cosine similarities to corresponding concept text encodings. Then, we find diverse concepts whose score distribution differs between the sets. Finally, we use an MLLM to refine concepts. Right: The method returns a list of distinguishing concepts for each set.

Figure 2: Example images for VisDiffBench (left) and MetaShift (right). The summer ( $A$ , top) and autumn ( $B$ , bottom) sets differ in a whole-image color shift, whereas the helmet ( $A$ , top) and motorcycle ( $B$ , bottom) sets differ in small objects in otherwise similar street scenes.

	VDB-Easy			VDB-Med.			VDB-Hard			MetaShift
	$D$	$Rel$	Sim.	$D$	$Rel$	Sim.	$D$	$Rel$	Sim.	$D$	$Rel$	Sim.
MLLM + SigLIP (PTR)	0.981	0.809	0.549	0.891	0.873	0.425	0.815	0.899	0.331	0.891	0.601	0.441
MLLM + FG-CLIP (PTR)	0.983	0.853	0.523	0.897	0.907	0.424	0.816	0.940	0.353	0.884	0.663	0.473
Distrib. Shift: U-WaX	0.972	0.335	0.306	0.816	0.262	0.185	0.757	0.196	0.147	0.834	0.232	0.400
Distrib. Shift: Linear	0.969	0.347	0.319	0.842	0.307	0.200	0.752	0.221	0.157	0.844	0.239	0.405
Caption Summary	0.974	0.613	0.338	0.858	0.734	0.267	0.805	0.654	0.236	0.825	0.423	0.252

Table 1: Performance comparison on VisDiffBench and MetaShift. We compare the methods across three metrics: discrimination (

D

), relevance (

Rel

), and embedding similarity (Sim). Best scores per benchmark and metric are highlighted in bold, and second-best scores are underlined.

	LVIS			MS COCO			SUN RGB-D			DOTA
	MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J
RAM++	31.5	56.9	35.9	44.7	62.8	44.1	15.1	60.5	39.7	29.4	53.6	31.4
MLLM + SigLIP (PTR)	26.4	41.9	27.1	40.2	48.5	37.3	10.1	42.6	23.5	33.8	41.7	30.2
MLLM + FG-CLIP (PTR)	22.2	39.8	23.9	27.4	40.6	25.9	15.8	42.1	27.4	31.1	40.8	27.0
+ Multi-Level Masking	27.5	43.1	27.5	42.4	49.4	38.2	20.6	47.7	37.3	32.4	40.0	27.6
+ Residual-OMP	32.8	45.8	33.2	53.5	54.2	48.5	22.9	49.9	44.2	45.5	47.0	41.1

Table 2: Evaluation on noisy and multi-target datasets. We compare concept discovery using different proposal sources (MLLM vs. COCA wordlist) and embedding models (SigLIP vs. FG-CLIP), and we progressively add our proposed Multi-Level Masking, Residual-OMP, and MLLM Optimization. Bold and underlined denote the best and second-best results, respectively.

Figure 3: Examples of spurious class-concept correlations in ImageNet. For each class $\rightarrow$ concept pair, we show the ratio of class images containing the concept (according to Qwen3.8-27B) with three representative samples.

Figure 4: Demographic bias detection example ( “Indian” ). Left: Generated images comparing base prompts against prompts with “+ Indian” across male (top) and female (bottom) subjects. Right: MLLM-judged presence rates for top discovered concepts in base (hatched) versus “+ Indian” .

Appendix figures & tables16 assets

Supplementary material from the paper’s appendix.

Appendix

	VDB-Easy			VDB-Med.			VDB-Hard			MetaShift
	$D$	$Rel$	$Rec$	$D$	$Rel$	$Rec$	$D$	$Rel$	$Rec$	$D$	$Rel$	$Rec$
MLLM + SigLIP (PTR)	0.978	0.756	0.549	0.880	0.807	0.425	0.806	0.845	0.331	0.863	0.561	0.441
Distrib. Shift: U-WaX	0.963	0.313	0.306	0.799	0.258	0.185	0.714	0.214	0.147	0.805	0.229	0.400
Distrib. Shift: Linear	0.968	0.317	0.319	0.832	0.297	0.200	0.716	0.244	0.157	0.798	0.233	0.405
Caption Summary	0.968	0.584	0.338	0.843	0.713	0.267	0.794	0.640	0.236	0.822	0.403	0.252
Image Tagging: MLLM	0.968	0.552	0.346	0.858	0.635	0.288	0.783	0.608	0.243	0.817	0.392	0.310

Appendix

Table 3: Replication of using Qwen3.8-27B as MLLM judge for

D

and

Rel

metrics.

Figure 5: MLLM ablation on MetaShift. Discrimination performance (

D

) of the Embedding Sim. + Multimodal Optim. method across different MLLMs and inference configurations.

Figure 6: Fine-grained attribute detection performance (F1 score) across six datasets.

Figure 7: Propose-then-rank effectiveness across target concept prevalences. Shaded regions indicate variance.

	Gemma4-31B				Qwen3.8-27B
Dataset	Acc	Prec	Rec	F1	Acc	Prec	Rec	F1
BDD100K	0.784	0.893	0.676	0.758	0.799	0.930	0.649	0.760
DOTA	0.778	0.930	0.619	0.721	0.788	0.948	0.610	0.730
CelebA	0.792	0.821	0.778	0.762	0.777	0.815	0.735	0.736
SUN RGB-D	0.810	0.818	0.828	0.813	0.808	0.839	0.785	0.802
MS COCO	0.904	0.968	0.836	0.890	0.879	0.988	0.768	0.855

Appendix

Table 4: MLLM binary detection accuracy, macro-averaged over classes inside each dataset. Evaluation uses up to 500 positive + 500 negative sampled images per class.

		LVIS			MS COCO			SUN RGB-D			DOTA			Avg.
		MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J
MLLM	Top- $K$	27.5	43.1	27.5	42.4	49.4	38.2	20.6	47.7	37.3	32.4	40.0	27.6	30.7	45.1	32.7
	MMR	32.7	45.7	32.9	52.9	54.0	48.2	22.5	49.5	43.6	42.5	45.7	37.6	37.7	48.7	40.6
	mRMR	32.3	45.6	32.9	51.9	53.9	47.3	22.7	49.5	42.9	40.2	44.7	34.9	36.8	48.4	39.5
	mRMR cross-level	30.8	44.9	32.3	50.4	52.4	45.5	21.2	49.0	42.3	39.5	44.7	34.7	35.5	47.8	38.7
	Forward Selection	29.2	44.0	29.3	46.1	50.2	40.7	21.2	49.1	41.9	38.3	42.7	32.6	33.7	46.5	36.1

Appendix

Table 5: Comparing Residual-OMP as redundancy-aware selection method to alternative methods.

	$k=5$			$k=10$			$k=15$			$k=20$			$k=30$
	R@ $k$	Sim.	R J	R@ $k$	Sim.	R J	R@ $k$	Sim.	R J	R@ $k$	Sim.	R J	R@ $k$	Sim.	R J
MLLM + FG-CLIP	20.3	36.1	20.9	24.1	40.8	26.0	26.7	43.7	30.6	28.7	45.9	34.0	31.7	48.9	39.3
+ Multi-Level Masking	25.6	39.7	25.4	30.7	45.1	32.7	34.1	48.4	38.5	36.6	50.8	42.7	40.6	54.3	49.8
+ Residual-OMP	30.5	42.1	30.0	38.6	49.2	41.8	42.4	52.8	49.2	44.6	54.9	53.6	47.6	57.9	60.3
+ MLLM Optimization	36.1	54.4	36.8	44.5	62.3	49.5	48.3	66.1	56.5	50.7	68.5	61.4	53.3	71.6	68.0
COCA + SigLIP	22.0	42.4	19.4	26.6	47.1	25.1	30.0	49.8	29.7	32.6	51.8	33.3	35.9	54.7	38.9

Appendix

Table 6: Evaluating methods from with varying number of predicted concepts per side. Metrics averaged across datasets.

	LVIS			MS COCO			SUN RGB-D			DOTA
	MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J	MCC-R	Sim.	R J
MLLM + Optimization	39.6	61.0	42.9	68.0	75.7	66.4	24.3	58.0	46.9	46.3	55.2	43.7
+ unexplained images	39.4	60.4	42.5	67.6	75.2	65.3	24.2	57.8	47.1	46.2	54.9	43.7
COCA + Optimization	40.4	62.1	43.1	66.7	74.3	63.1	25.7	62.2	49.7	49.7	63.0	52.1
+ unexplained images	39.8	61.6	42.5	65.7	74.0	62.0	26.0	62.2	49.7	48.4	63.3	51.7

Appendix

Table 7: Comparison of MLLM-based concept optimization. In the standard setting, we randomly sample images. In the evaluated variant, we specifically select concepts that are not explained by the current concept selection.

LVIS

MS COCO

SUN RGB-D

DOTA

MCC-R

Sim.

R J

MCC-R

Sim.

R J

MCC-R

Sim.

R J

MCC-R

Sim.

R J

MLLM + FG-CLIP + Multi-Level Masking

Published

(10,50,100)

27.5

43.1

27.5

42.4

49.4

38.2

20.6

47.7

37.3

32.4

40.0

27.6

3 levels, sampled

27.4

{}_{27.0\text{--}28.1}

43.1

{}_{42.8\text{--}43.3}

27.7

{}_{26.9\text{--}28.3}

42.1

{}_{41.4\text{--}42.7}

49.3

{}_{49.0\text{--}49.8}

38.2

{}_{37.4\text{--}39.2}

20.6

{}_{20.0\text{--}21.1}

47.6

{}_{47.3\text{--}47.8}

36.7

{}_{36.2\text{--}37.3}

32.5

{}_{31.7\text{--}34.1}

40.3

{}_{40.1\text{--}40.6}

28.0

{}_{27.2\text{--}29.4}

4 levels, sampled

27.3

{}_{26.8\text{--}28.0}

43.0

{}_{42.8\text{--}43.2}

27.6

{}_{27.0\text{--}27.8}

42.1

{}_{41.4\text{--}42.5}

49.3

{}_{49.0\text{--}49.5}

38.0

{}_{37.4\text{--}38.5}

20.5

{}_{19.8\text{--}21.2}

47.6

{}_{47.4\text{--}47.9}

36.8

{}_{36.2\text{--}37.5}

32.4

{}_{31.7\text{--}33.8}

40.1

{}_{39.8\text{--}40.3}

27.9

{}_{27.0\text{--}29.7}

COCA + FG-CLIP + Multi-Level Masking

Appendix

Table 8: Sensitivity to the masking levels. Multi-Level Masking with the published per-image top-

K

levels

(10,50,100)

against 20 randomly sampled level sets per scheme. Sampled rows report the mean over the 20 configurations, with the range (min-max) as a subscript. All other settings (FG-CLIP, Top-

K

selection, 10 concepts per side) are unchanged.

Figure 8: Examples of discovered class–concept correlations in ImageNet. Each row displays representative images for a class alongside the correlated concept and its MLLM-verified presence score (in parentheses).

Panel of Figure 9 — Figure 9: Examples of detected biased attributes across country/religion/race demographic groups across Flux and SD-3.5-Large T2I models.

Figure 10: Example visual input grids provided to the MLLM during optimization. Each grid displays sample images from Set $A$ (top row, blue) and Set $B$ (bottom row, orange) for tasks from VisDiffBench (left) and MetaShift (right). Ground-truth descriptions above each grid are shown for reference only and are not included in the model’s input.

Figure 11: Sensitivity analysis over the MCC-match threshold

\tau

. MCC-Recall (%) across varying recovery thresholds

\tau\in[0.2,0.8]

for different concept discovery and scoring methods compared against oracle baselines. The vertical dotted line denotes the default threshold

\tau=0.5

used in the main experiments.

Figure 12: LLM Judge Recall

R_{J}

evaluated with Qwen3.8-27B and gpt-oss-120b. Scores are averaged over datasets. Error bars show 95% CIs.

	LVIS				MS COCO				SUN RGB-D				DOTA
	MCC-R	$D$	$D_{\mathrm{c}}$	$D_{\mathrm{b}}$	MCC-R	$D$	$D_{\mathrm{c}}$	$D_{\mathrm{b}}$	MCC-R	$D$	$D_{\mathrm{c}}$	$D_{\mathrm{b}}$	MCC-R	$D$	$D_{\mathrm{c}}$	$D_{\mathrm{b}}$
RAM++	31.5	57.8	72.8	72.6	44.7	60.4	76.9	76.4	15.1	60.5	68.4	68.2	29.4	66.2	75.9	75.0
MLLM + SigLIP	26.4	54.7	80.6	68.8	40.2	57.3	87.6	73.0	10.1	58.7	74.8	66.8	33.8	70.9	87.4	74.6
MLLM + FG-CLIP, no masking	22.2	56.5	78.6	70.3	27.4	58.8	84.1	73.4	15.8	61.5	72.8	66.5	31.1	69.3	83.3	74.1
MLLM + FG-CLIP	27.5	54.9	80.5	72.2	42.4	57.5	87.8	77.6	20.6	60.8	75.2	69.9	32.4	68.8	83.3	74.3
+ Residual-OMP	32.8	56.2	82.9	78.1	53.5	58.9	90.7	84.5	22.9	61.7	77.3	73.2	45.5	70.9	86.5	82.4

Appendix

Table 9: Comparison of MCC-Recall and alternative discrimination metrics across datasets. We report Set-Level Discrimination (

D

), Concept-Level Discrimination (

D_{\mathrm{c}}

), and Concept-Level Binarized Discrimination (

D_{\mathrm{b}}

) on LVIS, MS COCO, SUN RGB-D, and DOTA. Bold and underlined values denote the best and second-best performing methods.

Figure 13: Distribution of LLM-assigned filtering scores. Histograms show the distributions of integer scores (1 to 10) across candidate concepts and class-concept pairs. Note the logarithmic count scale. Left: Visual concreteness scores for concepts (Gemma4-31B). Center: Plausibility scores for class-concept pairs (Qwen3.8-27B). Right: Interestingness scores for class-concept pairs.

Distribution Shift Detection

Distribution shift detection identifies when data or model behavior changes between reference and deployment settings.

See all 63 papers

LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images
Apr 28, 2026James Flora, Kowshik Thopalli, Akshay R. Kulkarni +2Distribution Shift Distribution Shift Detection
School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA · Lawrence Livermore National Laboratory, Livermore, California, USA · University of California San Diego, California, USA
VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification
Sep 5, 2026Yixin Wan, Tianle Zheng, Kai-Wei ChangVLM Evaluation Vision-Language Models
Department of Computer Science University of California, Los Angeles
Concept-wise Attention for Fine-grained Concept Bottleneck Models
Apr 17, 2026Minghong Zhong, Guoshuai Zou, Kanghao Chen +2Concept Bottleneck Models Vision-Language Alignment
Sun Yat-sen University · Peng Cheng Laboratory · Key Laboratory of Machine Intelligence and Advanced Computing, MOE +1

Apr 28, 2026J/K move · Enter open · S save

LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images

James Flora, Kowshik Thopalli, Akshay R. Kulkarni, Weng-Keen Wong+1

School of Electrical Engineering and Computer Science, Oregon State University, Corvallis, Oregon, USA · Lawrence Livermore National Laboratory, Livermore, California, USA · University of California San Diego, California, USA

Distribution Shift Distribution Shift Detection Density Ratio Estimation

Open paper AskPDF arXiv

We present LatentDiff, a scalable framework for semantic dataset comparison that operates directly in the latent space of pretrained vision encoders. By combining sparse autoencoder-based divergence testing with density ratio estimation, LatentDiff identifies interpretable semantic differences between datasets at a fraction of the computational cost of caption-based alternatives. We also introduce Noisy-Diff, a benchmark capturing realistic sparse distribution shifts that cause existing methods to struggle. Experiments demonstrate that LatentDiff achieves superior accuracy while remaining robust to settings where an extremely small fraction of images (from 5% to <1% ) differ semantically.

Abstract

Distribution Shift Detection

Explore similar work

LatentDiff: Scaling Semantic Dataset Comparison to Millions of Images

VDiff-Bench: A Challenging Benchmark for Fine-Grained Image Difference Identification

Concept-wise Attention for Fine-grained Concept Bottleneck Models