Coverage-aware Semantic Representation Learning for Underrepresented Visual Concepts

Sakib Ahammed; Xia Cui; Wenqi Lu; Xinqi Fan; Bill Cassidy; Xueli Chen; Moi Hoon Yap

cs.CVFeb 18, 2026

Coverage-aware Semantic Representation Learning for Underrepresented Visual Concepts

Authors: Sakib Ahammed, Xia Cui, Wenqi Lu, Xinqi Fan, Bill Cassidy, Xueli Chen, Moi Hoon Yap

Organizations: Department of Computing and Mathematics, Manchester Metropolitan University The Dalton Building, Chester Street, M1 5GD Manchester

Abstract

Modern vision models increasingly rely on rich semantic representations that extend beyond class labels to include descriptive concepts, attributes, and contextual cues. However, semantic concepts are not uniformly represented across classes: a concept may be frequent globally yet remain underrepresented within a class, resulting in low class-concept coverage. We formalize this class-concept phenomenon as Semantic Coverage Imbalance (SCI) and study its relationship with learned semantic representations. We find that lower class-concept coverage is consistently associated with weaker semantic representations. To address this, we introduce Coverage-aware Semantic Representation Learning, which shares more semantic structure for low coverage or uncertain relations while preserving more pair-specific structure for high coverage relations. We instantiate this principle in SemCovNet with Coverage-Calibrated Semantic Sharing (CCSS), a coverage- and uncertainty-aware partial-pooling mechanism. Using a range of vision datasets spanning facial attributes and fine-grained recognition, as well as real-world medical datasets, we show that SCI is widespread, degrades representation quality, and can be mitigated through coverage-aware semantic representation learning.

Figures & tables

Figure 1: Motivation and principle of coverage-aware semantic representation learning. (a) Semantic Coverage Imbalance ( SCI ) occurs when class–concept pair coverage is highly uneven, even when class and concept marginals appear adequate. (b) We show that low coverage weakens semantic representations. (c) Coverage-aware semantic representation learning adapts more pair-specific structure for high coverage pairs and shares more structure for low coverage or uncertain pairs.

Figure 2: Intuitive view of Semantic Coverage Imbalance ( SCI ). (a) Each class–concept pair $(y,k)$ defines a relation in the semantic space ${\mathcal{S}}$ . (b) In the observation space ${\mathcal{X}}$ , the training data grounds each semantic relation as ${\mathcal{G}}:(y,k)\mapsto\mathcal{D}_{y,k}$ , producing high- and low coverage pairs. (c) We hypothesize that low coverage leads to weaker learned semantic representations. We refer to the phenomenon of uneven class–concept pair coverage as SCI .

Figure 3: SemCovNet for coverage-aware semantic representation learning. A Concept Query Module maps dense features to image-driven concept tokens. For each class–concept pair, CCSS combines the empirical class–concept representation, a shared semantic structure, and calibration signals. low coverage or uncertain pairs shrink toward the shared structure, while high coverage pairs retain more pair-specific structure. The resulting target supervises the concept tokens, and a residual pathway preserves complementary visual information for task prediction.

Figure 4: SCI across visual domains. Ranked training-set class–concept coverage $C_{y,k}$ for CelebA, CUB, and Derm7pt. Coverage is highly uneven: 26.9%, 67.2%, and 33.9% of class–concept pairs have $C_{y,k}<0.05$ in CelebA, CUB, and Derm7pt, respectively.

Figure 5: Lower coverage is associated with weaker semantic representations. (a) Frozen linear-probe class-conditioned concept Macro-F1 from the learned concept tokens across Tail, Middle, and Head class–concept pairs for DINOv3–ERM and concept supervised learning. Error bars show 95% CIs over class–concept pairs. (b–d) Concept supervised representations projected using a common PCA basis fitted separately for each dataset. Dispersion varies across coverage groups and datasets; $+$ denotes class–concept centroids.

Variant	Concept supervised	CCSS	Adaptation	CUB			CelebA
Variant	Concept supervised	CCSS	Adaptation	Task $\uparrow$	Tail $\uparrow$	$\Delta_{\mathrm{H-T}}$ $\downarrow$	Task $\uparrow$	Tail $\uparrow$	$\Delta_{\mathrm{H-T}}$ $\downarrow$
ERM	$\times$	$\times$	Frozen	0.886 $\pm$ 0.006	N/A	N/A	0.953 $\pm$ 0.001	N/A	N/A
Concept	$\checkmark$	$\times$	Frozen	0.881 $\pm$ 0.007	0.003 $\pm$ 0.002	0.171 $\pm$ 0.032	0.954 $\pm$ 0.004	0.441 $\pm$ 0.102	0.426 $\pm$ 0.091
Concept $+$ CCSS (Ours)	$\checkmark$	$\checkmark$	Frozen	0.888 $\pm$ 0.002	0.005 $\pm$ 0.001	0.145 $\pm$ 0.011	0.956 $\pm$ 0.002	0.463 $\pm$ 0.006	0.406 $\pm$ 0.008
$\Delta_{\mathrm{Tail}}$ (Concept+CCSS $-$ Concept), Frozen [95% CI]				0.002 [0.001, 0.002]			0.022 [0.015, 0.060]
Concept	$\checkmark$	$\times$	LoRA	0.886 $\pm$ 0.003	0.004 $\pm$ 0.002	0.188 $\pm$ 0.026	0.949 $\pm$ 0.006	0.411 $\pm$ 0.072	0.448 $\pm$ 0.049

Table 1: Controlled CCSS model ladder on CUB and CelebA. The ladder isolates the effect of CCSS under frozen and LoRA-adapted DINOv3 encoders (last three layers). Concept denotes standard semantic supervision without CCSS. Tail reports class-conditioned concept Macro-F1 and

\Delta_{\mathrm{H-T}}

the Head–Tail representation gap. Highlighted rows show the paired Tail gain from adding CCSS under the same adaptation setting, with 95% CIs.

Method	CelebA	CUB	Derm7pt	MILK10k
JTT	0.887 $\pm$ 0.004	0.890 $\pm$ 0.004	0.852 $\pm$ 0.008	0.879 $\pm$ 0.019
DFR	0.933 $\pm$ 0.001	0.837 $\pm$ 0.002	0.866 $\pm$ 0.011	0.859 $\pm$ 0.017
MORE	0.925 $\pm$ 0.005	0.892 $\pm$ 0.001	0.854 $\pm$ 0.009	0.879 $\pm$ 0.013
TS-MOF	0.907 $\pm$ 0.013	0.885 $\pm$ 0.001	0.858 $\pm$ 0.003	0.878 $\pm$ 0.014
IVQ-CBM	0.932 $\pm$ 0.002	0.881 $\pm$ 0.008	0.870 $\pm$ 0.005	0.861 $\pm$ 0.015
SemCovNet (Ours)	0.955 $\pm$ 0.001	0.891 $\pm$ 0.002	0.851 $\pm$ 0.018	0.887 $\pm$ 0.009

Table 2: Task-level comparison with existing methods. We report the standard task metric for each dataset: CelebA BAcc, CUB Top-1 accuracy, and AUROC for Derm7pt and MILK10k. Higher is better.

Figure 6: Controlled class–concept coverage intervention on CUB. (a) Frozen-probe representation quality as training evidence for target class–concept pairs is reduced from $100\%$ to $0\%$ ; “one” retains one positive example. (b) Paired zero-coverage effect $\Delta_{0}$ positive values indicate higher representation quality for SemCovNet . Error bars show paired 95% CIs over target pairs. At zero coverage, SemCovNet retains higher representation quality than Coverage reweight and Semantic GroupDRO.

Figure 7: Coverage-aware learning under clinical semantic supervision. (a) On Derm7pt expert-defined clinical concepts, SemCovNet improves Tail representation quality and reduces the Head–Tail gap over Concept supervision. (b) On MILK10k continuous soft semantic supervision, Tail representation quality remains stable as semantic evidence becomes less informative. Error bars show 95% CIs.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Configuration
Visual encoder	DINOv3 ViT-B/16
Input resolution	224 $\times$ 224
Encoder adaptation	Frozen / LoRA on final 3 transformer blocks
LoRA rank $r$	8
LoRA scaling $\alpha$	16
LoRA dropout	0.05

Appendix

Table 3: Implementation details. Methods use the same backbone, data splits, optimization settings, and random seeds.

Dataset	#train	Class	Concept	Pairs	$Q_{25}$	Median	$Q_{75}$	Near-zero (%)	Zero (%)
CelebA	162770	2	39	78	0.048	0.143	0.338	26.9	0.0
CUB	5394	200	312	62,400	0.000	0.000	0.111	67.2	53.9
Derm7pt	413	2	28	56	0.031	0.117	0.455	33.9	12.5
Waterbirds	4795	2	2	4	0.050	0.500	0.950	25.0	0.0
MILK10k	3668	11	7	77	0.253	0.368	0.433	0.0	0.0

Appendix

Table 4: Dataset-level characterization of Semantic Coverage Imbalance. Coverage

C_{y,k}

is computed per class–concept pair on the training split only. Near-zero and Zero denote the fraction of pairs with

C_{y,k}<0.05

and

C_{y,k}=0

, respectively.

Dataset	Significant heterogeneity (%)	$\rho_{\mathrm{bal}}$
CelebA	97.4	1.000
CUB	96.5	0.996
Derm7pt	64.3	0.993
Waterbirds	100.0	0.800
MILK10k	85.7	—

Appendix

Table 5: Statistical controls for SCI . Significant heterogeneity is the share of concepts whose across-class coverage variation exceeds a prevalence-preserving permutation null after BH–FDR correction (

q<0.05

\rho_{\mathrm{bal}}

is the Spearman correlation between full-data and class-balanced coverage estimates; “—” denotes not applicable.

Method	$100\%$	$75\%$	$50\%$	$25\%$	One	$0\%$	$\Delta_{100\%\!\rightarrow 0\%}$
Concept supervised	0.675 [0.630, 0.717]	0.666 [0.619, 0.708]	0.661 [0.617, 0.703]	0.665 [0.619, 0.709]	0.661 [0.617, 0.704]	0.665 [0.619, 0.708]	0.010
Coverage reweighting	0.650 [0.608, 0.691]	0.657 [0.613, 0.698]	0.659 [0.618, 0.699]	0.661 [0.617, 0.702]	0.661 [0.620, 0.701]	0.661 [0.623, 0.697]	-0.012
Semantic GroupDRO	0.655 [0.614, 0.696]	0.658 [0.617, 0.698]	0.649 [0.605, 0.689]	0.659 [0.618, 0.698]	0.665 [0.621, 0.707]	0.660 [0.617, 0.698]	-0.005
LoRA Concept	0.664 [0.623, 0.704]	0.658 [0.618, 0.697]	0.665 [0.628, 0.702]	0.669 [0.628, 0.709]	0.661 [0.620, 0.703]	0.670 [0.629, 0.710]	-0.006
SemCovNet (Ours)	0.679 [0.631, 0.724]	0.672 [0.623, 0.717]	0.677 [0.629, 0.722]	0.676 [0.627, 0.721]	0.678 [0.631, 0.721]	0.682 [0.634, 0.726]	-0.003

Appendix

Table 6: Full coverage-response intervention on CUB. We report frozen-probe representation quality

q_{\mathrm{repr}}

for each method as target-pair training evidence decreases from

100\%

to zero. “One” retains a single positive example and therefore has no nominal retention fraction. Only the target pair’s own evidence is intervened on; the corresponding class and concept remain observable through other training pairs. We report concept Macro-F1 as the mean over the target pairs with

95\%

CIs. We average 5 seeds within each pair before bootstrapping. The final column reports the

\Delta_{100\%\!\rightarrow 0\%}

, difference between 100% and 0% change. Smaller degradation indicates greater robustness to loss of pair-specific evidence; positive values indicate degradation, while values near zero indicate stability.

Method	One example	Zero evidence
No class sharing	0.678 [0.627,0.724]	0.678 [0.628,0.722]
Shared prior only	0.667 [0.625,0.708]	0.667 [0.624,0.709]
Fixed specialization	0.677 [0.627,0.723]	0.684 [0.635,0.728]
CCSS (Ours)	0.678 [0.631,0.721]	0.682 [0.634,0.726]

Appendix

Table 7: Representation-sharing ablations under extreme semantic under-coverage. We evaluate structural variants in the one-example and zero-evidence conditions, where borrowing shared semantic structure matters most. We designed these variants as low-coverage mechanism ablations and therefore did not evaluate them over the full retention trajectory. We report results as means over the target class–concept pairs with 95% CIs. We average 5 seeds within each pair before bootstrapping.

Method	Task			Concept			Coverage Slope $\beta_{1}$ $\downarrow$
Method	AUROC $\uparrow$	M-F1 $\uparrow$	BAcc $\uparrow$	F1 $\uparrow$	Tail AUROC $\uparrow$	$\Delta_{\mathrm{H-T}}$ $\downarrow$	Coverage Slope $\beta_{1}$ $\downarrow$
Concept supervised	0.848 $\pm$ 0.007	0.738 $\pm$ 0.007	0.717 $\pm$ 0.011	0.485 $\pm$ 0.016	0.647 $\pm$ 0.049	0.085 $\pm$ 0.049	0.115 $\pm$ 0.030
LoRA Concept	0.858 $\pm$ 0.005	0.751 $\pm$ 0.012	0.729 $\pm$ 0.014	0.419 $\pm$ 0.196	0.615 $\pm$ 0.021	0.110 $\pm$ 0.021	0.071 $\pm$ 0.020
Semantic only	0.867 $\pm$ 0.006	0.761 $\pm$ 0.008	0.745 $\pm$ 0.005	0.404 $\pm$ 0.137	0.683 $\pm$ 0.055	0.038 $\pm$ 0.053	0.101 $\pm$ 0.027
Semantic + residual	0.855 $\pm$ 0.008	0.762 $\pm$ 0.007	0.743 $\pm$ 0.008	0.490 $\pm$ 0.010	0.655 $\pm$ 0.057	0.063 $\pm$ 0.046	0.092 $\pm$ 0.024
Frozen SemCovNet	0.852 $\pm$ 0.007	0.756 $\pm$ 0.008	0.736 $\pm$ 0.007	0.365 $\pm$ 0.130	0.673 $\pm$ 0.061	0.054 $\pm$ 0.062	0.093 $\pm$ 0.038

Appendix

Table 8: Expert clinical semantic supervision on Derm7pt. Task metrics evaluate diagnosis, while concept metrics evaluate the learned semantic representation. Tail AUROC and the

\Delta_{\mathrm{H-T}}

Head–Tail gap use coverage strata defined from the training split; a smaller positive gap and smaller coverage slope

\beta_{1}

indicate less dependence of representation quality on semantic coverage. Results are reported as mean

\pm

standard deviation over five seeds.

$\eta$	Method	Task AUROC $\uparrow$	Tail Concept F1 $\uparrow$	Soft Spearman $\uparrow$
0.0	Concept supervised	0.877 $\pm$ 0.017	0.785 $\pm$ 0.004	0.655 $\pm$ 0.114
0.0	SemCovNet	0.887 $\pm$ 0.018	0.781 $\pm$ 0.021	0.637 $\pm$ 0.024
0.1	Concept supervised	0.879 $\pm$ 0.022	0.789 $\pm$ 0.001	0.643 $\pm$ 0.115
0.1	SemCovNet	0.887 $\pm$ 0.019	0.786 $\pm$ 0.027	0.624 $\pm$ 0.022
0.2	Concept supervised	0.879 $\pm$ 0.021	0.790 $\pm$ 0.006	0.626 $\pm$ 0.111
0.2	SemCovNet	0.888 $\pm$ 0.019	0.786 $\pm$ 0.025	0.604 $\pm$ 0.016

Appendix

Table 9: Soft semantic supervision on MILK10k. We perturb the continuous training evidence to make the semantic target less informative. Coverage strata are fixed from the unperturbed training evidence. Tail Concept F1 measures representation quality for under-covered semantics, Task AUROC measures clinical discrimination, and Soft Spearman measures agreement with the continuous semantic teacher.

Method	Avg. Acc. $\uparrow$	WGA $\uparrow$	Avg–WGA $\downarrow$
Concept supervised	0.920 $\pm$ 0.016	0.816 $\pm$ 0.026	0.103 $\pm$ 0.017
JTT	0.917 $\pm$ 0.010	0.822 $\pm$ 0.020	0.095 $\pm$ 0.015
MORE	0.930 $\pm$ 0.003	0.819 $\pm$ 0.036	0.112 $\pm$ 0.039
IVQ-CBM	0.910 $\pm$ 0.025	0.776 $\pm$ 0.021	0.134 $\pm$ 0.026
GroupDRO	0.939 $\pm$ 0.003	0.882 $\pm$ 0.013	0.057 $\pm$ 0.013
Hierarchical-DRO	0.944 $\pm$ 0.005	0.891 $\pm$ 0.011	0.052 $\pm$ 0.010

Appendix

Table 10: Group robustness on WaterBirds. WGA is the minimum accuracy across the four target

\times

background groups, and Avg–WGA measures the gap between average and worst-group accuracy.

Method	Land/Land	Land/Water	Water/Land	Water/Water
Concept supervised	0.997 $\pm$ 0.001	0.858 $\pm$ 0.036	0.816 $\pm$ 0.026	0.968 $\pm$ 0.005
JTT	0.996 $\pm$ 0.001	0.861 $\pm$ 0.024	0.824 $\pm$ 0.026	0.973 $\pm$ 0.007
MORE	0.997 $\pm$ 0.002	0.882 $\pm$ 0.021	0.823 $\pm$ 0.044	0.975 $\pm$ 0.009
IVQ-CBM	0.998 $\pm$ 0.001	0.871 $\pm$ 0.022	0.783 $\pm$ 0.020	0.956 $\pm$ 0.003
GroupDRO	0.994 $\pm$ 0.002	0.894 $\pm$ 0.010	0.883 $\pm$ 0.014	0.955 $\pm$ 0.005
Hierarchical-DRO	0.994 $\pm$ 0.001	0.905 $\pm$ 0.012	0.891 $\pm$ 0.011	0.956 $\pm$ 0.004

Appendix

Table 11: Per-group accuracy on WaterBirds. Groups correspond to the four combinations of bird class and background. Dedicated group-robustness methods particularly improve the conflicting landbird-on-water and waterbird-on-land groups.

Variant	Concept	Shared	Pair spec.	Calibration / adaptation	Tail F1 $\uparrow$	$\Delta_{\mathrm{H-T}}$ $\downarrow$
ERM	–	–	–	–	0.528 $\pm$ 0.000	0.019 $\pm$ 0.000
Concept	✓	–	–	–	0.550 $\pm$ 0.039	-0.005 $\pm$ 0.041
Shared	✓	✓	–	–	0.577 $\pm$ 0.037	-0.032 $\pm$ 0.037
Fixed-spec	✓	✓	✓	Fixed	0.571 $\pm$ 0.029	-0.020 $\pm$ 0.030
CCSS frozen	✓	✓	✓	Statistical / frozen	0.577 $\pm$ 0.039	-0.028 $\pm$ 0.040
SemCovNet (Ours)	✓	✓	✓	Statistical / LoRA-last3	0.619 $\pm$ 0.054	-0.069 $\pm$ 0.055

Appendix

Table 12: Component decomposition of coverage-aware learning on CUB. The ladder adds concept supervision, shared semantic structure, pair-specific specialization, coverage calibration, and DINOv3 adaptation.

$\alpha$ rule	Tail F1 $\uparrow$	$\Delta_{\mathrm{H-T}}\downarrow$
Frequency	0.608 $\pm$ 0.057	-0.062 $\pm$ 0.058
Coverage	0.617 $\pm$ 0.042	-0.072 $\pm$ 0.043
Effective support	0.625 $\pm$ 0.028	-0.079 $\pm$ 0.029
$n+\sigma^{2}$	0.617 $\pm$ 0.042	-0.069 $\pm$ 0.044
CCSS (Ours)	0.634 $\pm$ 0.057	-0.087 $\pm$ 0.060

Appendix

Table 13: Component ablation of coverage-aware learning on CUB. Each row introduces one additional component, from concept supervision and shared semantic structure to pair-specific specialization, coverage calibration, and DINOv3 adaptation. We report results as mean

\pm

standard deviation across five seeds.

Shared prior	Method	Tail F1 $\uparrow$	$\Delta_{\mathrm{H-T}}\downarrow$
Concept	$\tilde{{\bm{\mu}}}_{k}$	0.579 $\pm$ 0.008	-0.032 $\pm$ 0.002
Class + concept	$\tilde{{\bm{\mu}}}_{y}+\tilde{{\bm{\mu}}}_{k}$	0.598 $\pm$ 0.004	-0.047 $\pm$ 0.003
Global + class + concept (Ours)	${\bm{\mu}}_{0}+\tilde{{\bm{\mu}}}_{y}+\tilde{{\bm{\mu}}}_{k}$	0.608 $\pm$ 0.007	-0.057 $\pm$ 0.005

Appendix

Table 14: Shared-prior decomposition on CUB. Effect of shared semantic structure on Tail, Middle, and Head representation quality.

$\lambda/\lambda^{*}$	$\lambda_{\mathrm{CCSS}}$	Tail F1 $\uparrow$	H–T Gap $\downarrow$	$\Delta$ vs. $\lambda^{*}$ [95% CI]
$0.25\times$	0.0625	0.580 $\pm$ 0.056	-0.033 $\pm$ 0.056	-0.006 [-0.020, +0.006]
$0.5\times$	0.1250	0.606 $\pm$ 0.056	-0.058 $\pm$ 0.055	-0.002 [-0.011, +0.007]
$1\times$	0.2500	0.599 $\pm$ 0.071	-0.047 $\pm$ 0.072	–
$2\times$	0.5000	0.617 $\pm$ 0.068	-0.065 $\pm$ 0.069	+0.008 [-0.001, +0.015]
$4\times$	1.0000	0.608 $\pm$ 0.057	-0.056 $\pm$ 0.056	+0.008 [-0.007, +0.021]

Appendix

Table 15: Sensitivity to

\lambda_{\mathrm{CCSS}}

on CUB. We vary the CCSS weight around the main setting

\lambda_{\mathrm{CCSS}}^{*}=0.25

\Delta

reports the paired change in Tail Macro-F1 relative to

\lambda_{\mathrm{CCSS}}^{*}

with 95% CIs.

Explore similar work

Jun 12, 2026cs.CV

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning

Current representation learning paradigms force a fundamental compromise: self-supervised methods scale to massive datasets but yield opaque features, whereas interpretable models remain bottlenecked by the need for dense human annotation. We introduce Self-Supervised Concept discOvery via Preference lEarning (\model), a label-free framework that resolves this dilemma. Instead of treating Vision-Large-Language Models (VLLMs) as static feature extractors, \model leverages them as active participants in a self-supervised preference optimization loop. By autonomously hypothesizing, validating, and reinforcing candidate visual attributes directly from raw imagery, our framework discovers novel, structured concepts without a single label. Extensive experiments across natural, medical, and physics domains demonstrate that \model successfully extracts domain-specific concepts where standard VLLMs often fail to generate. By amortizing concept discovery directly into the VLLM backbone through our self-supervised preference objective -- rather than relying on static generation and disjoint filtering -- we achieve up to a 24-point absolute improvement in downstream top-1 classification accuracy on unseen data. Our work suggest that interpretability can emerge through a model's autonomous interaction with incidental visual structures, without any human supervision.

Shilong Xiang, Zirui Zhang, Chengzhi Mao

Rutgers University

Unsupervised Preference Learning Label-Free

Ask PDF

May 10, 2026cs.LG

SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations

Latent representations learned by neural networks often exhibit semantic structure, where concept similarity is reflected by geometric proximity in embedding space. However, comparing such spaces across models remains difficult: changes in architecture, pretraining data, objective, or random seed can yield embeddings with similar content but incompatible geometry. This latent space alignment problem is central to interpretability, transfer and multimodal learning, federated systems, and semantic communication; however, progress remains limited by the lack of large-scale, model-diverse, and metadata-rich benchmarks. To address this gap, we introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models across eight standard image-classification benchmarks. SEMASIA pairs embeddings with structured metadata describing architectures, training regimes, pretraining sources, and model scale. We demonstrate three applications of the resource. First, we analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering and hierarchical semantic neighborhoods across models and datasets. Second, we benchmark supervised alignment mappings between latent spaces using reconstruction error and downstream task performance. Third, we perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings. By coupling representational scale with standardized metadata, SEMASIA provides a reproducible foundation for studying latent geometry, evaluating alignment methods, and developing next-generation heterogeneous and interoperable AI systems.

Mario Edoardo Pandolfo, Enrico Grimaldi, Lorenzo Marinucci +4

Dept. Computer, Control and Management Engineering, Sapienza University of Rome, Rome, Italy. · National Inter-University Consortium for Telecommunications (CNIT), Parma, Italy. · Dept. of Statistical Sciences, Sapienza University of Rome, Rome, Italy. +1

Latent Representation Alignment Representational Capacity Transfer Learning Mediaref

Ask PDF

Oct 7, 2026cs.CV

DisParQ: Self-Supervised Part Concepts for Interpretable Vision Foundation Models

Concept-based vision models represent images through an intermediate layer of human-inspectable concepts, so what a model relies on can be traced to those concepts. However, those models are often limited to fixed categories or depend on language to define their concepts. We introduce DisParQ (Discrete Parts with Quantized attributes), a method that learns spatially grounded, discrete concept representations from a powerful frozen vision-only self-supervised backbone. It requires no class labels and no language supervision. Each image patch is assigned to exactly one concept from a learnable prototype dictionary, and only a sparse subset of concepts may activate per image. To capture how each concept varies across images (e.g., the type of a "wheel"), we learn continuous residuals alongside the concepts and then quantize them into discrete attributes. A spatial decoder reconstructs the backbone's representation from the concepts and attributes alone, so successful reconstruction means that the discrete representation preserves the backbone's information. We evaluate DisParQ across seven datasets, from general recognition (ImageNet, PartImageNet, Places) to fine-grained benchmarks (CUB, Cars, Dogs, Flowers). We show that DisParQ closely matches its frozen DINOv2 teacher on ImageNet linear probing (83.2% top-1), achieves higher concept consistency than language-aligned models, remains competitive on fine-grained recognition, and enables cross-category part-based retrieval.

Adam Pardyl, Siddhartha Gairola, Sukrut Rao +4

Jagiellonian University, Faculty of Mathematics and Computer Science · Jagiellonian University, Doctoral School of Exact and Natural Sciences · Max Planck Institute for Informatics, Saarland Informatics Campus +2

Topic pending

Ask PDF

Abstract

Explore similar work

S2^22COPE: Self-Supervised Concept Discovery via Preference Learning

SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations

DisParQ: Self-Supervised Part Concepts for Interpretable Vision Foundation Models

S $^2$ COPE: Self-Supervised Concept Discovery via Preference Learning