This Looks Distinctly Like That: Grounding Interpretable Recognition in Stiefel Geometry against Neural Collapse

Junhao Jia; Jiaqi Wang; Yunyou Liu; Haodong Jing; Yueyi Wu; Xian Wu; Yefeng Zheng

cs.CVMar 9, 2026

This Looks Distinctly Like That: Grounding Interpretable Recognition in Stiefel Geometry against Neural Collapse

Authors: Junhao Jia, Jiaqi Wang, Yunyou Liu, Haodong Jing, Yueyi Wu, Xian Wu, Yefeng Zheng

Organizations: Medical Artificial Intelligence Lab, Westlake University, Hangzhou, China · Hangzhou Dianzi University, Hangzhou, China · Tencent Jarvis Lab, Shenzhen, China · Xi’an Jiaotong University, Xi’an, China

Abstract

Prototype networks provide an intrinsic case based explanation mechanism, but their interpretability is often undermined by prototype collapse, where multiple prototypes degenerate to highly redundant evidence. We attribute this failure mode to the terminal dynamics of Neural Collapse, where cross entropy optimization suppresses intra class variance and drives class conditional features toward a low dimensional limit. To mitigate this, we propose Adaptive Manifold Prototypes (AMP), a framework that leverages Riemannian optimization on the Stiefel manifold to represent class prototypes as orthonormal bases and make rank one prototype collapse infeasible by construction. AMP further learns class specific effective rank via a proximal gradient update on a nonnegative capacity vector, and introduces spatial regularizers that reduce rotational ambiguity and encourage localized, non overlapping part evidence. Extensive experiments on fine-grained benchmarks demonstrate that AMP achieves state-of-the-art classification accuracy while significantly improving causal faithfulness over prior interpretable models.

Figures & tables

Figure 1: Motivation of Adaptive Manifold Prototypes (AMP). (a) Contrast between diverse and redundant prototypes. (b) Loss regularizations yield soft constraints. (c) AMP imposes a geometric constraint on the Stiefel manifold, ensuring part diversity.

Figure 2: The overview of our proposed AMP framework. (a) Training: learn Stiefel bases and capacity matrices via Riemannian gradients. (b) Inference: project features onto class manifolds, then aggregate activations to class scores for interpretable predictions..

Method	CUB-200-2011				Stanford Cars
Method	V16	R34	R50 †	D161	V16	R34	R50	D161
Baseline	76.5 $\pm$ 0.8	80.0 $\pm$ 0.7	85.1 $\pm$ 0.5	84.5 $\pm$ 0.4	85.1 $\pm$ 0.4	86.7 $\pm$ 0.4	88.0 $\pm$ 0.3	89.0 $\pm$ 0.3
Black-box Classifiers
RA-CNN	83.1 $\pm$ 0.6	84.9 $\pm$ 0.5	88.7 $\pm$ 0.4	88.0 $\pm$ 0.3	91.4 $\pm$ 0.3	92.2 $\pm$ 0.3	92.9 $\pm$ 0.2	93.4 $\pm$ 0.2
NTS-Net	82.4 $\pm$ 0.6	84.1 $\pm$ 0.5	87.9 $\pm$ 0.4	87.2 $\pm$ 0.3	90.8 $\pm$ 0.3	91.7 $\pm$ 0.3	92.4 $\pm$ 0.2	92.9 $\pm$ 0.2
PMG	84.7 $\pm$ 0.5	85.8 $\pm$ 0.5	89.2 $\pm$ 0.3	88.5 $\pm$ 0.3	92.0 $\pm$ 0.3	92.9 $\pm$ 0.2	93.6 $\pm$ 0.2	94.0 $\pm$ 0.2

Table 1: Predictive performance comparison (top-1 accuracy, %) on CUB-200-2011 and Stanford Cars. Best results among intrinsically interpretable models are highlighted as first , second and third . † denotes ResNet50 pre-trained on iNaturalist for CUB.

Figure 3: Qualitative visualization of the AMP reasoning process on the CUB-200-2011 (left) and Stanford Cars (right) datasets. For a given test image, AMP explicitly decomposes the final Class Evidence Score into a sparse sum of localized visual evidence.

Black-box Classifiers
Method	CUB-200-2011				Stanford Cars
Method	Cons.	Stab.	OIRR	DAUC	Cons.	Stab.	OIRR	DAUC
RA-CNN	23.40 ±1.90	36.70 ±1.60	51.60 ±1.40	8.05 ±0.58	15.30 ±1.95	59.60 ±1.35	56.20 ±1.30	10.35 ±0.60
NTS-Net	24.60 ±2.05	37.20 ±1.90	50.40 ±1.70	7.82 ±0.61	16.50 ±2.10	60.90 ±1.55	54.90 ±1.55	9.95 ±0.58
PMG	26.00 ±2.25	38.60 ±2.15	47.90 ±2.00	7.35 ±0.60	18.10 ±2.30	62.30 ±1.90	53.20 ±1.95	9.38 ±0.56
Intrinsically Interpretable Models

Table 2: Interpretability evaluation on CUB-200-2011 and Stanford Cars. Best results are highlighted as first , second and third .

Figure 4: Hyperparameter sensitivity on CUB-200-2011 and Standford Cars.

Method	CUB-200-2011					Stanford Cars
Method	Acc	Cons.	Stab.	OIRR	DAUC	Acc	Cons.	Stab.	OIRR	DAUC
w/o Stiefel	85.2	65.0	44.2	36.5	4.95	90.2	41.2	72.3	40.8	6.95
w/o $\Sigma_{c}$	87.9	75.8	47.6	28.9	3.55	91.8	48.7	76.0	35.4	5.85
w/o $\mathcal{L}_{SEM}$	87.0	73.2	47.8	30.4	3.92	91.1	46.2	74.5	35.0	6.20
w/o $\mathcal{L}_{overlap}$	88.1	74.6	48.6	29.1	3.60	91.6	49.0	75.7	34.0	5.78
AMP (Full)	88.4	76.8	49.2	28.1	3.45	92.0	50.2	76.4	33.5	5.65

Table 3: Ablation study on CUB-200-2011 and Stanford Cars. Best results are highlighted as first , second and third .

Figure 5: Human evaluation results and dynamic rank distribution.

Prototype-Based Classification

Prototype-based classification assigns labels by comparing inputs with representative examples or learned class centers, often aiding interpretation and data efficiency.

See all 75 papers

Beyond Points: Spherical Distributional Part Prototypes for Interpretable Classification
Jun 25, 2026Duarte Leão, Diogo Pereira Araújo, Catarina Barata +1Prototype-Based Classification Image Classification
Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University
Diverse by Design: Architectural Constraints for Prototype-Based Interpretability
Sep 23, 2026Xinmiao Lin, Matthew WrightPrototype-Based Classification Prototype Learning
Rochester Institute of Technology
Engram Memory Network: Brain-Inspired Prototype Explanations
Oct 20, 2026Hyunjun Kim, Myoung Hoon HaPrototype-Based Explanation Explainable Image Classification
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea · Center for Neuroscience-inspired AI, KAIST, Daejeon, South Korea

Jun 25, 2026J/K move · Enter open · S save

Beyond Points: Spherical Distributional Part Prototypes for Interpretable Classification

Duarte Leão, Diogo Pereira Araújo, Catarina Barata, Carlos Santiago

Instituto Superior Técnico, Universidade de Lisboa · Carnegie Mellon University

Prototype-Based Classification Image Classification Prototype-Based Explanation

Open paper AskPDF arXiv

Prototype-based neural networks aim to provide intrinsic interpretability by grounding predictions in a small set of part prototypes. However, modern vision backbones typically operate in normalized, directional embedding spaces where each semantic part exhibits substantial intra-class variability. As a result, point prototypes often become redundant or unstable, hurting both explanation quality and robustness. We propose vMFProto, a distributional part-prototype framework that models each class as a mixture of von Mises-Fisher components on the hypersphere. Each prototype learns its own concentration, capturing part-specific variability, and we use entropic optimal transport (OT) to obtain structured patch-to-prototype assignments. A two-stage training schedule performs OT-driven prototype discovery followed by end-to-end refinement with patch-level distillation and distribution-aware diversity regularization. Experiments with frozen DINO backbones show that vMFProto achieves leading consistency and distinctiveness on CUB-200-2011 and competitive classification accuracy across CUB, Stanford Dogs, and Stanford Cars. Qualitative results confirm that vMFProto yields localized, non-redundant part evidence.

Abstract

Prototype-Based Classification

Explore similar work

Beyond Points: Spherical Distributional Part Prototypes for Interpretable Classification

Diverse by Design: Architectural Constraints for Prototype-Based Interpretability

Engram Memory Network: Brain-Inspired Prototype Explanations