Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing

Hengjie Zhu; Dayan Wu; Zihao Zhang; Xinze Liu; Jingxuan Yu; Peng Fu; Zheng Lin; Weiping Wang

cs.CVAug 25, 2026

Absorbing Gradient Conflicts: Modeling Semantic Variance via Kent Distributions for Cross-Modal Hashing

Authors: Hengjie Zhu, Dayan Wu, Zihao Zhang, Xinze Liu, Jingxuan Yu, Peng Fu, Zheng Lin, Weiping Wang

Organizations: Institute of Information Engineering, Chinese Academy of Sciences · School of Cyber Security, University of Chinese Academy of Sciences · School of Cyber Science and Engineering, Southeast University

Abstract

Supervised proxy-based deep cross-modal hashing has become the dominant paradigm for large-scale retrieval. However, prevalent methods model class proxies as deterministic points in the embedding space. This rigid assumption causes severe gradient conflicts in multi-label scenarios, where gradient conflicts arising from label co-occurrence lead to severe gradient contention and optimization collapse. To resolve this, we propose Kent-based Distributional Proxy Hashing (KDPH), a novel framework that shifts proxy representation from static points to flexible anisotropic Kent distributions on the hypersphere. Unlike point proxies that must shift their positions to accommodate conflicting gradients, KDPH absorbs these conflicts by dynamically adjusting its directional variance. This allows the proxy to maintain a stable semantic mean direction while stretching to cover diverse label correlations. Furthermore, to ensure stable training of these geometric parameters, we derive a tailored loss function incorporating the Cayley transform to enforce strict orthogonality. To the best of our knowledge, KDPH is the first framework to successfully introduce the Kent distributions into cross-modal hashing. Experiments on three benchmark datasets demonstrate that KDPH mitigates proxy collapse and chaotic oscillation, significantly outperforms state-of-the-art methods. Code is available at https://github.com/Senmo996/KDPH-official-code.

Figures & tables

Figure 1: Comparison of proxy optimization behaviors. (a)-(b) Conventional proxy learning suffers from proxy collapse , where learned proxies move toward the anchor and become poorly separated, whereas KDPH preserves distinct proxy distributions. (c)-(d) Conventional methods exhibit slow and oscillatory optimization trajectories, while KDPH achieves smoother trajectories and faster convergence. Colors denote different categories.

Figure 2: Overview of the Kent-based Distributed Proxy Hashing (KDPH) framework. It consists of a CLIP-based feature extraction module and a Kent-based Distributional Proxy Learner. As shown in the right panel, we model class proxies as learnable anisotropic distributions rather than static points. Through the gradient decoupling mechanism by separating centroid $\mu$ and variance $B,\Gamma$ , the model dynamically stretches the proxy’s semantic volume to accommodate multi-label variance, effectively mitigating proxy collapse and oscillation.

Task	Method	Reference	MIRFLICKR-25K			NUS-WIDE			MS COCO
Task	Method	Reference	16bits	32bits	64bits	16bits	32bits	64bits	16bits	32bits	64bits
I2T	DCPH	TKDE’22	0.7679	0.7733	0.7764	0.6161	0.6156	0.6180	0.5206	0.5806	0.5939
	DCMHT	MM’22	0.8263	0.8272	0.8441	0.6799	0.6992	0.7038	0.6447	0.6757	0.6915
	nivMF	ECCV’22	0.8241	0.8245	0.8352	0.7023	0.6902	0.7024	0.7075	0.7242	0.7523
	MIAN	TKDE’22	0.8123	0.8220	0.8355	0.6130	0.5894	0.6057	0.5350	0.5579	0.5404
	DSPH	TCSVT’23	0.8129	0.8482	0.8541	0.6830	0.6979	0.7162	0.7044	0.7510	0.7694

Table 1: mAP comparisons on three benchmark datasets with references. The best results are highlighted in bold.

Panel of Figure 3 — Figure 3: Impact of Semantic Complexity on Model Performance.

Figure 4: Training dynamics comparing DSPH and KDPH.

Components				mAP @ 64 bits
Kent	$\mathcal{L}_{Reg}$	$\mathcal{L}_{CM}$	$\mathcal{L}_{U}$	I $\to$ T	T $\to$ I
	✓	✓	✓	0.7615	0.7542
$\beta=0$	✓	✓	✓	0.7693	0.7628
✓		✓	✓	0.7326	0.7305
✓	✓		✓	0.7566	0.7585
✓	✓	✓		0.7537	0.7574

Table 2: Ablation study of different components on the MS COCO.

Figure 5: t-SNE visualization of the embedding spaces learned by DSPH and KDPH.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Analysis of Semantic Proxy Distribution Shifts. (a) The scatter plot illustrates the topological quality of semantic super-groups. KDPH (blue stars) consistently migrates towards the top-left Ideal Corner compared to the Baseline (orange circles), indicating higher intra-group cohesion and lower inter-group interference. (b) The Separation Scores confirm that KDPH establishes sharper boundaries between semantic concepts. (c) The heatmap details the specific contributions of Intra-Group Gain and Inter-Group Reduction to the overall performance improvement.

Figure 7: Parameter Sensitivity Analysis on MS COCO (@64 bits). We evaluate the impact of the loss weights

\epsilon,\eta,\zeta

and the number of principal axes

N_{axes}

on the Image-to-Text (I2T) mAP. The results justify our default settings (

\epsilon=40,\eta=0.1,\zeta=0.3,N_{axes}=8

Figure 8: Supplementary for RQ1: Training Dynamics on High Co-occurrence Classes. The Baseline (orange) exhibits chaotic oscillation (high flip rates) and jagged precision curves due to gradient contention. KDPH (blue) achieves rapid stabilization and superior convergence, validating its ability to resolve semantic conflicts.

Figure 9: Supplementary for RQ1: Training Dynamics on Low Co-occurrence Classes. Both methods maintain high stability. This confirms that KDPH’s anisotropic modeling is robust and does not negatively impact learning in simple, distinct semantic scenarios.

Figure 10: Chaos Index vs. Real Performance (High Co-occurrence Classes). The charts correlate kinematic stability (Chaos Index, bars) with retrieval accuracy (Precision@50, lines). For complex classes (Avg ACL

\approx

4.1), the Baseline suffers from high chaos and low precision. KDPH effectively suppresses this chaos, resulting in higher path efficiency and significantly improved retrieval performance.

Figure 11: Chaos Index vs. Real Performance (Low Co-occurrence Classes). In scenarios with low semantic interference (Avg ACL

\approx

1.1), both methods maintain low Chaos Indices and high Autocorrelation. This demonstrates that KDPH retains the optimization efficiency of point-based methods when gradient contention is absent.

Figure 12: Ground-Truth Score Divergence (GTSD) Analysis. (a) As the number of labels increases, the Baseline exhibits a widening gap between the best and worst scored labels, indicating proxy collapse. KDPH maintains a stable, lower divergence. (b) The Baseline’s high maximum scores suggest overfitting to dominant labels. (c) KDPH demonstrates increasing efficacy in complex scenarios, reducing the divergence by up to

14.8\%

in dense multi-label settings.

Cross-Modal Retrieval

Cross-modal retrieval matches queries and items across different data types, such as text, images, video, audio, and sensor signals.

See all 184 papers

Attribute-Prompted Kernel Hashing for Unsupervised Data-Efficient Cross-Modal Retrieval
Jul 1, 2026Runhao Li, Xiaoxu Ma, Zhenyu Weng +5Cross-Modal Alignment Cross-Modal Retrieval
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, China · College of Computer and Information Engineering, Henan Normal University, China +2
Differentially Private Motif-Preserving Multi-modal Hashing
May 14, 2026Zehua Cheng, Wei Dai, Jiahao SunCross-Modal Learning Cross-Modal Retrieval
Department of Computer Science University of Oxford Oxford, United Kingdom · FLock.io London, United Kingdom
DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing
Jul 27, 2026Tobias J. Bauer, Christian Riess, Daniel Loebenberger +1Cross-Modal Representation Learning Cross-Modal Retrieval
Fraunhofer Institute for Applied and Integrated Security AISEC · Friedrich-Alexander-Universität Erlangen-Nürnberg · Ostbayerische Technische Hochschule Amberg-Weiden

Jul 1, 2026J/K move · Enter open · S save

Attribute-Prompted Kernel Hashing for Unsupervised Data-Efficient Cross-Modal Retrieval

Runhao Li, Xiaoxu Ma, Zhenyu Weng, Yue Zhang+4

School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · Shien-Ming Wu School of Intelligent Engineering, South China University of Technology, Guangzhou, China · College of Computer and Information Engineering, Henan Normal University, China+2

Cross-Modal Alignment Cross-Modal Retrieval Multimodal Contrastive Learning

Open paper AskPDF arXiv

Unsupervised cross-modal hashing enables efficient retrieval of semantically related instances across different modalities without requiring manual semantic annotation. However, existing unsupervised methods rely heavily on large-scale image-text pairs. Collecting such data can be costly, particularly in scenarios where well-aligned pairs are scarce due to privacy and specialized constraints. More critically, existing methods tend to overfit to seen training data, restricting their generalization performance on unseen categories that the constrained training data cannot cover. To address these limitations, we propose Attribute-Prompted Kernel Hashing (APKH), a novel data-efficient approach that constructs a compact, modality-aligned Hamming space driven by the generalized attribute priors of vision-language foundation models. Specifically, APKH introduces two core modules: Context-optimized Attribute Kernel Mapping (CAKM) and Kernel-Smoothed Contrastive Alignment (KSCA). CAKM formulates cross-modal alignment through hyperspherical Radial Basis Function kernel mapping, optimizing dynamic attribute kernels via prompt learning to capture modality-invariant semantics. Furthermore, KSCA extends conventional point-to-point contrastive learning by modeling limited paired data as continuous kernel distributions. This explicit smoothing of the modality gap alleviates overfitting to sparse pairwise correlations. Extensive experiments demonstrate that APKH outperforms state-of-the-art hashing methods in the challenging cross-modal retrieval tasks from seen to unseen categories under data-constrained scenarios.

Abstract

Cross-Modal Retrieval

Explore similar work

Attribute-Prompted Kernel Hashing for Unsupervised Data-Efficient Cross-Modal Retrieval

Differentially Private Motif-Preserving Multi-modal Hashing

DSCH-Loss: A Dynamic Semantic Channel Objective for Deep Semantic Hashing