Fine-grained image-text retrieval requires a model not only to assess global semantic consistency between an image and a text, but also to model visual evidence for local textual semantics, including objects, attributes, quantities, actions, and relations, so as to distinguish semantically similar candidates. Existing methods commonly use fixed image patches, detected regions, or text tokens as the basic matching units and then aggregate local interactions through max pooling, cross attention, or similarity accumulation. Such strategies remain limited in semantic unit construction and global matching consistency. This paper proposes a hierarchical fine-grained cross-modal alignment framework, termed HFCMA. The framework progressively transforms low-level patch and token representations into adaptive visual regions and textual phrases, and then performs structured region-phrase matching within a global retrieval scoring process. Specifically, HFCMA first adopts dynamic granularity discovery (DGD) to aggregate raw patches and tokens into semantically compact visual regions and textual phrases. It then formulates region-phrase matching as an entropy-regularized optimal transport problem, thereby estimating a structured alignment matrix under global marginal constraints. To reduce background noise and local semantic drift, HFCMA further introduces background debiasing, textual importance weighting, and prototype consistency regularization. Experiments on Flickr30K and MS-COCO show that HFCMA effectively combines dynamic semantic unit construction with OT-based structured matching under a dual-encoder retrieval framework. The resulting fine-grained score provides structured local alignment information for candidate ranking and improves the discrimination of subtle semantic differences.
Figures & tables
Figure 1: Overall framework of HFCMA . A frozen CLIP-style dual encoder extracts visual patch and textual token features. Dynamic granularity discovery organizes them into adaptive visual and textual semantic units, followed by visual background debiasing and phrase importance weighting. The refined units are matched by entropy-regularized OT , and the resulting fine-grained score is combined with global CLIP similarity for final image-text retrieval.
Figure 2: Illustration of dynamic granularity discovery. Patch and token sequences are softly assigned to learnable visual and phrase slots, producing compact semantic units without requiring bounding-box or phrase-boundary annotations. Sparse and diverse regularization encourages sample-adaptive slot activation and reduces redundant local units.
Figure 3: OT -based structured matching between discovered visual and textual slots. HFCMA builds a cross-modal cost matrix, estimates a transport plan with Sinkhorn iteration under visual and textual marginal constraints, and converts the transport cost into a fine-grained score for reranking together with the global similarity.
Benchmark
Method
I2T R@1
R@5
R@10
T2I R@1
R@5
R@10
RSUM
Flickr30K-1K
VSE++ (2018) [ 50 ]
52.9
80.5
87.2
39.6
70.1
79.5
409.8
SCAN (2018) [ 14 ]
67.4
90.3
95.8
48.6
77.7
85.2
465.0
VSRN (2019) [ 15 ]
71.3
90.6
96.0
54.7
81.8
88.2
482.6
IMRAM (2020) [ 51 ]
74.1
93.0
96.6
53.9
79.4
87.2
484.2
NAAF (2022) [ 19 ]
81.9
96.1
98.3
61.0
85.3
90.6
513.2
CHAN (2023) [ 20 ]
80.6
96.1
97.8
63.9
87.5
92.6
518.5
Table 1: Main retrieval results on Flickr30K-1K and MS-COCO.
Cross-modal retrieval has been advanced by vision-language pre-trained models that encode images and texts into a shared dense embedding space. While dense representations effectively capture overall semantic similarity, they often obscure fine-grained visual-textual information needed for precise cross-modal matching. Recent methods introduce a learned sparse branch to complement dense matching with lexical evidence, but they rely on a redundant language-model token space and lack explicit grounding for sparse dimensions. We propose GRASP, a compact and grounded sparse learning framework that mines visual-textual concepts from the corpus. A lightweight sparse head is trained to predict concepts relevant to each image or text, yielding interpretable concept-level evidence that complements dense semantic matching. Extensive experiments show that GRASP improves retrieval accuracy over the state-of-the-art dense-sparse baselines while yielding a more compact and grounded sparse space.
Yoonseo Kim, Jungwoo Choi, Cheonyoung Park +3
Sungkyunkwan University South Korea · KT Corporation South Korea
Fine-grained cross-modal alignment aims to establish precise local correspondences between vision and language, forming a cornerstone for visual question answering and related multimodal applications. Current approaches face challenges in addressing patch redundancy and ambiguity, which arise from the inherent information density disparities across modalities. Recently, Multimodal Large Language Models (MLLMs) have emerged as promising solutions to bridge this gap through their robust semantic generation capabilities. However, the dense textual outputs from MLLMs may introduce conflicts with the original sparse captions. Furthermore, accurately quantifying semantic relevance between rich visual patches and concise textual descriptions remains a core challenge. To overcome these limitations, we introduce the Semantic-Enhanced Patch Slimming (SEPS) framework, which systematically addresses patch redundancy and ambiguity. Our approach employs a two-stage mechanism to integrate unified semantics from both dense and sparse texts, enabling the identification of salient visual patches. Additionally, it leverages relevance-aware selection with mean value computation to highlight crucial patch-word correspondences, thereby improving cross-modal similarity assessment. Comprehensive experiments on Flickr30K and MS-COCO datasets validate that SEPS achieves superior performance, surpassing existing approaches by 23%-86% in rSum across diverse model architectures, with notable enhancements in text-to-image retrieval scenarios. Our implementation is available at https://github.com/Sweet4tars/seps.git.
Xinyu Mao, Junsi Li, Haoji Zhang +2
School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China.
Remote sensing (RS) image-text retrieval plays a critical role in understanding massive RS imagery. However, the dense multi-object distribution and complex backgrounds in RS imagery make it difficult to simultaneously achieve fine-grained cross-modal alignment and efficient retrieval. Existing methods either rely on complex cross-modal interactions that lead to low retrieval efficiency, or depend on large-scale vision-language model pre-training, which requires massive data and computational resources. To address these issues, we propose a fast-then-fine (FTF) two-stage retrieval framework that decomposes retrieval into a text-agnostic recall stage for efficient candidate selection and a text-guided rerank stage for fine-grained alignment. Specifically, in the recall stage, text-agnostic coarse-grained representations are employed for efficient candidate selection; in the rerank stage, a parameter-free balanced text-guided interaction block enhances fine-grained alignment without introducing additional learnable parameters. Furthermore, an inter- and intra-modal loss is designed to jointly optimize cross-modal alignment across multi-granular representations. Extensive experiments on public benchmarks demonstrate that the FTF achieves competitive retrieval accuracy while significantly improving retrieval efficiency compared with existing methods.
Xi Chen, Xu Chen, Xiangyang Jia +3
School of Computer Science, Wuhan University, Wuhan, China · Beijing Institute for General Artificial Intelligence (BIGAI), Beijing, China