Fine-grained image-text retrieval requires a model not only to assess global semantic consistency between an image and a text, but also to model visual evidence for local textual semantics, including objects, attributes, quantities, actions, and relations, so as to distinguish semantically similar candidates. Existing methods commonly use fixed image patches, detected regions, or text tokens as the basic matching units and then aggregate local interactions through max pooling, cross attention, or similarity accumulation. Such strategies remain limited in semantic unit construction and global matching consistency. This paper proposes a hierarchical fine-grained cross-modal alignment framework, termed HFCMA. The framework progressively transforms low-level patch and token representations into adaptive visual regions and textual phrases, and then performs structured region-phrase matching within a global retrieval scoring process. Specifically, HFCMA first adopts dynamic granularity discovery (DGD) to aggregate raw patches and tokens into semantically compact visual regions and textual phrases. It then formulates region-phrase matching as an entropy-regularized optimal transport problem, thereby estimating a structured alignment matrix under global marginal constraints. To reduce background noise and local semantic drift, HFCMA further introduces background debiasing, textual importance weighting, and prototype consistency regularization. Experiments on Flickr30K and MS-COCO show that HFCMA effectively combines dynamic semantic unit construction with OT-based structured matching under a dual-encoder retrieval framework. The resulting fine-grained score provides structured local alignment information for candidate ranking and improves the discrimination of subtle semantic differences.
Figures & tables
Figure 1: Overall framework of HFCMA . A frozen CLIP-style dual encoder extracts visual patch and textual token features. Dynamic granularity discovery organizes them into adaptive visual and textual semantic units, followed by visual background debiasing and phrase importance weighting. The refined units are matched by entropy-regularized OT , and the resulting fine-grained score is combined with global CLIP similarity for final image-text retrieval.
Figure 2: Illustration of dynamic granularity discovery. Patch and token sequences are softly assigned to learnable visual and phrase slots, producing compact semantic units without requiring bounding-box or phrase-boundary annotations. Sparse and diverse regularization encourages sample-adaptive slot activation and reduces redundant local units.
Figure 3: OT -based structured matching between discovered visual and textual slots. HFCMA builds a cross-modal cost matrix, estimates a transport plan with Sinkhorn iteration under visual and textual marginal constraints, and converts the transport cost into a fine-grained score for reranking together with the global similarity.
Benchmark
Method
I2T R@1
R@5
R@10
T2I R@1
R@5
R@10
RSUM
Flickr30K-1K
VSE++ (2018) [ 50 ]
52.9
80.5
87.2
39.6
70.1
79.5
409.8
SCAN (2018) [ 14 ]
67.4
90.3
95.8
48.6
77.7
85.2
465.0
VSRN (2019) [ 15 ]
71.3
90.6
96.0
54.7
81.8
88.2
482.6
IMRAM (2020) [ 51 ]
74.1
93.0
96.6
53.9
79.4
87.2
484.2
NAAF (2022) [ 19 ]
81.9
96.1
98.3
61.0
85.3
90.6
513.2
CHAN (2023) [ 20 ]
80.6
96.1
97.8
63.9
87.5
92.6
518.5
Table 1: Main retrieval results on Flickr30K-1K and MS-COCO.