Vision-language models such as CLIP often struggle to faithfully understand long, detail-rich captions, relying on dominant scene cues while overlooking fine-grained visual evidence. We propose a hierarchical vision-language learning principle for understanding scenes as part-to-whole compositions: before forming a whole-scene representation, a model should uncover what semantic parts appear where in the image. To this end, we propose CAFT (Cross-domain Alignment of Forest and Trees), a vision-language model that jointly learns local text-region alignment at intermediate representations and global image-text alignment at the final representation. Exploiting the organization of long captions, where local descriptions often correspond to scene parts, CAFT employs a fine-to-coarse image encoder and a part-whole text encoder to discover localized part semantics and progressively compose them into a global image-text representation. Trained on 30M image-text pairs, CAFT achieves state-of-the-art performance on six long-text retrieval benchmarks and exhibits strong scaling behavior. Experiments show that CAFT learns fine-grained representations that localize textual semantics in image regions without explicit region-level supervision.
Figures & tables
Figure 1: Fine-grained image-text retrieval requires composing local evidence into global understanding. (a) Among visually similar bus images, only the correct image contains all entities described in the long caption, including the hatchback car and the stone building . Correct retrieval requires identifying what appears where and integrating this local evidence into a whole-scene understanding. (b) Our model retrieves the correct image with a clearly higher matching score. This result reflects its ability to ground individual textual descriptions in the corresponding image regions and compose these grounded part-level semantics into a global scene representation.
Figure 2: Hierarchical vision-language learning. A correct understanding of the whole is established by understanding its constituent parts. We place part-level alignment ( trees ) beneath whole-level alignment ( forest ), so that whole semantics are built from localized part semantics.
Figure 3: Overview of CAFT. The model constructs hierarchical representations for both vision and language, aligning them at matched granularities. Vision Branch : Starting from superpixel tokens, the model performs fine-to-coarse scene parsing via progressive token grouping interleaved with ViT blocks. Language Branch : A Part-text Transformer encodes each part-text independently, followed by a Whole-text Transformer that aggregates them into a holistic embedding. Alignment : We establish a bottom-up hierarchy where intermediate visual features align with part-text embeddings for localized grounding ( aligning trees ), while final visual features align with whole-text embeddings to capture global scene semantics ( aligning forest ).
Method
Data
DCI
DOCCI
SV-1k
SV-10k
Urban-1k
IIW
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
Trained on Short-Captions Only
OpenCLIP
2B
56.0
55.4
-
-
90.3
87.7
69.6
66.8
69.5
65.8
-
-
SigLIP
10B
57.7
56.2
-
-
85.8
83.4
83.4
63.0
62.7
62.1
-
-
Trained on Short Captions → Finetuned on Long-Captions
Long-CLIP
400M → 1M
51.6
57.1
63.1
71.4
90.6
87.4
73.1
62.0
78.9
79.5
-
-
Table 1: CAFT achieves state-of-the-art results on zero-shot long text-image retrieval. I2T and T2I denote image-to-text and text-to-image R@1, respectively. The best results are bold , second-best are underlined , and † denotes use of bounding box annotations.
Figure 4: CAFT exhibits strong scaling behavior. We evaluate how CAFT scales with training data by comparing it with FLAIR, a strong competitor trained on the same datasets, across training set sizes of 3M, 12M, 15M, and 30M. CAFT achieves consistent improvements as the training set grows, highlighting its robust scaling behavior and the effectiveness of its hierarchical inductive bias.
Figure 5: CAFT provides spatially precise grounding. We show head-averaged attention maps from the AttnPool module for each sub-caption. CAFT precisely localizes relevant regions across different scales and positions, with sharp, object-aware boundaries. The maps also exhibit inactivation for irrelevant captions, demonstrating robustness to mismatched semantic cues (last example).
Method
RefCOCO
RefCOCO+
GRef
val
testA
testB
val
testA
testB
val
GViT
8.0
6.2
10.5
8.5
6.8
10.6
10.7
MCLIP
11.5
11.9
12.1
11.9
12.0
12.6
12.7
SaG
21.8
19.0
25.0
22.2
19.9
24.9
25.9
FLAIR
19.8
19.8
20.0
18.5
18.3
18.9
20.2
CAFT
28.9
30.8
28.5
28.5
29.8
28.3
31.4
Table 2: Zero-shot referring image segmentation. We report mIoU for models trained without mask supervision. CAFT achieves the best performance across all splits.
Figure 6: Hierarchical Part → Whole alignment is key to global scene understanding. We compare three alignment strategies: (a) Whole , (b) Part + Whole , and (c) Part → Whole . The table presents zero-shot long-caption retrieval performance (R@1) across five benchmarks, with the corresponding alignment strategies illustrated below. Our hierarchical Part → Whole alignment substantially outperforms both alternative alignment strategies.
Method
Average
DCI
DOCCI
SV-10k
Urban-1k
IIW
VH
TH
HA
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
(1)
69.5
51.6
50.2
59.4
56.5
76.7
76.4
74.2
76.7
87.4
85.5
(2)
✓
71.2
53.7
51.3
62.2
58.5
78.8
78.5
76.2
76.9
89.2
86.4
(3)
✓
68.7
48.3
45.9
58.7
55.1
78.6
77.9
76.3
77.2
85.1
84.2
(4)
✓
✓
74.4
53.4
54.4
64.3
62.7
83.0
81.8
80.7
84.6
90.0
89.2
(5)
✓
✓
✓
78.1
57.6
59.6
67.3
67.1
86.2
85.4
83.6
87.2
93.6
93.3
Table 4: Ablation study on components. We ablate three main components of CAFT on CC3M-recap: visual hierarchy ( VH ) replaces ViT with the fine-to-coarse image encoder, textual hierarchy ( TH ) replaces the long-text encoder with the part-whole text encoder, and hierarchical alignment ( HA ) adds additional part-level alignment on top of the whole-level alignment loss.
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
Dataset
# Images
Training Hours
GPU Hours (Days)
CC3M-recap
0 2.82M
0 12.3
0 98.6 ( 0 4.1)
CC12M-recap
10.01M
0 42.5
340.0 (14.2)
YFCC15M-recap
14.07M
0 59.5
476.0 (19.8)
Merged-30M
26.90M
114.4
915.2 (38.1)
Appendix
Table 5: Number of images and the corresponding GPU-hours for each dataset scales.
Dataset
# Images
# Texts
# Sub-captions per Text
# Tokens per Text
DCI
0 7,805
0 7,805
10.81
172.73
DOCCI
0 5,000
0 5,000
0 7.12
141.52
ShareGPT4v-1k
0 1,000
0 1,000
0 8.15
173.24
ShareGPT4v-10k
10,000
10,000
0 8.24
173.66
Urban-1k
0 1,000
0 1,000
0 5.97
131.36
IIW
00 612
00 612
10.16
239.73
Appendix
Table 6: Statistics of the long-text-image retrieval datasets.
Config
CC3M-recap
CC12M-recap
YFCC15M-recap
Merged-30M
Batch size
2,048
Optimizer
AdamW [ Loshchilov and Hutter, 2017 ]
Learning rate
5×10−4
Weight decay
0.5
0.5
0.5
0.2
Adam β
β1,β2=(0.9,0.98)
Adam ϵ
1×10−8
1×10−8
1×10−8
1×10−6
Appendix
Table 7: Training configuration for different datasets.
Model
Image Encoder
Text Encoder
Total
FLAIR
18.46
3.08
21.54
LongCLIP
18.32
10.91
29.23
CAFT
11.34
8.27
19.61
Appendix
Table 8: Computational complexity comparison. We report GFLOPs per single image-text pair, separately for the image encoder, text encoder, and their total cost. CAFT is computationally cheaper overall than both FLAIR and LongCLIP.
α
Selection
Zero-shot Evaluation
DreamLIP-3M (30K)
DCI
SV-10k
Urban-1k
IIW
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
0.0
97.4
91.9
69.4
69.4
95.1
94.5
93.6
93.9
97.4
96.1
0.3
97.5
92.3
69.7
72.4
95.5
94.8
93.6
95.4
97.5
97.4
0.7
97.0
92.1
63.2
74.1
94.3
94.3
92.2
96.3
97.4
97.9
1.0
94.3
90.8
50.5
71.6
90.3
92.3
86.2
94.6
95.3
97.4
Appendix
Table 9: Selection and ablation of the inference weight α . We select α using a randomly sampled 30K subset (approximately 1%) of DreamLIP-3M training data, without using any evaluation benchmark. We additionally report zero-shot retrieval R@1 across four benchmarks to analyze sensitivity to α . α=0.0 and α=1.0 correspond to whole-only and part-only inference, respectively.
Inference mode
Latency (s)
Peak memory (GB)
Average R@1
Whole-only ( k=0 )
30.74
6.14
87.04
Top- k ( k=16 )
31.94
6.39
87.83
Top- k ( k=64 )
32.70
6.93
87.79
Top- k ( k=128 )
33.49
8.23
87.77
Top- k ( k=256 )
34.58
15.27
87.78
Full scoring
38.53
19.77
87.88
Appendix
Table 10: Inference accuracy-efficiency trade-off. We compare whole-only inference, top- k reranking with α=0.3 , and full part-level scoring. Latency and peak GPU memory are measured on SV-10k using a single A100 GPU. Average R@1 is computed over five long-text retrieval benchmarks.
Method
COCO
Flickr30k
I2T
T2I
I2T
T2I
LongCLIP [ Zhang et al., 2024 ]
57.3
40.3
85.9
70.7
TULIP [ Najdenkoska et al., 2024 ]
56.8
40.7
–
–
FineLIP [ Asokan et al., 2025 ]
58.7
40.4
–
–
LoTLIP [ Wu et al., 2024 ]
59.7
38.1
86.9
65.2
DeBias-CLIP [ Lavoie et al., 2026 ]
61.3
43.0
–
–
Appendix
Table 11: Comparison with long-text CLIP models on short-text retrieval. CAFT substantially outperforms existing long-text CLIP models on MSCOCO and Flickr30k.
Method
COCO
Flickr30k
Urban-1k
I2T
T2I
I2T
T2I
I2T
T2I
OpenCLIP [ Radford et al., 2021 ]
59.3
41.7
87.5
71.9
69.5
65.8
MetaCLIP [ Xu et al., 2023 ]
59.3
41.3
85.6
70.8
68.9
63.3
SigLIP [ Zhai et al., 2023 ]
65.5
47.2
89.1
75.6
62.7
62.1
SigLIP 2 [ Tschannen et al., 2025 ]
68.9
52.1
93.0
80.7
60.3
57.8
FLAIR [ Xiao et al., 2025a ]
68.0
53.3
94.7
81.1
83.6
87.7
Appendix
Table 12: Comparison with standard CLIP models on short-text retrieval. CAFT outperforms SigLIP, and remains competitive with recent strong models on short-text retrieval, while achieving substantially stronger performance on long-caption retrieval (gray column).
Method
Part-alignment layer
# Visual tokens
Average R@1
Whole
–
196
68.7
Part + Whole
Final (11th)
196
68.6
Part → Whole
Intermediate (6th)
196
74.4
Appendix
Table 13: Alignment strategies with a ViT backbone. We repeat the alignment-strategy ablation using a standard ViT with 196 visual tokens across all layers. Average R@1 is reported over five long-text retrieval benchmarks and all experiments are conducted on DreamLIP-3M.
Model
Similarity drop (%)
p -value
LongCLIP
1.41
1.4×10−53
DeBias-CLIP
3.02
9.8×10−39
CAFT
6.54
–
Appendix
Table 14: Counterfactual visual intervention. Mean normalized similarity drop (%) after masking an annotated image region. The p -values are from paired one-sided Wilcoxon signed-rank tests against CAFT.
Model
Different-caption
Same-caption
LongCLIP
3.66
1.06
DeBias-CLIP
10.11
2.91
CAFT
17.79
5.76
Appendix
Table 15: Counterfactual textual interventions. Mean normalized similarity drop (%) when an annotated entity or attribute phrase is replaced with a phrase from a different or the same caption.
N
Resampled (%)
Discarded (%)
Truncated (%)
3
0.00
15.44
7.25
4
12.64
2.49
2.32
5
35.90
0.15
0.97
Appendix
Table 16: Caption processing rates on DreamLIP-3M. Percentage of captions involving sentence resampling, sentence discarding, or token truncation for different numbers of chunks N .
N
Average
DCI
DOCCI
SV-10k
Urban-1k
IIW
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
3
62.0
42.9
42.6
51.0
47.9
71.0
69.9
64.2
69.3
80.9
80.1
4
62.7
43.2
42.1
51.7
48.3
71.2
69.9
65.0
69.1
84.6
81.5
5
63.8
44.6
44.9
52.5
49.7
71.3
70.3
66.7
70.7
85.3
81.5
Appendix
Table 17: Retrieval sensitivity to the number of chunks. All variants are trained for four epochs on DreamLIP-3M. We report average R@1 and I2T / T2I R@1 on five benchmarks.
Strategy
Average
DCI
DOCCI
SV-10k
Urban-1k
IIW
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
I2T
T2I
1–3 sentences, N=4
78.1
57.6
59.6
67.3
67.1
86.2
85.4
83.6
87.2
93.6
93.3
Single sentence, N≤12
76.6
56.3
57.5
66.5
65.3
85.0
84.6
81.9
85.2
92.5
90.9
Appendix
Table 18: Chunking strategy ablation on DreamLIP-3M. We compare fixed 1–3-sentence grouping with variable-length single-sentence chunking. We report average R@1 and I2T / T2I R@1 across five benchmarks.
Method
Hard
Medium
Easy
Trivial
Trained w/o dense annotations
CLIP
12.0
23.1
22.2
58.5
EVA-CLIP
14.0
30.1
29.4
58.3
Long-CLIP
9.2
18.4
16.2
51.8
CAFT
16.8
32.8
38.6
62.5
Trained w/ dense annotations
Appendix
Table 20: FG-OVD results across difficulty levels.
Dataset
Direction
CAFT
LongCLIP
DeBias-CLIP
Mean
86.89→86.91
70.48→70.49
84.15→84.15
DCI
I2T
69.68→69.72
57.66→57.66
67.61→67.63
T2I
72.42→72.50
51.10→51.10
68.96→68.97
SV-10k
I2T
95.47→95.47
76.59→76.60
90.30→90.29
T2I
94.79→94.81
78.64→78.64
91.72→91.71
Urban-1k
I2T
93.60→93.59
79.10→79.16
92.70→92.69
Appendix
Table 21: Retrieval performance before and after overlap removal. R@1 before → after removing overlapping evaluation images. Mean averages the six retrieval directions shown.
DCI (7)
SV-10k (6)
Urban-1k (2)
sa_1552159.jpg
sa_000007/images/sa_85112.jpg
Urban1k/images/542.jpg
sa_1546970.jpg
sa_000040/images/sa_452601.jpg
Urban1k/images/579.jpg
sa_1549938.jpg
sa_000002/images/sa_26529.jpg
sa_1553651.jpg
sa_000004/images/sa_46609.jpg
sa_1547809.jpg
sa_000007/images/sa_87906.jpg
sa_1545429.jpg
sa_000008/images/sa_93454.jpg
Appendix
Table 22: Confirmed training–evaluation image overlaps. All 15 overlapping evaluation images are listed by dataset. DCI filenames are relative to dci/densely_captioned_images/photos/ .
Figure 8: Vision-language understanding requires hierarchical semantics to be aligned across domains. Traditional textual hierarchies , such as syntactic parse trees or semantic graphs, do not naturally correspond to the spatial organization of visual scenes. In contrast, spatially oriented textual hierarchies , where each part describes a distinct image region, align naturally with visual hierarchies .
Figure 9: CAFT provides spatially grounded, compositional visual grounding. We visualize attention maps from the AttnPool module for three subcaptions in each example. The leftmost image is the input; the three heatmaps to the right show head-averaged attention with respect to each sub-caption. The scribble example shows CAFT distinguishing pink circles from green triangles . Querying with the text ORANGE TREE IMP , CAFT grounds it accurately. These are preliminary examples, with possible confounding factors.
Figure 10: CAFT is capable of visual grounding across multiple levels of granularity . A single caption may describe different objects simultaneously, and can be decomposed into smaller units. To reflect this hierarchical nature of caption, we decompose the sentence-level caption into smaller phrases and visualize their corresponding grounding results. Remarkably, CAFT (1) can locate multiple objects at the same time (e.g., both a man and horses in the bottom left) and (2) precisely identify individual parts in the phrases (e.g., holding a broom in the bottom right).