Aligning Forest and Trees in Images & Long Captions for Visually Grounded Understanding
Organizations: Agency for Defense Development · University of Michigan · POSTECH
Abstract
Vision-language models such as CLIP often struggle to faithfully understand long, detail-rich captions, relying on dominant scene cues while overlooking fine-grained visual evidence. We propose a hierarchical vision-language learning principle for understanding scenes as part-to-whole compositions: before forming a whole-scene representation, a model should uncover what semantic parts appear where in the image. To this end, we propose CAFT (Cross-domain Alignment of Forest and Trees), a vision-language model that jointly learns local text-region alignment at intermediate representations and global image-text alignment at the final representation. Exploiting the organization of long captions, where local descriptions often correspond to scene parts, CAFT employs a fine-to-coarse image encoder and a part-whole text encoder to discover localized part semantics and progressively compose them into a global image-text representation. Trained on 30M image-text pairs, CAFT achieves state-of-the-art performance on six long-text retrieval benchmarks and exhibits strong scaling behavior. Experiments show that CAFT learns fine-grained representations that localize textual semantics in image regions without explicit region-level supervision.
Figures & tables
| Method | Data | DCI | DOCCI | SV-1k | SV-10k | Urban-1k | IIW | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| Trained on Short-Captions Only | |||||||||||||
| OpenCLIP | 2B | 56.0 | 55.4 | - | - | 90.3 | 87.7 | 69.6 | 66.8 | 69.5 | 65.8 | - | - |
| SigLIP | 10B | 57.7 | 56.2 | - | - | 85.8 | 83.4 | 83.4 | 63.0 | 62.7 | 62.1 | - | - |
| Trained on Short Captions Finetuned on Long-Captions | |||||||||||||
| Long-CLIP | 400M 1M | 51.6 | 57.1 | 63.1 | 71.4 | 90.6 | 87.4 | 73.1 | 62.0 | 78.9 | 79.5 | - | - |
| Method | RefCOCO | RefCOCO+ | GRef | ||||
|---|---|---|---|---|---|---|---|
| val | testA | testB | val | testA | testB | val | |
| GViT | 8.0 | 6.2 | 10.5 | 8.5 | 6.8 | 10.6 | 10.7 |
| MCLIP | 11.5 | 11.9 | 12.1 | 11.9 | 12.0 | 12.6 | 12.7 |
| SaG | 21.8 | 19.0 | 25.0 | 22.2 | 19.9 | 24.9 | 25.9 |
| FLAIR | 19.8 | 19.8 | 20.0 | 18.5 | 18.3 | 18.9 | 20.2 |
| CAFT | 28.9 | 30.8 | 28.5 | 28.5 | 29.8 | 28.3 | 31.4 |
| Method | Average | DCI | DOCCI | SV-10k | Urban-1k | IIW | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VH | TH | HA | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| (1) | 69.5 | 51.6 | 50.2 | 59.4 | 56.5 | 76.7 | 76.4 | 74.2 | 76.7 | 87.4 | 85.5 | |||
| (2) | 71.2 | 53.7 | 51.3 | 62.2 | 58.5 | 78.8 | 78.5 | 76.2 | 76.9 | 89.2 | 86.4 | |||
| (3) | 68.7 | 48.3 | 45.9 | 58.7 | 55.1 | 78.6 | 77.9 | 76.3 | 77.2 | 85.1 | 84.2 | |||
| (4) | 74.4 | 53.4 | 54.4 | 64.3 | 62.7 | 83.0 | 81.8 | 80.7 | 84.6 | 90.0 | 89.2 | |||
| (5) | 78.1 | 57.6 | 59.6 | 67.3 | 67.1 | 86.2 | 85.4 | 83.6 | 87.2 | 93.6 | 93.3 | |||
Appendix figures & tables20 assets
Supplementary material from the paper’s appendix.
Appendix
| Dataset | # Images | Training Hours | GPU Hours (Days) |
|---|---|---|---|
| CC3M-recap | 0 2.82M | 0 12.3 | 0 98.6 ( 0 4.1) |
| CC12M-recap | 10.01M | 0 42.5 | 340.0 (14.2) |
| YFCC15M-recap | 14.07M | 0 59.5 | 476.0 (19.8) |
| Merged-30M | 26.90M | 114.4 | 915.2 (38.1) |
| Dataset | # Images | # Texts | # Sub-captions per Text | # Tokens per Text |
| DCI | 0 7,805 | 0 7,805 | 10.81 | 172.73 |
| DOCCI | 0 5,000 | 0 5,000 | 0 7.12 | 141.52 |
| ShareGPT4v-1k | 0 1,000 | 0 1,000 | 0 8.15 | 173.24 |
| ShareGPT4v-10k | 10,000 | 10,000 | 0 8.24 | 173.66 |
| Urban-1k | 0 1,000 | 0 1,000 | 0 5.97 | 131.36 |
| IIW | 00 612 | 00 612 | 10.16 | 239.73 |
| Config | CC3M-recap | CC12M-recap | YFCC15M-recap | Merged-30M |
|---|---|---|---|---|
| Batch size | 2,048 | |||
| Optimizer | AdamW [ Loshchilov and Hutter, 2017 ] | |||
| Learning rate | ||||
| Weight decay | 0.5 | 0.5 | 0.5 | 0.2 |
| Adam | ||||
| Adam | ||||
| Model | Image Encoder | Text Encoder | Total |
|---|---|---|---|
| FLAIR | 18.46 | 3.08 | 21.54 |
| LongCLIP | 18.32 | 10.91 | 29.23 |
| CAFT | 11.34 | 8.27 | 19.61 |
| Selection | Zero-shot Evaluation | |||||||||
| DreamLIP-3M (30K) | DCI | SV-10k | Urban-1k | IIW | ||||||
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | |
| 0.0 | 97.4 | 91.9 | 69.4 | 69.4 | 95.1 | 94.5 | 93.6 | 93.9 | 97.4 | 96.1 |
| 0.3 | 97.5 | 92.3 | 69.7 | 72.4 | 95.5 | 94.8 | 93.6 | 95.4 | 97.5 | 97.4 |
| 0.7 | 97.0 | 92.1 | 63.2 | 74.1 | 94.3 | 94.3 | 92.2 | 96.3 | 97.4 | 97.9 |
| 1.0 | 94.3 | 90.8 | 50.5 | 71.6 | 90.3 | 92.3 | 86.2 | 94.6 | 95.3 | 97.4 |
| Inference mode | Latency (s) | Peak memory (GB) | Average R@1 |
|---|---|---|---|
| Whole-only ( ) | 30.74 | 6.14 | 87.04 |
| Top- ( ) | 31.94 | 6.39 | 87.83 |
| Top- ( ) | 32.70 | 6.93 | 87.79 |
| Top- ( ) | 33.49 | 8.23 | 87.77 |
| Top- ( ) | 34.58 | 15.27 | 87.78 |
| Full scoring | 38.53 | 19.77 | 87.88 |
| Method | COCO | Flickr30k | ||
|---|---|---|---|---|
| I2T | T2I | I2T | T2I | |
| LongCLIP [ Zhang et al., 2024 ] | 57.3 | 40.3 | 85.9 | 70.7 |
| TULIP [ Najdenkoska et al., 2024 ] | 56.8 | 40.7 | – | – |
| FineLIP [ Asokan et al., 2025 ] | 58.7 | 40.4 | – | – |
| LoTLIP [ Wu et al., 2024 ] | 59.7 | 38.1 | 86.9 | 65.2 |
| DeBias-CLIP [ Lavoie et al., 2026 ] | 61.3 | 43.0 | – | – |
| Method | COCO | Flickr30k | Urban-1k | |||
|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | |
| OpenCLIP [ Radford et al., 2021 ] | 59.3 | 41.7 | 87.5 | 71.9 | 69.5 | 65.8 |
| MetaCLIP [ Xu et al., 2023 ] | 59.3 | 41.3 | 85.6 | 70.8 | 68.9 | 63.3 |
| SigLIP [ Zhai et al., 2023 ] | 65.5 | 47.2 | 89.1 | 75.6 | 62.7 | 62.1 |
| SigLIP 2 [ Tschannen et al., 2025 ] | 68.9 | 52.1 | 93.0 | 80.7 | 60.3 | 57.8 |
| FLAIR [ Xiao et al., 2025a ] | 68.0 | 53.3 | 94.7 | 81.1 | 83.6 | 87.7 |
| Method | Part-alignment layer | # Visual tokens | Average R@1 |
|---|---|---|---|
| Whole | – | 196 | 68.7 |
| Part + Whole | Final (11th) | 196 | 68.6 |
| Part Whole | Intermediate (6th) | 196 | 74.4 |
| Model | Similarity drop (%) | -value |
|---|---|---|
| LongCLIP | 1.41 | |
| DeBias-CLIP | 3.02 | |
| CAFT | 6.54 | – |
| Model | Different-caption | Same-caption |
|---|---|---|
| LongCLIP | 3.66 | 1.06 |
| DeBias-CLIP | 10.11 | 2.91 |
| CAFT | 17.79 | 5.76 |
| Resampled (%) | Discarded (%) | Truncated (%) | |
|---|---|---|---|
| 3 | 0.00 | 15.44 | 7.25 |
| 4 | 12.64 | 2.49 | 2.32 |
| 5 | 35.90 | 0.15 | 0.97 |
| Average | DCI | DOCCI | SV-10k | Urban-1k | IIW | ||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| 3 | 62.0 | 42.9 | 42.6 | 51.0 | 47.9 | 71.0 | 69.9 | 64.2 | 69.3 | 80.9 | 80.1 |
| 4 | 62.7 | 43.2 | 42.1 | 51.7 | 48.3 | 71.2 | 69.9 | 65.0 | 69.1 | 84.6 | 81.5 |
| 5 | 63.8 | 44.6 | 44.9 | 52.5 | 49.7 | 71.3 | 70.3 | 66.7 | 70.7 | 85.3 | 81.5 |
| Strategy | Average | DCI | DOCCI | SV-10k | Urban-1k | IIW | |||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | I2T | T2I | ||
| 1–3 sentences, | 78.1 | 57.6 | 59.6 | 67.3 | 67.1 | 86.2 | 85.4 | 83.6 | 87.2 | 93.6 | 93.3 |
| Single sentence, | 76.6 | 56.3 | 57.5 | 66.5 | 65.3 | 85.0 | 84.6 | 81.9 | 85.2 | 92.5 | 90.9 |
| Method | Hard | Medium | Easy | Trivial |
|---|---|---|---|---|
| Trained w/o dense annotations | ||||
| CLIP | 12.0 | 23.1 | 22.2 | 58.5 |
| EVA-CLIP | 14.0 | 30.1 | 29.4 | 58.3 |
| Long-CLIP | 9.2 | 18.4 | 16.2 | 51.8 |
| CAFT | 16.8 | 32.8 | 38.6 | 62.5 |
| Trained w/ dense annotations | ||||
| Dataset | Direction | CAFT | LongCLIP | DeBias-CLIP |
|---|---|---|---|---|
| Mean | ||||
| DCI | I2T | |||
| T2I | ||||
| SV-10k | I2T | |||
| T2I | ||||
| Urban-1k | I2T | |||
| DCI (7) | SV-10k (6) | Urban-1k (2) |
|---|---|---|
| sa_1552159.jpg | sa_000007/images/sa_85112.jpg | Urban1k/images/542.jpg |
| sa_1546970.jpg | sa_000040/images/sa_452601.jpg | Urban1k/images/579.jpg |
| sa_1549938.jpg | sa_000002/images/sa_26529.jpg | |
| sa_1553651.jpg | sa_000004/images/sa_46609.jpg | |
| sa_1547809.jpg | sa_000007/images/sa_87906.jpg | |
| sa_1545429.jpg | sa_000008/images/sa_93454.jpg |
Vision-Language Alignment
Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.
See all 373 papers