A General Concept-based Decomposition for Vision-Language Embeddings
Organizations: Politecnico di Torino Torino, Italy · EPFL Lausanne, Switzerland
Abstract
Vision-Language Models are designed to capture the compositional structure of the world through multi-modal alignment; however, it remains unclear whether their representations truly align with the way a human would naturally describe the same visual input. Prior works attempt to decompose visual embeddings into their concept-level contributions, but their instance-level optimization is inherently local and does not capture the model's global semantic structure, which is essential for compositional generalization across different inputs, tasks, and domains. We introduce GRACE, a method that learns this concept-based structure at the feature space level, producing semantic and general concept decompositions. To achieve this, we propose a novel training objective that preserves the geometric relations of different samples in the concept space, enabling a better understanding of how concepts compose and relate to each other. We validate GRACE on several image and video benchmarks, showing that its concept decomposition is aligned with human perception and achieves faithful decompositions across different tasks and domains. With minimal overhead, GRACE makes existing Vision-Language Models more interpretable, while preserving most of their original performance.
Figures & tables
| Model | Method | Main Action | Main Subjects | Main Objects | Location | Order of Actions | Average |
|---|---|---|---|---|---|---|---|
| / | / | / | / | / | / | ||
| Text-to-Text | |||||||
| mPLUG-Owl3 [ 48 ] | SBERT | 52.1 / 36.2 | 45.5 / 31.1 | 55.1 / 38.9 | 28.4 / 19.6 | 49.9 / 34.9 | 46.2 / 32.1 |
| InternVL2.5 [ 49 ] | SBERT | 39.4 / 27.4 | 31.2 / 21.4 | 45.0 / 31.1 | 29.9 / 20.5 | 42.0 / 29.5 | 37.5 / 26.0 |
| Qwen2.5-VL (7B) [ 50 ] | SBERT | 42.3 / 29.5 | 45.4 / 31.7 | 45.4 / 31.8 | 38.5 / 26.7 | 40.9 / 28.4 | 42.5 / 29.6 |
| Qwen3.5 (9B) [ 51 ] | SBERT | 36.5 / 25.3 | 37.1 / 25.8 | 42.9 / 30.0 | 34.8 / 23.9 | 37.0 / 25.6 | 37.6 / 26.1 |
| Method | Zero-Shot | EgoMCQ | Charades | |||
|---|---|---|---|---|---|---|
| Intra Acc. | Inter Acc. | Faith. | Acc. | Faith. | ||
| Perception Encoder [ 56 ] | ✓ | 37.64 | 84.88 | N/A | 31.30 | N/A |
| SpLiCE [ 13 ] | ✓ | 34.60 | 79.04 | 70.36 | 26.94 | 29.70 |
| EXPLAIN-R [ 14 ] | ✓ | 30.10 | 76.17 | 63.70 | 21.73 | 25.92 |
| GRACE | ✓ | 35.43 | 81.35 | 77.86 | 25.99 | 35.15 |
| GRACE ( + distill. ) | ✓ | 35.50 | 82.94 | 80.48 | 25.97 | 39.53 |
| Method mAP ( ) Disent. ( ) Complet. ( ) L 0 Norm ( ) SpLiCE [ 13 ] 86.1 8.9 18.3 633.2 GRACE ( ) 72.5 53.4 93.8 28.5 GRACE ( + ) 73.3 53.9 93.8 27.8 | |
| Figure 3 : Cumulative Reconstruction Mass (CRM@k) for different sparsity regimes. is the average number of concepts predicted by SpLiCE. | Table 3 : Evaluation of the concepts disentanglement with DCI metrics [ 63 ] . We use linear probing to predict the video scenario, e.g., cooking or sports , from just the reconstruction weights. |
| Method | Concepts | Zero | Flickr30k I T | Flickr30k T I | COCO I T | COCO T I | ||||||||
| Vocab. | Shot | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | |
| Task Metrics | ||||||||||||||
| CLIP (ViT-B) [ 4 ] | N/A | ✓ | 67.17 | 88.90 | 93.80 | 63.60 | 86.78 | 92.26 | 51.90 | 81.26 | 90.24 | 47.48 | 77.10 | 87.08 |
| SpLiCE [ 13 ] | LAION | ✓ | 36.43 | 65.03 | 75.36 | 36.83 | 63.50 | 74.20 | 32.26 | 62.94 | 76.44 | 29.92 | 59.22 | 73.00 |
| EXPLAIN-R [ 14 ] | ImageNet | ✓ | 45.67 | 73.97 | 83.30 | 49.30 | 78.45 | 87.10 | 44.22 | 75.28 | 87.30 | 47.42 | 78.04 | 88.12 |
| GRACE | ImageNet | ✓ | 48.48 | 76.17 | 84.50 | 42.84 | 69.97 | 79.26 | 41.80 | 71.98 | 83.66 | 33.56 | 63.34 | 75.92 |
| Method | Concepts | Zero | ImageNet | Flowers102 | Food101 | |||
| Vocab. | Shot | Acc. | Faith. | Acc. | Faith. | Acc. | Faith. | |
| CLIP (ViT-B) [ 4 ] | N/A | ✓ | 61.92 | N/A | 63.39 | N/A | 80.88 | N/A |
| SpLiCE [ 13 ] | LAION | ✓ | 42.91 | 51.33 | - | 26.12 | - | 51.80 |
| EXPLAIN-R [ 14 ] | ImageNet | ✓ | 50.77 | 50.19 | 15.51 | 16.59 | 45.86 | 45.81 |
| GRACE | ImageNet | ✓ | 48.04 | 60.15 | 32.14 | 41.78 | 62.67 | 66.83 |
| GRACE ( + distill. ) | ImageNet | ✓ | 47.84 | 60.68 | 28.82 | 41.18 | 63.24 | 67.49 |
| Label Most activated Concepts volcano volcano, chile , view towards, indonesia , … rickshaw tokyo , flower delivery, bike, fast delivery, … cassette player casio g , radio station, cassette, … digital clock clock, casio g , temperature, heartbeat, … digital watch casio g , chronograph, clock, time, … oscilloscope ultrasound, heartbeat, synthesis, casio g , … | |
| Figure 4 : Transferability of the decomposition. We train GRACE on custom ImageNet folds with minimal concept overlap, demonstrating robust transfer under severe concept-distribution shifts. | Table 7 : Bias analysis of the decomposition. We report the most activated concepts for some representative ImageNet classes, showing interesting patterns, e.g., strong entity-location correlation. |
Appendix figures & tables9 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | AUPRC |
|---|---|
| BLIP-2 [ 16 ] | 0.098 |
| LLaVA-NeXT [ 70 ] | 0.302 |
| ConceptScope [ 32 ] | 0.399 |
| GRACE (TIPSv2) | 0.482 |
| Projection | EgoMCQ | ConVisBench | ||
|---|---|---|---|---|
| Intra Acc. | Inter Acc. | Faith. | / | |
| Identity | 32.59 | 75.93 | 67.08 | 53.9 / 38.4 |
| Linear | 36.08 | 81.33 | 79.94 | 48.3 / 34.2 |
| MLP | 36.15 | 81.12 | 80.37 | 47.9 / 33.9 |
| Linear (Orthogonal) - GRACE | 35.43 | 81.35 | 77.86 | 51.0 / 36.2 |
| Method | Captions | Concepts | ImageNet | Flowers102 | Food101 | |||
|---|---|---|---|---|---|---|---|---|
| Vocab. | Acc. | Faith. | Acc. | Faith. | Acc. | Faith. | ||
| GRACE | ✗ | ImageNet | 48.04 | 60.15 | 32.14 | 41.78 | 62.67 | 66.83 |
| GRACE | ✗ | LAION | 47.36 | 59.87 | 25.11 | 31.92 | 61.60 | 64.84 |
| GRACE | ✓ | ImageNet | 47.84 | 60.68 | 28.82 | 41.18 | 63.24 | 67.49 |
| GRACE | ✓ | LAION | 48.27 | 60.96 | 28.57 | 36.79 | 60.80 | 64.65 |
| Method | Captions | Concepts | Flickr30k I T | Flickr30k T I | COCO I T | COCO T I | ||||||||
| Vocab. | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | ||
| Task Metrics | ||||||||||||||
| GRACE | ✗ | ImageNet | 48.48 | 76.17 | 84.50 | 42.84 | 69.97 | 79.26 | 41.80 | 71.98 | 83.66 | 33.56 | 63.34 | 75.92 |
| GRACE | ✗ | LAION | 48.24 | 75.85 | 84.23 | 42.49 | 69.55 | 78.88 | 41.32 | 71.88 | 83.70 | 32.36 | 61.54 | 74.18 |
| GRACE | ✓ | ImageNet | 52.30 | 79.30 | 86.97 | 46.87 | 74.43 | 83.41 | 42.78 | 72.84 | 84.46 | 36.34 | 65.82 | 78.66 |
| GRACE | ✓ | LAION | 52.56 | 79.29 | 87.18 | 47.24 | 74.62 | 83.66 | 43.48 | 73.88 | 84.80 | 35.94 | 66.60 | 79.18 |
| Method | Evidence | EgoMCQ | Charades | ImageNet | Flowers102 | Food101 | Flickr30k | COCO | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc. | Faith. | Acc. | Faith. | Acc. | Faith. | Acc. | Faith. | Acc. | Faith. | R@1 | Faith. | R@1 | Faith. | ||
| EXPLAIN-R | Captions | 53.14 | 63.70 | 21.73 | 25.92 | 50.77 | 50.19 | 15.51 | 16.59 | 45.86 | 45.81 | 45.67 | 44.48 | 44.22 | 46.72 |
| EXPLAIN-R | Visual | 57.15 | 73.94 | 24.88 | 44.50 | 52.51 | 67.72 | 26.67 | 35.00 | 55.65 | 61.03 | 37.78 | 47.14 | 35.16 | 51.34 |
| GRACE | Visual | 58.39 | 77.86 | 25.99 | 35.15 | 48.04 | 60.15 | 32.14 | 41.78 | 62.67 | 66.83 | 48.48 | 55.24 | 41.80 | 54.78 |
Vision-Language Models
Vision-language models connect visual inputs with text for retrieval, question answering, generation, and grounded multimodal reasoning.
See all 769 papers