TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State
Organizations: Alibaba Group · Hangzhou International Innovation Institute, Beihang University · Sino-Italian Joint Research Laboratory on Prognostics and Health Management for Intelligent Maintenance, Beihang University
Abstract
Multimodal retrieval increasingly supports agentic systems, where user requests can involve cross-level relations, combine multiple constraints, and evolve across dialogue turns. In these settings, a highly similar result may still violate the user's requirements, while retrieved content may directly inform downstream actions without human screening. We introduce TRIAD, a multimodal retrieval benchmark for evaluating how well retrieval systems satisfy these complex, evolving requests. TRIAD comprises three tracks: abstraction, testing bidirectional retrieval between visual instances and concepts at different hierarchical levels; constraints, testing the joint satisfaction of positive and negative conditions; and state, testing the integration of requirements across dialogue turns and the updating of earlier conditions when users revise their requests. The benchmark contains six subsets, each with 1,000 queries and a fixed pool of 1,000 candidates per query, evaluated using a common multi-positive mean average precision (mAP) protocol. Simpler reference tasks and hard negatives grouped by the requirements they violate help characterize specific retrieval difficulties. We evaluate 20 configurations of 17 systems and find that no system leads across all six subsets. Performance declines with hierarchical distance, and negative-only queries generally underperform positive-only queries with the same number of constraints. Across paired tasks expressing the same final intent, multi-turn retrieval yields lower mAP in 39 of 40 configuration-domain comparisons. The benefits of explicit reasoning also vary across models and tasks. These findings highlight the gap between retrieving similar content and satisfying a complete request.
Figures & tables
| Dimension | MMEB-V2 | NegBench | ChatSearch | TRIAD (ours) |
|---|---|---|---|---|
| Targets | Classification, QA, retrieval | Object presence / absence | Dialogue-specified images | Instances higher/lower concepts; products |
| Conditions | Task-specific; composed retrieval | Positive, negative, hybrid | Descriptions and feedback | 1–3 conditions; 7 polarity mixes; preserved attributes |
| State | Context tasks (e.g., VisDial) | Single-turn | Single/multi-turn; text/image–text | Accumulate/replace; matched single turn |
| Diagnostics | Task-specific | Negation distractors; presence/absence pairs | Context-format comparisons | Typed violations and traps; Self/Same/single-turn |
| Abstraction | Constraints | State | ||||
| Model | I2C | C2I | Fashion | Cars | Fashion | Cars |
| Dual-tower encoders | ||||||
| CLIP ViT-L/14 ( Radford et al., 2021 ) | 0.318 | 0.224 | 0.185 | 0.065 | 0.180 | 0.062 |
| SigLIP-2-so400m ( Tschannen et al., 2025 ) | 0.232 | 0.278 | 0.173 | 0.082 | 0.144 | 0.069 |
| jina-clip-v2 ( Koukounas et al., 2024 ) | 0.354 | 0.401 | 0.123 | 0.052 | 0.117 | 0.050 |
| Discriminative embedders | ||||||
| Model | MMEB-V2 Image | I2C | C2I | B-Fashion | B-Cars | C-Fashion | C-Cars |
|---|---|---|---|---|---|---|---|
| WeMM-9B | 1 | 4 ( 3) | 1 (—) | 1 (—) | 1 (—) | 1 (—) | 1 (—) |
| Qwen3-VL-8B | 2 | 6 ( 4) | 2 (—) | 2 (—) | 4 ( 2) | 2 (—) | 5 ( 3) |
| Rzen-v2 | 3 | 1 ( 2) | 3 (—) | 3 (—) | 2 ( 1) | 3 (—) | 2 ( 1) |
| Qwen3-VL-2B | 4 | 7 ( 3) | 8 ( 4) | 4 (—) | 10 ( 6) | 11 ( 7) | 12 ( 8) |
| B-Tokens-2B | 5 | 8 ( 3) | 10 ( 5) | 6 ( 1) | 9 ( 4) | 6 ( 1) | 6 ( 1) |
| Configuration | Fashion | Cars | ||||
|---|---|---|---|---|---|---|
| Single | Multi | mAP | Single | Multi | mAP | |
| WeMM-Embedding-9B | 0.547 | 0.498 | 0.367 | 0.291 | ||
| RzenEmbed-v2 | 0.482 | 0.441 | 0.298 | 0.272 | ||
| Qwen3-VL-Embedding-8B | 0.508 | 0.454 | 0.230 | 0.177 | ||
| Qwen3-VL-Embedding-2B | 0.442 | 0.212 | 0.157 | 0.085 | ||
| Embed-RL-2B + T-CoT | 0.403 | 0.374 | 0.221 | 0.195 | ||
| Model | Abstraction | Constraints | State | |||
|---|---|---|---|---|---|---|
| I2C | C2I | Fashion | Cars | Fashion | Cars | |
| UME-R1-7B (Disc) | 0.538 | 0.577 | 0.436 | 0.249 | 0.321 | 0.195 |
| mAP | ||||||
| Embed-RL-4B (no CoT) | 0.518 | 0.527 | 0.356 | 0.227 | 0.321 | 0.194 |
| mAP | ||||||
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
| Fashion field | Values | Items | Fashion field | Values | Items |
|---|---|---|---|---|---|
| band_color | 24 | 736 | heel_type | 4 | 593 |
| band_material | 4 | 736 | leg_length | 4 | 879 |
| body_color | 26 | 11,840 | lens_color | 19 | 370 |
| bra_type | 5 | 286 | lens_tint | 3 | 370 |
| buckle_type | 4 | 462 | neckline | 11 | 5,079 |
| cap_type | 6 | 103 | pattern | 19 | 6,398 |
| A: Abstraction | B: Constraints | C: State | ||||
|---|---|---|---|---|---|---|
| Type | I2C | C2I | Fashion | Cars | Fashion | Cars |
| Pos. | 1.768 | 5.000 | 1.653 | 1.982 | 1.653 | 1.982 |
| Sib. | 9.053 | 150.872 | – | – | – | – |
| P-V | – | – | 4.539 | 3.714 | 4.539 | 3.714 |
| N-V | – | – | 2.384 | 2.552 | 2.384 | 2.552 |
| PN-V | – | – | 1.401 | 1.350 | 1.401 | 1.350 |
| Model | PyTorch | Transformers | Flash-Attn | PEFT | NumPy | Precision |
|---|---|---|---|---|---|---|
| CLIP / SigLIP-2 | 2.6.0 | 4.52.4 | 2.7.4 | NI | 1.26.4 | – |
| jina-clip-v2 | 2.6.0 | 4.46.3 | 2.7.4 | NI | 1.26.4 | – |
| WeMM-Embedding-9B | 2.6.0 | 5.14.1 | 2.7.2.post1 | NI | 1.26.4 | BF16 |
| E5-Omni-7B | 2.6.0 | 5.14.1 | 2.7.2.post1 | NI | 1.26.4 | BF16 |
| RzenEmbed-v2 | 2.5.1 | 4.49.0 | 2.7.4.post1 | NI | 2.2.6 | – |
| Qwen3-VL-Embedding-8B / 2B | 2.6.0 | 5.14.1 | 2.7.2.post1 | NI | 1.26.4 | BF16 |
| Model | Abstraction | Constraints | State | |||
|---|---|---|---|---|---|---|
| I2C | C2I | Fashion | Cars | Fashion | Cars | |
| CLIP ViT-L/14 | 0.318 | 0.281 | 0.090 | 0.016 | 0.092 | 0.014 |
| SigLIP-2-so400m | 0.233 | 0.381 | 0.100 | 0.035 | 0.068 | 0.022 |
| jina-clip-v2 | 0.328 | 0.467 | 0.071 | 0.017 | 0.063 | 0.017 |
| WeMM-Embedding-9B | 0.490 | 0.687 | 0.464 | 0.318 | 0.411 | 0.249 |
| E5-Omni-7B | 0.341 | 0.627 | 0.246 | 0.115 | 0.110 | 0.035 |
| Configuration | Self | Same | ||
|---|---|---|---|---|
| I2C | C2I | Fashion | Cars | |
| CLIP-L/14 | 0.372 | 0.374 | 0.246 | 0.265 |
| SigLIP-2 | 0.275 | 0.429 | 0.184 | 0.311 |
| jina-clip-v2 | 0.550 | 0.506 | 0.426 | 0.381 |
| WeMM-9B | 0.801 | 0.809 | 0.353 | 0.560 |
| E5-Omni-7B | 0.641 | 0.738 | 0.322 | 0.465 |
| Fashion | Cars | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Model | D1 | D2 | D3 | D4 | D5 | D6 | D7 | D1 | D2 | D3 | D4 | D5 | D6 | D7 |
| Rzen | 0.68 | 0.49 | 0.62 | 0.25 | 0.45 | 0.50 | 0.23 | 0.45 | 0.27 | 0.47 | 0.08 | 0.27 | 0.30 | 0.12 |
| Qwen-8B | 0.69 | 0.50 | 0.66 | 0.31 | 0.46 | 0.51 | 0.32 | 0.38 | 0.22 | 0.34 | 0.08 | 0.20 | 0.20 | 0.11 |
| Qwen-2B | 0.70 | 0.41 | 0.62 | 0.19 | 0.41 | 0.42 | 0.19 | 0.32 | 0.16 | 0.18 | 0.06 | 0.13 | 0.13 | 0.07 |
| OmniEmbed-3B | 0.30 | 0.28 | 0.17 | 0.15 | 0.17 | 0.10 | 0.09 | 0.18 | 0.11 | 0.07 | 0.04 | 0.07 | 0.05 | 0.04 |
| VLM2Vec-V2 | 0.49 | 0.34 | 0.40 | 0.21 | 0.28 | 0.29 | 0.14 | 0.32 | 0.15 | 0.28 | 0.06 | 0.15 | 0.19 | 0.07 |
| Control | Left / right | Right lower | Median | Range of |
|---|---|---|---|---|
| Count-matched | 1P / 1N | 32/34 | ||
| 2P / 2N | 32/34 | |||
| 2P / 1P1N | 28/34 | |||
| Count + category/attributes | 1P / 1N | 31/34 | ||
| 2P / 2N | 30/34 | |||
| 2P / 1P1N | 28/34 |
| Configuration | Fashion | Cars | ||||
|---|---|---|---|---|---|---|
| Single | Multi | mAP | Single | Multi | mAP | |
| CLIP-L/14 | 0.185 | 0.180 | 0.065 | 0.062 | ||
| SigLIP-2 | 0.173 | 0.144 | 0.082 | 0.069 | ||
| jina-clip-v2 | 0.123 | 0.117 | 0.052 | 0.050 | ||
| WeMM-9B | 0.547 | 0.498 | 0.367 | 0.291 | ||
| E5-Omni-7B | 0.364 | 0.223 | 0.186 | 0.101 | ||
| Model | Domain | Natural-2 | Natural-3 | Conflict-2 | Conflict-3 |
|---|---|---|---|---|---|
| Rzen | Fashion | +0.016 | +0.003 | -0.106 † | -0.060 † |
| Rzen | Cars | +0.011 | +0.008 | -0.063 † | -0.051 † |
| Qwen-8B | Fashion | +0.012 | +0.002 | -0.101 † | -0.117 † |
| Qwen-8B | Cars | -0.011 | -0.007 | -0.091 † | -0.098 † |
| Qwen-2B | Fashion | -0.208 † | -0.204 † | -0.257 † | -0.245 † |
| Qwen-2B | Cars | -0.061 † | -0.061 † | -0.089 † | -0.075 † |
| Task | UME-R1-7B | Embed-RL-2B | Embed-RL-4B |
|---|---|---|---|
| I2C | [-0.006,+0.004] | [+0.045,+0.067] | [+0.010,+0.030] |
| C2I | [+0.007,+0.016] | [+0.045,+0.061] | [+0.041,+0.058] |
| B–Fashion | [-0.090,-0.066] | [+0.035,+0.060] | [-0.012,+0.020] |
| B–Cars | [-0.084,-0.066] | [+0.015,+0.032] | [+0.043,+0.066] |
| C–Fashion | [-0.098,-0.074] | [+0.054,+0.077] | [+0.040,+0.070] |
| C–Cars | [-0.075,-0.057] | [+0.018,+0.034] | [+0.054,+0.075] |
| Original scoring | Other-distance positives removed | |||||
|---|---|---|---|---|---|---|
| Configuration | Level 1 | Level 2 | Level 1 | Level 2 | [95% CI] | |
| CLIP ViT-L/14 | 0.392 | 0.222 | 0.415 | 0.268 | ||
| SigLIP-2-so400m | 0.324 | 0.125 | 0.340 | 0.150 | ||
| jina-clip-v2 | 0.477 | 0.200 | 0.498 | 0.252 | ||
| WeMM-Embedding-9B | 0.644 | 0.297 | 0.684 | 0.415 | ||
| E5-Omni-7B | 0.484 | 0.170 | 0.507 | 0.217 | ||
| Configuration | Image + leaf name | Leaf name only | mAP |
|---|---|---|---|
| Qwen3-VL-Embedding-8B | 0.5523 | 0.4980 | |
| Qwen3-VL-Embedding-2B | 0.5257 | 0.4699 | |
| OmniEmbed-3B | 0.4407 | 0.3506 | |
| RzenEmbed-v2 | 0.6178 | 0.5290 | |
| B-Tokens-2B | 0.5002 | 0.3950 | |
| VLM2Vec-2B | 0.3470 | 0.3287 |
| Subset | best pooled system † | Qwen embed. base | Top-30 reranked | reranked base | positive coverage@30 | oracle @30 |
|---|---|---|---|---|---|---|
| A–I2C | 0.569 | 0.494 | 0.607 | +0.113 | 0.865 | 0.868 |
| A–C2I | 0.638 | 0.637 | 0.674 | +0.037 | 0.869 | 0.877 |
| B–Fashion | 0.547 | 0.508 | 0.566 | +0.058 | 0.884 | 0.887 |
| B–Cars | 0.367 | 0.230 | 0.327 | +0.097 | 0.657 | 0.663 |
| C–Fashion | 0.498 | 0.454 | 0.516 | +0.062 | 0.891 | 0.894 |
| C–Cars | 0.291 | 0.177 | 0.258 | +0.081 | 0.616 | 0.623 |
| System | R1 | R2 | R3 | R4 | R5 | Dedup. |
|---|---|---|---|---|---|---|
| CLIP ViT-L/14 | None | |||||
| SigLIP-2-so400m | None | |||||
| jina-clip-v2 | None | |||||
| RzenEmbed-v2 | Partial | |||||
| Qwen3-VL-Embedding-8B | Partial | |||||
| Qwen3-VL-Embedding-2B | Partial |
| Subset | No issue found | Concerns without impact on understanding | May affect understanding |
|---|---|---|---|
| B–Fashion | 17 | 0 | 3 |
| B–Cars | 17 | 3 | 0 |
| C–Fashion | 16 | 2 | 2 |
| C–Cars | 17 | 3 | 0 |
| Total | 67 | 8 | 5 |
| Domain | Correct | Incorrect | Not visible/ undetermined | Determinate accuracy | Not visible/ undetermined rate |
| Fashion | 352 | 3 | 6 | 99.15% | 1.66% |
| Cars | 155 | 3 | 2 | 98.10% | 1.25% |
| Total | 507 | 6 | 8 | 98.83% | 1.54% |
| Subset | Human mAP | Reranker mAP | Difference | |
|---|---|---|---|---|
| A-I2C | 20 | 0.5512 | 0.6288 | |
| A-C2I | 20 | 0.7280 | 0.7657 | |
| B-Fashion | 20 | 0.7021 | 0.5725 | |
| B-Cars | 20 | 0.6610 | 0.4369 | |
| C-Fashion | 20 | 0.6750 | 0.5108 | |
| C-Cars | 20 | 0.7343 | 0.4160 |
| Benchmark | Cross-level retrieval | Condition control | State updates | Multi-positive scoring | Typed negatives | Low-complexity reference |
|---|---|---|---|---|---|---|
| M-BEIR | — | Partial a | — | Partial b | — | — |
| MMEB | — | Partial a | Partial d | — | — | — |
| MMEB-V2 | — | Partial a | Partial d | Partial b | — | — |
| CIRCO | — | Partial a | — | — | — | |
| GeneCIS | — | Partial a | — | — | — | |
| NegBench | — | — | — | Partial c |
| Model/configuration | MMEB image | I2C | C2I | B Fashion | B Cars | C Fashion | C Cars |
|---|---|---|---|---|---|---|---|
| WeMM-Embedding-9B | 1 | 4 ( 3) | 1 (—) | 1 (—) | 1 (—) | 1 (—) | 1 (—) |
| Qwen3-VL-Embedding-8B | 2 | 6 ( 4) | 2 (—) | 2 (—) | 4 ( 2) | 2 (—) | 5 ( 3) |
| RzenEmbed-v2 | 3 | 1 ( 2) | 3 (—) | 3 (—) | 2 ( 1) | 3 (—) | 2 ( 1) |
| Qwen3-VL-Embedding-2B | 4 | 7 ( 3) | 8 ( 4) | 4 (—) | 10 ( 6) | 11 ( 7) | 12 ( 8) |
| B-Tokens-2B | 5 | 8 ( 3) | 10 ( 5) | 6 ( 1) | 9 ( 4) | 6 ( 1) | 6 ( 1) |
| E5-Omni-7B | 6 | 11 ( 5) | 6 (—) | 7 ( 1) | 6 (—) | 9 ( 3) | 10 ( 4) |
| TRIAD subset | Shared systems (n=13) | Without external T-CoT (n=11) |
|---|---|---|
| I2C | 0.582 | 0.727 |
| C2I | 0.775 | 0.809 |
| B-Fashion | 0.934 | 0.991 |
| B-Cars | 0.681 | 0.800 |
| C-Fashion | 0.703 | 0.800 |
| C-Cars | 0.560 | 0.718 |
Multimodal IR
Multimodal information retrieval finds relevant images, videos, documents, audio, or other media using queries that may combine different modalities.
See all 135 papers