TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State
Authors: Xiangyang Wu, Chen Ju, Xu Chen, Likang Peng, Jinsong Lan, Haihong Tang, Yang Hu
Organizations: Alibaba Group · Hangzhou International Innovation Institute, Beihang University · Sino-Italian Joint Research Laboratory on Prognostics and Health Management for Intelligent Maintenance, Beihang University
Multimodal retrieval increasingly supports agentic systems, where user requests can involve cross-level relations, combine multiple constraints, and evolve across dialogue turns. In these settings, a highly similar result may still violate the user's requirements, while retrieved content may directly inform downstream actions without human screening. We introduce TRIAD, a multimodal retrieval benchmark for evaluating how well retrieval systems satisfy these complex, evolving requests. TRIAD comprises three tracks: abstraction, testing bidirectional retrieval between visual instances and concepts at different hierarchical levels; constraints, testing the joint satisfaction of positive and negative conditions; and state, testing the integration of requirements across dialogue turns and the updating of earlier conditions when users revise their requests. The benchmark contains six subsets, each with 1,000 queries and a fixed pool of 1,000 candidates per query, evaluated using a common multi-positive mean average precision (mAP) protocol. Simpler reference tasks and hard negatives grouped by the requirements they violate help characterize specific retrieval difficulties. We evaluate 20 configurations of 17 systems and find that no system leads across all six subsets. Performance declines with hierarchical distance, and negative-only queries generally underperform positive-only queries with the same number of constraints. Across paired tasks expressing the same final intent, multi-turn retrieval yields lower mAP in 39 of 40 configuration-domain comparisons. The benefits of explicit reasoning also vary across models and tasks. These findings highlight the gap between retrieving similar content and satisfying a complete request.
Figures & tables
Figure 1: Overview of TRIAD. Cross-level, multi-constraint, and multi-turn retrieval test hierarchical relations, constraint composition, and intent updates. Green/red indicate satisfied/violated requirements. Dashed boxes show simpler diagnostic references.
Table 1: Design comparison with representative benchmarks. Extended criteria and comparisons appear in Appendix F .
Abstraction
Constraints
State
Model
I2C
C2I
Fashion
Cars
Fashion
Cars
Dual-tower encoders
CLIP ViT-L/14 ( Radford et al., 2021 )
0.318
0.224
0.185
0.065
0.180
0.062
SigLIP-2-so400m ( Tschannen et al., 2025 )
0.232
0.278
0.173
0.082
0.144
0.069
jina-clip-v2 ( Koukounas et al., 2024 )
0.354
0.401
0.123
0.052
0.117
0.050
Discriminative embedders
Table 2: mAP on six main subsets (higher is better). Bold/underlined values mark the best/second-best embedding configurations before rounding. The final reranking reference is excluded from these marks and family comparisons (Section 4.4 ).
Figure 2: Rank ranges across six subsets; vertical segments connect the best and worst ranks, with lower being better. The 17 displayed configurations follow main-table order, using Gen for UME-R1 and +T-CoT for Embed-RL. Ranks use scores before display rounding among all 20 configurations, with average ranks for ties.
Model
MMEB-V2 Image
I2C
C2I
B-Fashion
B-Cars
C-Fashion
C-Cars
WeMM-9B
1
4 ( ↓ 3)
1 (—)
1 (—)
1 (—)
1 (—)
1 (—)
Qwen3-VL-8B
2
6 ( ↓ 4)
2 (—)
2 (—)
4 ( ↓ 2)
2 (—)
5 ( ↓ 3)
Rzen-v2
3
1 ( ↑ 2)
3 (—)
3 (—)
2 ( ↑ 1)
3 (—)
2 ( ↑ 1)
Qwen3-VL-2B
4
7 ( ↓ 3)
8 ( ↓ 4)
4 (—)
10 ( ↓ 6)
11 ( ↓ 7)
12 ( ↓ 8)
B-Tokens-2B
5
8 ( ↓ 3)
10 ( ↓ 5)
6 ( ↓ 1)
9 ( ↓ 4)
6 ( ↓ 1)
6 ( ↓ 1)
Table 3: Rank comparison between MMEB-V2 Image and TRIAD (selected systems). Ranks are computed among the 13 systems in Table 2 shared with MMEB-V2; parentheses show positions gained or lost relative to MMEB-V2, and — denotes no change.
Figure 3: Hierarchical distance and retrieval direction. (a) mAP for Self and ancestors one and two levels away; (b) C2I mAP by concept depth; (c) I2C/C2I ranks of five representative configurations. Panels (a,b) show 17 configurations; the black dashed line is the median. Configuration selection and ranking follow Figure 2 .
Figure 4: (a) Query-weighted mAP by constraint count; (b) matched-count positive-only and negative-only queries (P/N: positive/negative conditions). Panels (a,b) average Fashion and Cars equally per configuration; dashed black lines mark configuration medians. (c) Top-10 entry rates by negative type, normalized by that type’s candidate count and averaged across configurations. The 17 configurations follow Figure 2 .
Configuration
Fashion
Cars
Single
Multi
Δ mAP
Single
Multi
Δ mAP
WeMM-Embedding-9B
0.547
0.498
−0.049
0.367
0.291
−0.076
RzenEmbed-v2
0.482
0.441
−0.041
0.298
0.272
−0.026
Qwen3-VL-Embedding-8B
0.508
0.454
−0.054
0.230
0.177
−0.053
Qwen3-VL-Embedding-2B
0.442
0.212
−0.230
0.157
0.085
−0.072
Embed-RL-2B + T-CoT
0.403
0.374
−0.030
0.221
0.195
−0.026
Table 4: Single-to-multi-turn mAP changes for selected configurations. Δ mAP is multi-turn minus single-turn, computed from unrounded means and rounded to three decimals. UME-R1 uses Gen and Embed-RL uses +T-CoT; all 20 configurations appear in Appendix C.4 .
Model
Abstraction
Constraints
State
I2C
C2I
Fashion
Cars
Fashion
Cars
UME-R1-7B (Disc)
0.538
0.577
0.436
0.249
0.321
0.195
Δ mAP
−0.001
+0.011
−0.077
−0.075
−0.086
−0.066
Embed-RL-4B (no CoT)
0.518
0.527
0.356
0.227
0.321
0.194
Δ mAP
+0.020
+0.049
+0.004
+0.055
+0.055
+0.065
Table 5: mAP before and after reasoning. Each pair reports the non-reasoning baseline followed by Δ mAP (reasoning minus non-reasoning). Full results and tests appear in Appendix C.5 .
Appendix figures & tables29 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 5: Construction flow. The abstraction, constraints, and state tracks start from hierarchical relations, joint conditions, and multi-turn state, respectively, and produce queries, positives, and typed negatives through filtering and annotation. They share potential-positive screening, a fixed 1,000-candidate evaluation protocol, and corresponding lower-complexity references.
Figure 6: Data and attribute distributions. (a) WordNet lexicographer categories of 1,317 concepts, with ten smaller categories grouped as Other; (b) concept depths of 1,000 C2I queries; (c) D1–D7 profiles, where P/N denote positive/negative conditions; (d) natural/conflict sessions crossed with two/three turns. Panels (c,d) each use 1,000 queries per domain, with identical proportions in Fashion and Cars. (e) The 1,950 Fashion conditions: eight fields shown separately and the remaining 20 grouped; shares below 5% are unlabeled. (f) Cars: positive conditions on the inner ring (1,050), negative on the outer ring (900), with consistent field colors; text:year has no negative conditions.
Fashion field
Values
Items
Fashion field
Values
Items
band_color
24
736
heel_type
4
593
band_material
4
736
leg_length
4
879
body_color
26
11,840
lens_color
19
370
bra_type
5
286
lens_tint
3
370
buckle_type
4
462
neckline
11
5,079
cap_type
6
103
pattern
19
6,398
Appendix
Table 6: Normalized value counts and applicable-item counts of condition attributes. Use proportions appear in Figure 6 (e,f), so use counts are not repeated here. Applicable-item counts reflect the category coverage of structured attributes, not their missingness; for example, heel_type is defined only for footwear. Visual attributes in this table are used to construct and verify retrieval conditions, not supplied as additional product text. Model-visible text fields are listed in Appendix B.3 .
A: Abstraction
B: Constraints
C: State
Type
I2C
C2I
Fashion
Cars
Fashion
Cars
Pos.
1.768
5.000
1.653
1.982
1.653
1.982
Sib.
9.053
150.872
–
–
–
–
P-V
–
–
4.539
3.714
4.539
3.714
N-V
–
–
2.384
2.552
2.384
2.552
PN-V
–
–
1.401
1.350
1.401
1.350
Appendix
Table 7: Mean candidate counts per query. Counts follow mutually exclusive construction labels; “–” indicates that a type is not used in a subset. Pos.: positives; Sib.: hierarchical-neighbor HN; P-V/N-V: violations of positive/negative conditions only; PN-V: both polarities violated; Imp-V: preserved-attribute violations ( partial_match ); Vis.: visually similar HN; Trap: the exclusive revision-trap type. Trap flags may also overlay existing HN types; including overlays, C–Fashion/Cars means are 2.249/3.342. The table retains original types without double counting.
Figure 7: Candidate similarity and pool scaling. (a) Cosine-similarity densities pooled across Fashion/Cars under a fixed Qwen3-VL-Embedding-8B: positives, negation-only-violation negatives, and random fillers, with shared Gaussian KDE bandwidth 0.025. (b) Pool-scaling mAP for the same 1,000 Fashion constraints queries; full is the query-specific eligible corpus. Qwen, Rzen, and Embed-RL denote Qwen3-VL-Embedding-8B, RzenEmbed-v2, and Embed-RL-4B + T-CoT.
Model
PyTorch
Transformers
Flash-Attn
PEFT
NumPy
Precision
CLIP / SigLIP-2
2.6.0
4.52.4
2.7.4
NI
1.26.4
–
jina-clip-v2
2.6.0
4.46.3
2.7.4
NI
1.26.4
–
WeMM-Embedding-9B
2.6.0
5.14.1
2.7.2.post1
NI
1.26.4
BF16
E5-Omni-7B
2.6.0
5.14.1
2.7.2.post1
NI
1.26.4
BF16
RzenEmbed-v2
2.5.1
4.49.0
2.7.4.post1
NI
2.2.6
–
Qwen3-VL-Embedding-8B / 2B
2.6.0
5.14.1
2.7.2.post1
NI
1.26.4
BF16
Appendix
Table 8: Software dependencies and experimental precision by model. NI: not installed; –: unrecorded.
Model
Abstraction
Constraints
State
I2C
C2I
Fashion
Cars
Fashion
Cars
CLIP ViT-L/14
0.318
0.281
0.090
0.016
0.092
0.014
SigLIP-2-so400m
0.233
0.381
0.100
0.035
0.068
0.022
jina-clip-v2
0.328
0.467
0.071
0.017
0.063
0.017
WeMM-Embedding-9B
0.490
0.687
0.464
0.318
0.411
0.249
E5-Omni-7B
0.341
0.627
0.246
0.115
0.110
0.035
Appendix
Table 9: Top-1 labeled-positive hit rate on the six main subsets (higher is better). Each value is a proportion over 1,000 queries; configurations and row order follow Table 2 .
Configuration
Self
Same
I2C
C2I
Fashion
Cars
CLIP-L/14
0.372
0.374
0.246
0.265
SigLIP-2
0.275
0.429
0.184
0.311
jina-clip-v2
0.550
0.506
0.426
0.381
WeMM-9B
0.801
0.809
0.353
0.560
E5-Omni-7B
0.641
0.738
0.322
0.465
Appendix
Table 10: mAP on Self and Same (higher is better). Self contains 1,000 queries per direction and Same contains 500 per domain, each with 1,000 candidates. Self has one positive per query, so its mAP equals MRR. Configurations and row order follow Table 2 ; bold marks the best result in each column.
Figure 8: Rank agreement across the six subsets (Kendall τ -b, 20 configurations, all measured). Colour depth and cell values are proportional; a lower τ means the model order produced by one subset is less substitutable by the other. The 15 pairs span 0.505–0.874 (mean 0.659), with within- and cross-track means of 0.747 and 0.637; ranks are computed before display rounding with ties averaged.
Fashion
Cars
Model
D1
D2
D3
D4
D5
D6
D7
D1
D2
D3
D4
D5
D6
D7
Rzen
0.68
0.49
0.62
0.25
0.45
0.50
0.23
0.45
0.27
0.47
0.08
0.27
0.30
0.12
Qwen-8B
0.69
0.50
0.66
0.31
0.46
0.51
0.32
0.38
0.22
0.34
0.08
0.20
0.20
0.11
Qwen-2B
0.70
0.41
0.62
0.19
0.41
0.42
0.19
0.32
0.16
0.18
0.06
0.13
0.13
0.07
OmniEmbed-3B
0.30
0.28
0.17
0.15
0.17
0.10
0.09
0.18
0.11
0.07
0.04
0.07
0.05
0.04
VLM2Vec-V2
0.49
0.34
0.40
0.21
0.28
0.29
0.14
0.32
0.15
0.28
0.06
0.15
0.19
0.07
Appendix
Table 11: mAP across the seven difficulty tiers in both domains. Both Embed-RL configurations use +T-CoT.
Control
Left / right
Right lower
Median Δ
Range of Δ
Count-matched
1P / 1N
32/34
−0.165
[−0.285,0.110]
2P / 2N
32/34
−0.213
[−0.423,0.021]
2P / 1P1N
28/34
−0.123
[−0.211,0.031]
Count + category/attributes
1P / 1N
31/34
−0.206
[−0.326,0.066]
2P / 2N
30/34
−0.217
[−0.457,0.035]
2P / 1P1N
28/34
−0.131
[−0.265,0.033]
Appendix
Table 12: Polarity comparisons with matched condition counts and category/attribute standardization. Differences are right minus left. We report the number of negative differences among 34 configuration–domain combinations, their median, and their range. No query-paired or significance claim is made.
Configuration
Fashion
Cars
Single
Multi
Δ mAP
Single
Multi
Δ mAP
CLIP-L/14
0.185
0.180
−0.005
0.065
0.062
−0.002
SigLIP-2
0.173
0.144
−0.029
0.082
0.069
−0.013
jina-clip-v2
0.123
0.117
−0.006
0.052
0.050
−0.002
WeMM-9B
0.547
0.498
−0.049
0.367
0.291
−0.076
E5-Omni-7B
0.364
0.223
−0.142
0.186
0.101
−0.084
Appendix
Table 13: Single-to-multi-turn changes for all 20 configurations. Single and multi denote the constraints and state tracks, respectively, with 1,000 paired queries per configuration–domain comparison. Δ mAP is multi-turn minus single-turn, rounded to three decimals. B/C candidate pools overlap highly but are not identical.
Model
Domain
Natural-2
Natural-3
Conflict-2
Conflict-3
Rzen
Fashion
+0.016
+0.003
-0.106 †
-0.060 †
Rzen
Cars
+0.011
+0.008
-0.063 †
-0.051 †
Qwen-8B
Fashion
+0.012
+0.002
-0.101 †
-0.117 †
Qwen-8B
Cars
-0.011
-0.007
-0.091 †
-0.098 †
Qwen-2B
Fashion
-0.208 †
-0.204 †
-0.257 †
-0.245 †
Qwen-2B
Cars
-0.061 †
-0.061 †
-0.089 †
-0.075 †
Appendix
Table 14: Paired AP change by session type and turn count: state minus the corresponding constraints-track result. The 17 representatives use UME-R1 Gen and Embed-RL-2B/4B + T-CoT, giving 34 configuration–domain combinations with four session groups each. † indicates a 95% query-bootstrap interval excluding zero. Natural and conflict contain different queries and are not paired against each other. The main-text 39/40 count uses all 20 configurations and has a different denominator.
Figure 9: Paired AP change in conflict sessions, stratified by whether the obsolete-value trap enters the Top 10 (relative to the paired constraints-track single-turn task). Only the three strongest configurations are shown; red dots are the trap-in-Top-10 stratum and hollow grey dots the trap-free stratum, values are the stratum’s mean AP change, and the small numbers below give the session count of each stratum (459 and 517 trap-constructed sessions in Fashion and Cars, respectively). The two strata are descriptive and not a randomized treatment comparison.
Task
UME-R1-7B
Embed-RL-2B
Embed-RL-4B
I2C
−0.001 [-0.006,+0.004]
+0.056† [+0.045,+0.067]
+0.020† [+0.010,+0.030]
C2I
+0.011† [+0.007,+0.016]
+0.053† [+0.045,+0.061]
+0.049† [+0.041,+0.058]
B–Fashion
−0.077† [-0.090,-0.066]
+0.048† [+0.035,+0.060]
+0.004 [-0.012,+0.020]
B–Cars
−0.075† [-0.084,-0.066]
+0.023† [+0.015,+0.032]
+0.055† [+0.043,+0.066]
C–Fashion
−0.086† [-0.098,-0.074]
+0.065† [+0.054,+0.077]
+0.055† [+0.040,+0.070]
C–Cars
−0.066† [-0.075,-0.057]
+0.025† [+0.018,+0.034]
+0.065† [+0.054,+0.075]
Appendix
Table 15: Reasoning-minus-non-reasoning Δ mAP, with 95% intervals from 3,000 query-bootstrap resamples in brackets. † marks p<0.05 for the two-sided Wilcoxon test after Holm correction over ten tasks per column. Each main subset and Self direction contains 1,000 queries; Same contains 500 per domain. B/C denote the constraints and state tracks.
Original scoring
Other-distance positives removed
Configuration
Level 1
Level 2
Δ
Level 1
Level 2
Δ [95% CI]
CLIP ViT-L/14
0.392
0.222
−0.171
0.415
0.268
−0.148[−0.181,−0.115]
SigLIP-2-so400m
0.324
0.125
−0.199
0.340
0.150
−0.191[−0.220,−0.161]
jina-clip-v2
0.477
0.200
−0.277
0.498
0.252
−0.247[−0.277,−0.216]
WeMM-Embedding-9B
0.644
0.297
−0.347
0.684
0.415
−0.270[−0.297,−0.242]
E5-Omni-7B
0.484
0.170
−0.314
0.507
0.217
−0.290[−0.321,−0.259]
Appendix
Table 16: Hierarchy-scoring sensitivity on the same 711 queries. Original scoring retains positives at the other distance in the ranking; adjusted scoring removes only those labeled positives and compresses subsequent ranks. Δ is two-level minus one-level mAP; brackets give the adjusted difference’s 95% bootstrap interval. All 17 paired Wilcoxon tests in each scoring family remain significant after Holm correction.
Figure 10: Three-point ladder over last-turn-only, full-history, and paired single-turn inputs. 11 ablation-panel configurations × 2 domains × 3 strata; every cell is mAP. The top row is Fashion and the bottom row Cars; the three columns are the overall, natural, and conflict strata, with the query count of each stratum in parentheses. Rows are ordered by the Fashion overall paired single-turn mAP and kept identical across all six panels. Last-turn-only and full-history inputs use exactly the same pool, whereas the paired single-turn reference uses a highly overlapping but non-identical constraints-track pool and is therefore a task-level reference only. Across all 22 cells of the overall and natural columns, last-turn-only is below full history; 10 cells in the conflict column reverse this and are marked with hollow red squares.
Figure 11: Complete panel of the query-side single-modality ablation. 11 ablation-panel configurations × 2 tracks × 2 domains × 4 input variants, i.e. 176 cells of 1,000 queries each. “full” is the complete input, “text” keeps only the condition/dialogue text, “image” keeps only the reference image, and “refmeta” removes the image but restores the reference product text attributes; candidate-side inputs are unchanged throughout. Cells show mAP; color scales are normalized within each domain and are not used for cross-domain comparisons. Red boxes mark the 10 cells where refmeta exceeds full. All four panels share the row order of (a), sorted by its full values, so a configuration’s full rank within the other panels may differ. Models that must receive an image also have a gray-image control, which is not shown here.
Configuration
Image + leaf name
Leaf name only
Δ mAP
Qwen3-VL-Embedding-8B
0.5523
0.4980
−0.0543†
Qwen3-VL-Embedding-2B
0.5257
0.4699
−0.0558†
OmniEmbed-3B
0.4407
0.3506
−0.0901†
RzenEmbed-v2
0.6178
0.5290
−0.0887†
B-Tokens-2B
0.5002
0.3950
−0.1052†
VLM2Vec-2B
0.3470
0.3287
−0.0182
Appendix
Table 17: I2C with the leaf-class name retained and the original query image removed. Both inputs use the same 711 queries, query-specific candidate pools, and positive labels. mAP scores positives at both distances jointly; Δ is image-removed minus full input. † denotes paired Wilcoxon p<0.05 after Holm correction across the overall, one-level, and two-level contrasts within each configuration. The variant retains the I2C instruction; UME-R1 uses its text interface’s blank-image placeholder, without the original query image, and Embed-RL generates external reasoning from the leaf-class name. Means and differences are reported separately at archived precision, so subtraction of displayed values may differ in the last digit.
Subset
best pooled system †
Qwen embed. base
Top-30 reranked
reranked − base
positive coverage@30
oracle @30
A–I2C
0.569
0.494
0.607
+0.113
0.865
0.868
A–C2I
0.638
0.637
0.674
+0.037
0.869
0.877
B–Fashion
0.547
0.508
0.566
+0.058
0.884
0.887
B–Cars
0.367
0.230
0.327
+0.097
0.657
0.663
C–Fashion
0.498
0.454
0.516
+0.062
0.891
0.894
C–Cars
0.291
0.177
0.258
+0.081
0.616
0.623
Appendix
Table 18: Top-30 reranking results, paired analyses, and the recall-limited oracle upper bound on all 1,000 queries per subset. Qwen3-VL-Embedding-8B performs first-stage retrieval and Qwen3-VL-Reranker-8B reranks the Top-30. All six subsets show significant improvements under two-sided Wilcoxon signed-rank tests with Holm correction ( pHolm<10−11 ). Positive coverage@30 is the query-average fraction of positives recalled; oracle@30 ideally orders the same Top-30, providing a recall-limited upper bound. Values other than coverage are full-pool mAP or mAP differences. † denotes the best configuration for that subset in Table 2 , used only as a descriptive reference; paired tests compare Qwen first-stage retrieval with its own reranked output. Human relevance ratings on ten-candidate sets appear in Appendix E.2 and do not represent a human ceiling for the full 1,000-candidate pool.
Figure 12: Resolution of the benchmark. (a) Median 95% query-bootstrap half-width for each subset, computed with 3,000 resamples; the black line is the median over subsets and the dotted boundaries show their range. All six subsets summarize the 20 configurations in the main table using current query-level records. (b) Minimum detectable constant paired mAP improvement at 80% power, estimated with 1,000 Monte Carlo resamples of a representative adjacent-ranked pair per subset, two-sided Wilcoxon tests, and one six-subset Holm family. The red dashed line in both panels is the median gap (0.012) of the six selected adjacent pairs.
System
R1
R2
R3
R4
R5
Dedup.
CLIP ViT-L/14
∙
∙
∘
∘
∘
None
SigLIP-2-so400m
∙
∙
∘
∘
∘
None
jina-clip-v2
∙
∙
∘
∘
∘
None
RzenEmbed-v2
∙
∙
∘
∘
∘
Partial
Qwen3-VL-Embedding-8B
∙
∙
∘
∘
∘
Partial
Qwen3-VL-Embedding-2B
∙
∙
∘
∘
∘
Partial
Appendix
Table 19: Public-source exposure audit for 17 systems. ∙ : confirmed use at the source-dataset level; grey ∙ : cannot exclude; ∘ : content constructed in this study, unpublished before submission and not used to train the evaluated models. R1/R2 concern source images and labels/ontology; R3–R5 concern benchmark-specific queries, role assignments, and candidate combinations. Deduplication disclosure does not establish removal of TRIAD overlaps. Source materials for WeMM and E5-Omni were rechecked on September 20, 2026.
Figure 13: Qualitative examples across the three tracks. All panels show selected original candidate ranks from Qwen3-VL-Embedding-8B; green denotes annotated positives and red negatives. (a) Abstraction: the query contains an image and its leaf-class name. Toilet and room are ancestors one and two levels above the query; toiletry is an ordinary negative. Concept definitions are excerpted for display. (b) Constraints: the query requires sleeveless and not white. The white vest violates only the color exclusion, whereas the olive positive satisfies the full request; both retain brand, gender, and neckline. (c) State: the user revises grey to yellow. Both the old-value trap and the yellow positive retain Chevrolet and coupe. Queries and dialogue turns are reproduced verbatim; attributes follow the dataset annotations, and unshown ranks are omitted. Warm highlighting marks the exclusion phrase and superseded value; green highlighting marks the active value after revision.
Subset
No issue found
Concerns without impact on understanding
May affect understanding
B–Fashion
17
0
3
B–Cars
17
3
0
C–Fashion
16
2
2
C–Cars
17
3
0
Total
67
8
5
Appendix
Table 20: Human query-quality review of 20 tasks per subset. The three mutually exclusive categories distinguish whether an issue is found and its impact on understanding the requirements.
Domain
Correct
Incorrect
Not visible/ undetermined
Determinate accuracy
Not visible/ undetermined rate
Fashion
352
3
6
99.15%
1.66%
Cars
155
3
2
98.10%
1.25%
Total
507
6
8
98.83%
1.54%
Appendix
Table 21: Human visual-attribute verification of 40 images per domain: 361 attribute judgments for Fashion and 160 for Cars. Determinate accuracy excludes not-visible/undetermined items; their rate uses all attributes as the denominator.
Subset
n
Human mAP
Reranker mAP
Difference
A-I2C
20
0.5512
0.6288
−0.0776
A-C2I
20
0.7280
0.7657
−0.0377
B-Fashion
20
0.7021
0.5725
+0.1297
B-Cars
20
0.6610
0.4369
+0.2241
C-Fashion
20
0.6750
0.5108
+0.1641
C-Cars
20
0.7343
0.4160
+0.3183
Appendix
Table 22: Human and reranker comparison on identical ten-candidate sets. One human annotator rates all six subsets, with 20 score-complete tasks per row. Differences are human expected mAP minus reranker mAP; entries are descriptive means.
Benchmark
Cross-level retrieval
Condition control
State updates
Multi-positive scoring
Typed negatives
Low-complexity reference
M-BEIR
—
Partial a
—
Partial b
—
—
MMEB
—
Partial a
Partial d
—
—
—
MMEB-V2
—
Partial a
Partial d
Partial b
—
—
CIRCO
—
Partial a
—
✓
—
—
GeneCIS
—
Partial a
—
—
✓
—
NegBench
—
✓
—
—
Partial c
✓
Appendix
Table 23: Extended design comparison with related benchmarks.
Model/configuration
MMEB image
I2C
C2I
B Fashion
B Cars
C Fashion
C Cars
WeMM-Embedding-9B
1
4 ( ↓ 3)
1 (—)
1 (—)
1 (—)
1 (—)
1 (—)
Qwen3-VL-Embedding-8B
2
6 ( ↓ 4)
2 (—)
2 (—)
4 ( ↓ 2)
2 (—)
5 ( ↓ 3)
RzenEmbed-v2
3
1 ( ↑ 2)
3 (—)
3 (—)
2 ( ↑ 1)
3 (—)
2 ( ↑ 1)
Qwen3-VL-Embedding-2B
4
7 ( ↓ 3)
8 ( ↓ 4)
4 (—)
10 ( ↓ 6)
11 ( ↓ 7)
12 ( ↓ 8)
B-Tokens-2B
5
8 ( ↓ 3)
10 ( ↓ 5)
6 ( ↓ 1)
9 ( ↓ 4)
6 ( ↓ 1)
6 ( ↓ 1)
E5-Omni-7B
6
11 ( ↓ 5)
6 (—)
7 ( ↓ 1)
6 (—)
9 ( ↓ 3)
10 ( ↓ 4)
Appendix
Table 24: Complete rankings within the same 13-system set. Each cell gives rank and change relative to MMEB image ranking; B denotes constraints and C denotes state. Leaderboard mapping: RzenEmbed-v2 maps to RzenEmbed-v2-7B, B-Tokens-2B to Btoks, and E5-Omni-7B to e5-omni-7B. VLM2Vec-7B/2B map to VLM2Vec-V1-Qwen2VL-7B/2B, VLM2Vec-V2.0-2B to VLM2Vec-V2.0-Qwen2VL-2B, and OmniEmbed-3B to omni-embed-nemotron-3b. Other entries use the model names shown; UME-R1 uses Gen and Embed-RL uses +T-CoT.
TRIAD subset
Shared systems (n=13)
Without external T-CoT (n=11)
I2C
0.582
0.727
C2I
0.775
0.809
B-Fashion
0.934
0.991
B-Cars
0.681
0.800
C-Fashion
0.703
0.800
C-Cars
0.560
0.718
Appendix
Table 25: Spearman correlations between MMEB-V2 image rankings and TRIAD rankings.
Shanghai Jiao Tong University · Institute of Digital Twin, Eastern Institute of Technology, Ningbo · Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative +1
College of Computer Science, Sichuan University, Chengdu, China · National Key Laboratory of Fundamental Algorithms and Models for Engineering Simulation, Sichuan University, China