Multimodal AI systems are evaluated by downstream task accuracy, but high accuracy does not mean the underlying data is coherent. A model can score well on Visual Question Answering (VQA) while its inputs contradict each other. We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model. MCS decomposes coherence into four dimensions, identity, spatial, semantic, and decision, with weights learned via Nelder-Mead optimization. We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining. Across three fusion architectures, MCS discriminates quality with higher sensitivity than task accuracy alone (Spearman rho = 0.093 vs. 0.071). Perturbation experiments confirm each dimension responds independently to its failure mode with zero cross-talk. MCS is lightweight, requires no human annotation, and tells you not just that something broke, but what broke.
Figures & tables
Figure 1: Visual Genome image #2320169. Left: what a DETR detector sees (“zebra” for each animal). Right: what the dataset says (“animal,” “bunch,” “stripe,” and “zebra is next to zebra” repeated ten times). A VQA (Visual Question Answering) system returns the correct answer. The benchmark score looks fine. The data is not.
Figure 2: The MCS framework. A multimodal event (left) is evaluated along four independent dimensions (center), each weighted by learned coefficients, producing a single diagnostic score (right). The decomposition identifies which dimension to address.
Property
Visual Genome
COCO 2017
Open Images V7
Images
108,077
150 (test)
130 (test)
Obj/image
21 avg
7.3 avg
10.2 avg
Relations/image
18 avg
—
3.1 avg
Descriptions
5.4M total
5/image
—
QA pairs
1.7M total
—
—
Table 1: Dataset statistics. Visual Genome serves as the primary evaluation set. COCO and Open Images validate cross-dataset transfer.
Perturbation
Target
Expected Effect
Object swap
IC
IC ↓ ; SpC, SC stable
Bbox shuffle
SpC
SpC ↓ ; IC, SC stable
Caption swap
SC
SC ↓ ; IC, SpC stable
Compound
All
All dimensions ↓
Table 2: Perturbation types and expected effects.
Figure 3: MCS dimension scores across three fusion architectures on Visual Genome ( n=1,000 ). Contract-enforced fusion leads on IC and SpC. DC is nearly identical across all three, illustrating that downstream accuracy alone cannot distinguish fusion quality.
Architecture
IC
SpC
SC
DC
MCS w
Naive
0.121
0.239
0.204
0.456
0.214
Contract
0.272
0.500
0.265
0.460
0.330
Foundation
0.219
0.360
0.238
0.423
0.271
Table 3: Dimension scores by architecture. All differences significant ( p<0.001 , Kruskal-Wallis) except DC ( p=0.002 ). Best results in bold .
Figure 4: Left: perturbation impact by type. Only the diagonal shows degradation, confirming zero cross-talk between dimensions. Right: degradation scales linearly with corruption rate. All dimensions exceed the 20% significance threshold at 50% perturbation.
Perturbation (50%)
Δ IC
Δ SpC
Δ SC
Δ MCS
Object swap
− 35.6%
0.0%
0.0%
− 6.5%
Bbox shuffle
0.0%
− 38.3%
0.0%
− 12.6%
Caption swap
0.0%
0.0%
− 26.7%
− 8.6%
Compound
− 37.2%
− 39.4%
− 25.6%
− 29.4%
Table 4: Perturbation results at 50% corruption. Each type degrades only its targeted dimension. Non-target dimensions show zero change.
Figure 5: Left: learned dimension weights. SC carries 72.2% of the signal, indicating that the text-image gap is the primary failure mode in Visual Genome. Right: VG-learned weights transfer to COCO and Open Images with zero retraining.
Figure 6: MCS (excluding DC) vs. Decision Coherence across 200 Visual Genome images. The composite outpredicts any single dimension (Spearman ρ=0.093 , p=0.003 ).