cs.CLSep 30, 2026

Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis

Authors: Xia Hu, Brian Potetz, Chun-Ta Lu, Huanfen Yao, Leonidas Guibas, Zhicheng Wang, Howard Zhou, Pengfei Xing, +1 more

Organizations: Google DeepMind · Google Research · Stanford University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a cascading error from an upstream prerequisite. We propose a causal decomposition framework that isolates these two failure modes through controlled interventions on the prerequisite dependencies of each task. Our capability metrics (NC, IC, RC) score each task under unassisted, correct, or incorrect prerequisites to diagnose where failures arise; contribution metrics (N-Score, S-Score), adapted from probabilities of causation, quantify each prerequisite's necessity and sufficiency to determine why. We instantiate the framework in CADET, a diagnostic benchmark of 10 composite tasks decomposed into 46 unit tasks with over 33,000 human-annotated questions spanning perception, spatial, temporal, and cognitive categories. Diagnosing frontier MLLMs with our framework uncovers systematic patterns that end-to-end accuracy obscures. Capability-wise, supplying correct prerequisites eliminates 54% of errors on cognitive tasks, lifting them from weakest to above spatial and temporal. Prerequisite-wise, causal contributions are concentrated in a few critical prerequisites, and supplying the single most important one alone captures 84% of the gain from supplying all prerequisites.

Figures & tables

Figure 1 : An example CADET task ( left ) decomposed into 6 unit tasks formalized as an SCM ( middle ). Right : natural capability (NC) of each unit task ( top ), and their causal contribution ( NN / S(1,∅)S(1,\varnothing) ; bottom ) to final task-6, where supplying unit task-2 alone raises final accuracy from 0.27 to 0.65.
ContrastN-Score (Success drop from ⋯\cdots )S-Score (Failure recovery from ⋯\cdots )
(1,0)(1,0)corrupting XiX_{i} into a wrong answer in oracle env.fixing XiX_{i} into the correct answer in all-wrong env.
(1,∅)(1,\varnothing)leaving XiX_{i} unassisted in oracle env.providing the correct XiX_{i} alone in natural env.
(∅,0)(\varnothing,0)injecting a wrong answer for XiX_{i} in natural env.removing a wrong answer for XiX_{i} in all-wrong env.
(0,∅)(0,\varnothing)removing XiX_{i} ’s wrong answer in all-wrong env.injecting a wrong XiX_{i} as a scaffold in natural env.
Table 1: Operational semantics of N-Score, S-Score across 4 ordered state contrasts (a,a′)(a,a^{\prime}) .
Figure 2 : Overview of SCMs corresponding to 10 tasks in CADET.
Figure 3 : Natural(NC) and intrinsic(IC) capabilities on non-root unit tasks: (a) category-level NC and IC averaged over 12 models; (b–e) per-model NC and IC within each category.
Figure 6 : N/S(1,∅)N/S(1,\varnothing) across Pattern 1( a,b ) and Pattern 2 ( c,d ) tasks for top-4 SOTA ( a,c ) and remaining 8 Others ( b,d ) models ( sqrt(⋅)sqrt({\cdot}) -scaled axes, similar in Fig 5 ).
Figure 6
Figure 7
Appendix figures & tables26 assets

Supplementary material from the paper’s appendix.

Appendix

Task NameModalityCategory# InstancesAnswer Format
Music Note CountingImagePerception114Integer
Dice SumImagePerception130Integer
Map NavigationImageSpatial107Integer
MazeImageSpatial80Multiple-Choice
Surround LocalizationMulti-ImageSpatial154Integer
Multi-View SequencingMulti-ImageTemporal154Letter List
Appendix
Table 2 : Overview of the 10 overall evaluation tasks in CADET.
Task#Units#RootsDepth∣Pa(Y)∣|\mathrm{Pa}(Y)|Category#QuestionsHas Markov
Music Note Counting4223P/C1,163No
Dice Sum4223P/C1,919No
Map Navigation3122P/S/C918Yes
Maze2111P/S880Yes
Surround Localization6325P/S4,928No
Multi-View Sequencing6235P/S/T4,158No
Appendix
Table 3 : Structure of the task-level SCM of each evaluation task. – on multistep reasoning is due to it start from a Markov chain structure.
NodeUnit TaskCat.Answer Format#QParents
Music Note Counting
X1X_{1}Object IdentificationPMultiple-choice821—
X2X_{2}Object ClassificationPMultiple-choice114X1X_{1}
X3X_{3}Completeness VerificationPMultiple-choice114—
YYObject CountingCInteger114X1,X2,X3X_{1},X_{2},X_{3}
Dice Sum
Appendix
Table 4 : Unit task specification of the 10 evaluation tasks in CADET. YY denotes the overall task outcome; ↻\circlearrowright marks a Markov chain unit task.
Figure 11 : Prompt template for our evaluation under active causal interventions ( Ai∈{1,0}A_{i}\in\{1,0\} ). Under the natural state ( Ai=∅A_{i}=\varnothing ), only the target question and last paragraph of output format instructions are prompted.
Figure 12 : LLM-as-a-judge prompt template.
Figure 13 : Comparing NC on root unit tasks against NC and IC on non-root unit tasks, per category, averaged over the 12 models. Cognitive is not shown as it has no root unit task. The non-root bar stacks IC −- NC on top of NC, so its top is IC.
Figure 14 : Comparing NC on root unit tasks against NC and IC on non-root unit tasks for each of the 12 models. There are 16 root unit tasks and 30 non-root unit tasks. The non-root bar stacks IC −- NC on top of NC, so its top is IC. Models are ordered by overall non-root NC.
Figure 15 : Comparing NC and RC across the four categories on non-root unit tasks, averaged over the 12 models. Each bar shows NC with the signed NC/RC difference stacked out of it, so the boundary of the shaded band is RC.
Figure 16 : Comparing NC and RC for each of the 12 models within each category on non-root unit tasks. Circles mark NC and triangles mark RC; the shaded band spans the two and is colored by the sign of RC −- NC. The horizontal lines are the means over the 12 models.
 #unit tasksRootNon-root
Categoryrootnon-rootNCNCICIC −- NC
Perception1140.775 ±\pm 0.0770.590 ±\pm 0.1380.796 ±\pm 0.112+0.206
Spatial390.627 ±\pm 0.1000.556 ±\pm 0.0770.671 ±\pm 0.059+0.115
Temporal280.714 ±\pm 0.1190.460 ±\pm 0.0720.614 ±\pm 0.087+0.154
Cognitive09–0.406 ±\pm 0.0920.721 ±\pm 0.089+0.316
Appendix
Table 6 : NC on root unit tasks and NC, IC on non-root unit tasks, per category, with the number of unit tasks in each group. Entries are the unweighted mean over unit tasks, averaged over the 12 models, ±\pm the standard deviation across models.
ModelRoot NCNon-root NCNon-root ICNC(root) −{-} NC(non-root)NC(root) −{-} IC(non-root)
GPT-5.40.781 ±\pm 0.1080.612 ±\pm 0.1800.791 ±\pm 0.167+0.169−- 0.010
Gem 3.5F0.839 ±\pm 0.0990.595 ±\pm 0.2170.783 ±\pm 0.172+0.244+0.056
Gem 3.1P0.842 ±\pm 0.0900.577 ±\pm 0.2020.781 ±\pm 0.173+0.265+0.062
Gem 3F0.811 ±\pm 0.1190.528 ±\pm 0.2060.757 ±\pm 0.180+0.282+0.053
Qwen27B0.780 ±\pm 0.1320.515 ±\pm 0.2470.711 ±\pm 0.219+0.264+0.069
Qwen397B0.749 ±\pm 0.1440.487 ±\pm 0.2400.647 ±\pm 0.226+0.262+0.101
Appendix
Table 7 : NC on root unit tasks and NC, IC on non-root unit tasks for each of the 12 models. Entries are the unweighted mean over unit tasks ±\pm the standard deviation across unit tasks (16 root, 30 non-root). The last two columns are differences of the corresponding means.
CategoryNCICRCIC −- NCRC −- NC
Perception0.590 ±\pm 0.1380.796 ±\pm 0.1120.446 ±\pm 0.131+0.206−- 0.144
Spatial0.556 ±\pm 0.0770.671 ±\pm 0.0590.439 ±\pm 0.102+0.115−- 0.116
Temporal0.460 ±\pm 0.0720.614 ±\pm 0.0870.414 ±\pm 0.063+0.154−- 0.046
Cognitive0.406 ±\pm 0.0920.721 ±\pm 0.0890.445 ±\pm 0.060+0.316+0.040
Appendix
Table 8 : NC, IC and RC per category on the 30 non-root unit tasks. Entries are the unweighted mean over unit tasks, averaged over the 12 models, ±\pm the standard deviation across models.
 PerceptionSpatialTemporalCognitiveOverall
ModelNCICRCNCICRCNCICRCNCICRCNCICRC
GPT-5.40.636 ±\pm 0.1940.913 ±\pm 0.0870.507 ±\pm 0.2880.665 ±\pm 0.2020.747 ±\pm 0.2140.549 ±\pm 0.2130.560 ±\pm 0.1480.730 ±\pm 0.1810.537 ±\pm 0.2190.593 ±\pm 0.1920.835 ±\pm 0.0940.575 ±\pm 0.2740.612 ±\pm 0.1800.791 ±\pm 0.1670.548 ±\pm 0.232
Gem 3.5F0.722 ±\pm 0.1920.889 ±\pm 0.0420.599 ±\pm 0.1940.672 ±\pm 0.2310.754 ±\pm 0.2130.572 ±\pm 0.2520.564 ±\pm 0.1540.699 ±\pm 0.2020.481 ±\pm 0.1870.490 ±\pm 0.2320.839 ±\pm 0.0910.507 ±\pm 0.2190.595 ±\pm 0.2170.783 ±\pm 0.1720.532 ±\pm 0.212
Gem 3.1P0.761 ±\pm 0.1550.907 ±\pm 0.0270.678 ±\pm 0.1580.650 ±\pm 0.2170.740 ±\pm 0.2300.584 ±\pm 0.2070.515 ±\pm 0.1640.698 ±\pm 0.1750.450 ±\pm 0.2220.478 ±\pm 0.1740.839 ±\pm 0.0880.513 ±\pm 0.1940.577 ±\pm 0.2020.781 ±\pm 0.1730.540 ±\pm 0.206
Gem 3F0.664 ±\pm 0.2500.879 ±\pm 0.0720.439 ±\pm 0.3040.570 ±\pm 0.2680.710 ±\pm 0.2530.431 ±\pm 0.2400.524 ±\pm 0.1220.707 ±\pm 0.1630.422 ±\pm 0.2050.430 ±\pm 0.1540.796 ±\pm 0.1210.450 ±\pm 0.2230.528 ±\pm 0.2060.757 ±\pm 0.1800.435 ±\pm 0.222
Qwen27B0.682 ±\pm 0.1670.873 ±\pm 0.1190.418 ±\pm 0.2370.571 ±\pm 0.3080.663 ±\pm 0.2520.439 ±\pm 0.2490.515 ±\pm 0.2480.672 ±\pm 0.2100.466 ±\pm 0.2760.386 ±\pm 0.1590.722 ±\pm 0.2200.421 ±\pm 0.2130.515 ±\pm 0.2470.711 ±\pm 0.2190.438 ±\pm 0.233
Appendix
Table 9 : NC, IC and RC per model and category on the 30 non-root unit tasks, with Overall aggregating all 30. Entries are the unweighted mean over unit tasks ±\pm the standard deviation across unit tasks.
Figure 17 : Prerequisite N/SN/S -Scores under state contrasts (a) (1,0)(1,0) , (b) (0,∅)(0,\varnothing) , and (c) (∅,0)(\varnothing,0) , averaged across models. Dashed circles mark the highest- SS prerequisite within each task. Axes and quadrant thresholds follow Figure 5 .
Figure 18 : Prerequisite N/SN/S -Scores under (1,0)(1,0) (a–d), (0,∅)(0,\varnothing) (e–h), and (∅,0)(\varnothing,0) (i–l) by task pattern and model group. Large markers denote group means; grey markers indicate individual models.
Figure 19 : Pairwise Spearman rank correlation between models on NN -Scores (a–d) and SS -Scores (e–h) across prerequisites under the four state contrasts. Lines separate SOTA from Others.
Figure 20 : Prerequisite N/SN/S -Scores under the four state contrasts, averaged over evaluated models.
Figure 25
Figure 23 : Per-model NN - and SS -Scores for the parents of Map Nav. under the four state contrasts. The vertical line separates SOTA from Others.
Figure 24 : Same as Figure 23 , for Dice Sum.
Figure 25 : Same as Figure 23 , for Music Note.
Figure 26 : Same as Figure 23 , for Dashcam Count..
Figure 27 : Same as Figure 23 , for Multi-View Seq..
Figure 28 : Same as Figure 23 , for Street-View Ord..
Figure 29 : Same as Figure 23 , for Surround Loc..
Figure 30 : Same as Figure 23 , for Traffic Causation.

Explore similar work

CardsList

  1. STaD: Scaffolded Task Design for Identifying Compositional Skill Gaps in LLMs

    Apr 20, 2026Sungeun An, Swanand Ravindra Kadhe, Shailja Thakur +2Reasoning BenchmarkSkills

    IBM Research

  2. Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry

    Jun 11, 2026Jennifer Meng Lu, Ruochen Zhang, Isabelle Lee +3Large Language Models FailAdversarial Robustness

    Brown University · University of Southern California · Harvard University +1

  3. FailureScope: Cross-Regime Behavioral Diagnosis of Language Model Weaknesses

    Jun 3, 2026Nicholas SabanLarge Language Models FailClustering

    Patronus AI · University of California, Berkeley