What Breaks in Multi-Image Schema Application? Factorial Oracle Interventions Reveal an Instance-Information Bottleneck

Yiliu Xu; Xinjin Li; Yu Ma

cs.CVOct 14, 2026

What Breaks in Multi-Image Schema Application? Factorial Oracle Interventions Reveal an Instance-Information Bottleneck

Authors: Yiliu Xu, Xinjin Li, Yu Ma

Organizations: Columbia University New York, United States

Abstract

Modern vision-language models can accept multi-image contexts, but aggregate accuracy rarely reveals where a multi-image reasoning pipeline fails. We study a controlled schema-application task in which a model must locate two target characters in a 16-image sequence, identify their visual attributes, consult an item-specific visual codebook, and compose two ordered nonce code names. We introduce a within-item 2-by-2 factorial Oracle design: Oracle-A supplies a verified textual schema, Oracle-B supplies a verified ordered target-appearance representation, and both interventions retain the original images and final question. Across 72 new items and 1,188 analyzed model responses, Oracle-B increased strict exact accuracy by 54.6 percentage points (95% confidence interval: 46.3 to 63.0) and outperformed Oracle-A by 60.6 points (Holm-adjusted p-value no greater than .0002). An auxiliary retrieval diagnostic reached only 49.1% ordered-appearance accuracy even after removing code-name generation. Providing both Oracle sources yielded 100% accuracy across all models. These results localize the dominant recoverable deficit to upstream ordered instance recovery rather than primarily to downstream schema use and composition.

Figures & tables

Fig. 1: Overview of the multi-image schema-application task. (a) One item contains Evidence A, Evidence B, and 14 fixed distractors; A/B order and slot pair are counterbalanced. (b) Oracle-A supplies the item-specific schema transcript, whereas Oracle-B supplies the ordered target appearances. Both intermediate representations feed the same code-composition step. (c) Crossing the interventions gives four within-item conditions with identical images, question, and expected answer. Task-relevant color identities are also named in text

Condition	16 images	Schema text	Instance text
Full ( $A0B0$ )	Yes	No	No
Oracle-A ( $A1B0$ )	Yes	Yes	No
Oracle-B ( $A0B1$ )	Yes	No	Yes
Oracle-A+B ( $A1B1$ )	Yes	Yes	Yes

TABLE I: Within-item factorial conditions

Design property	Count
Base items	72
Target roles	144
Occurrences per active appearance	9
Occurrences per target object	24
Items per evidence center	24
A-before-B / B-before-A items	36 / 36

TABLE II: Dataset balance

Provider	Model identifier	Setting
OpenAI	gpt-4.1-2025-04-14	temp. 0
Anthropic	claude-sonnet-5	thinking off
OpenAI	gpt-4.1-mini-2025-04-14	temp. 0

TABLE III: Model panel and inference settings

Contrast	Est.	95% CI	Raw $p$	Holm $p$
$\Delta_{B}$	+54.6 pp	[46.3, 63.0]	$\leq.0001$	$\leq.0002$
$\Delta_{B-A}$	+60.6 pp	[52.8, 68.1]	$\leq.0001$	$\leq.0002$

TABLE IV: Primary fixed-panel Oracle contrasts

Fig. 2: Factorial Oracle results under strict exact scoring. (a) Condition accuracy by model; error bars are 95% item-bootstrap intervals. (b) Oracle-B rescue (

\Delta_{B}

) and the Oracle-B versus Oracle-A asymmetry (

\Delta_{B-A}

) by model and for the equal-weighted fixed panel; horizontal bars are 95% item-cluster bootstrap intervals

Model	Strict exact accuracy (%)				Oracle contrasts (pp)			Residual (%)
Model	Full	O-A	O-B	O-AB	B rescue	A rescue	$A\!\times\!B$	Residual (%)
GPT-4.1	63.9	58.3	100.0	100.0	+36.1	$-$ 5.6	+5.6	0.0
Claude Sonnet 5	43.1	33.3	95.8	100.0	+52.8	$-$ 9.7	+13.9	0.0
GPT-4.1-mini	20.8	18.1	95.8	100.0	+75.0	$-$ 2.8	+6.9	0.0

TABLE V: Strict exact accuracy and probability-scale Oracle effects by model

Model	Full	O-A	O-B	O-AB	$\Delta_{B}$	$\Delta_{B-A}$
GPT-4.1	63.9	58.3	100.0	100.0	+36.1	+41.7
Claude Sonnet 5	48.6	52.8	98.6	100.0	+50.0	+45.8
GPT-4.1-mini	25.0	27.8	95.8	100.0	+70.8	+68.1
Fixed-panel mean	45.8	46.3	98.1	100.0	+52.3	+51.9

TABLE VI: Semantic-only sensitivity analysis (percent)

Fig. 3: Descriptive within-item transitions from Full to each single-Oracle condition under strict exact scoring. Each bar partitions the same model–item pairs into persistently wrong, rescued, lost, and persistently correct outcomes. The fixed-panel bar aggregates three equally sized model samples. Numbers inside segments are percentages and emphasize rescue and larger loss components

Fig. 4: Representation-level diagnostic summary. Thin gray lines show individual models, and the thick black line shows the equal-weighted fixed-panel mean. (a) Semantic final-task accuracy across the four factorial conditions. (b) Diagnostic ladder separating ordered instance recovery from downstream schema use and composition. Retrieval-B scores the ordered target appearance at pair, target, and attribute-atom levels; Retrieval-A scores visual-schema use when target appearances are provided

Multi-Image Reasoning

See all 15 papers

OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models
Mar 8, 2026Yusuke Tozaki, Hisashi MiyamoriVLM Evaluation VLM Reasoning
Division of Frontier Informatics, Kyoto Sangyo University, Kyoto, Japan
Composition, Not Conversation: VLMs Lose the Scene, Not the Thread
Sep 29, 2026L. D. M. S. Sai Teja, Ufaq Khan, N. Siva Gopala Krishna +5VLM Evaluation Visual Question Answering
NIT Silchar · MBZUAI · BML Munjal University +1
Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models
Jul 25, 2026Sultan Alshehri, Zhantao Yang, Han Zhang +1Vision-Language Models VLM Reasoning
Carnegie Mellon University, Pittsburgh, PA, USA.

Mar 8, 2026J/K move · Enter open · S save

OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models

Yusuke Tozaki, Hisashi Miyamori

Division of Frontier Informatics, Kyoto Sangyo University, Kyoto, Japan

VLM Evaluation VLM Reasoning

Open paper AskPDF arXiv

Vision-Language Models (VLMs) have advanced across multimodal benchmarks but still show clear gaps in ordinal number understanding, i.e., the ability to track relative positions and generalize to large indices. We present OrdinalBench, a diagnostic benchmark that standardizes ordinal number understanding as an evaluation task for VLMs. The core task is N-th object identification, defined by a starting reference and traversal rule. Task difficulty is controlled along three axes: (i) ordinal magnitude, from small numbers to extreme cases up to 300; (ii) arrangement complexity, from single loops to maze-like paths; and (iii) object count. The benchmark provides 39,000 question-answer pairs, each annotated with a ground-truth reasoning trajectory and balanced across difficulty levels for controlled large-scale testing. Beyond answer-only evaluation, our framework requires models to generate structured stepwise traces of the counting process and provides an open evaluation toolkit that measures both final accuracy and step-level path consistency. Zero-shot evaluations of GPT-5, Gemini 2.5 Flash Lite, Qwen2.5-VL, InternVL3.5, and Molmo reveal sharp degradation under large-ordinal and complex-path conditions, highlighting weak generalization despite strong scores on standard multimodal tasks. By framing ordinal number understanding as a core target, OrdinalBench provides a reproducible benchmark and diagnostic framework for developing VLMs with stronger sequential reasoning. All data and code are available at https://ordinalbench.github.io/

Abstract

Multi-Image Reasoning

Explore similar work

OrdinalBench: A Benchmark Dataset for Diagnosing Generalization Limits in Ordinal Number Understanding of Vision-Language Models

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models