Rethinking Multi-Image Re-Representation in Multi-Image Understanding

Gengyuan Zhang; Xiao Han; Xinyu Xie; Tong Liu; Volker Tresp

cs.CVSep 30, 2026

Rethinking Multi-Image Re-Representation in Multi-Image Understanding

Authors: Gengyuan Zhang, Xiao Han, Xinyu Xie, Tong Liu, Volker Tresp

Organizations: LMU Munich · MCML

Abstract

Multi-image understanding requires MLLMs not only to recognise the content of individual images, but also to organise visual evidence distributed across them. We study this problem through multi-image re-representation, viewing prompted Chain-of-Thought reasoning and agentic visual tool use as different ways of re-organising visual evidence during reasoning. We introduce Mosaic, a general-purpose multi-image visual harness that enables an MLLM to actively construct visual intermediates with ten composable image operations. We compare five re-representation settings on existing multi-image benchmarks and on MosaicBench, a new grounding-focused benchmark for fine-grained multi-image understanding. Our experiments show that the relative benefits of textual and visual re-representation are strongly task-dependent. Visual re-representation is particularly effective for tasks requiring precise visual evidence, including hypothesis testing, precision comparison, and orientation-sensitive reasoning, while tasks dominated by higher-level semantic content show smaller or less consistent gains. Building on this finding, we train MosaicAgent-8B to use Mosaic with reinforcement learning using only accuracy and format rewards. Without demonstration trajectories or rewards for specific tool-use, the agent learns to compose visual operations over multiple steps and exhibits diverse problem-solving patterns unpromptedly. Code and data will be released at https://github.com/gengyuanmax/Mosaic.

Figures & tables

Figure 1: Mosaic: a visual harness for multi-image understanding. (a) The model selects source or derived image assets, issues a structured operation call, and inspects the returned view. Each output is retained as a new asset for subsequent reuse. (b) Image operations transform individual views, combine evidence across images, and expose pixel-level differences.

Figure 2: Task-level comparison of re-representation methods. Rows show the five settings for Qwen3-VL-8B, Qwen3-VL-32B, and GPT-5.4. Cell values are accuracy (%) on BLINK and M4Bench tasks and on MosaicBench. from the mean of T- $\mathrm{Re}^{2}$ and V- $\mathrm{Re}^{2}$ for each model and task. Bars on the right show overall benchmark accuracy.

Figure 3: MosaicBench overview. Left: task types organised by their categories and domains. Center: representative multi-image examples. Right: distributions across task (top) and domains (bottom).

Open-weight baselines
Model	MosaicBench							M4Bench				Mantis	BLINK	MMIU
Model	Resol.	Orient.	Prec.Comp.	Hyp.Test	Ctx.Int.	Spat.Ref.	Overall	D.Diff	S.Comp	I.Comp	Overall	Overall	Overall	Overall
InternVL3.5-8B ( Wang et al., 2025b )	.438	.392	.390	.167	.463	.292	.363	.218	.649	.228	.373	.696	.581	.524
LLaVA-OV-1.5-8B ( An et al., 2025 )	.338	.375	.340	.267	.400	.233	.325	.000	.601	.254	.315	.567	.460	.417
GLM-4.6V-Flash ( GLM-V Team, 2025 )	.625	.475	.690	.300	.600	.342	.505	.606	.740	.197	.536	.737	.695	.627
MiniCPM-V-4.5 ( Yao et al., 2025 )	.500	.475	.550	.267	.500	.425	.463	.349	.664	.254	.466	.728	.612	.552

Table 1: Overall performance. We report performance on five benchmarks including MosaicBench. Bold scores mark the best reported result among the open-weight baselines and MosaicAgent-8B.

Training	MosaicBench	Mantis	BLINK	M4Bench	MMIU
Mosaic w/o RL	45.2 $\pm$ 1.44	81.9 $\pm$ 1.00	64.4 $\pm$ 0.40	51.3 $\pm$ 1.20	61.0
RL w/o Mosaic	53.0 $\pm$ 0.54	80.4 $\pm$ 1.47	58.2 $\pm$ 0.60	51.6 $\pm$ 0.80	56.8
RL w/ Mosaic	63.3 $\pm$ 1.10	81.6 $\pm$ 1.20	62.8 $\pm$ 1.00	61.8 $\pm$ 1.40	57.2

Table 2: RL training with and without the visual harness. Both RL policies use the same training suite. The backbone and harness-trained MosaicAgent-8B are evaluated under both T-

\mathrm{Re}^{2}

and V-

\mathrm{Re}^{2}

, while the CoT RL control is evaluated under T-

\mathrm{Re}^{2}

. Results are accuracy (%).

Figure 6: Profiling answerability throughout trajectories. (a) Counts exhibiting progression, self-correction, or overtooling. (b)–(d) Mean prefix answerability within each group of trajectories on normalised trajectory progress. Dashed lines for pre-RL and solid lines for post-RL; bands show

\pm 1

standard error. Dotted lines in (d) mark the mean stabilisation point where tool benefits saturate.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

ID	Task	Source	# Images
Resolution
1.1	Small-target cross-frame tracking	VisDrone2019-MOT	4
1.3	Distant sign reading	TT100K	1
1.5	Pathology zoom	CAMELYON16	1
1.6	Micro-scratch detection	MVTec AD	1
1.7	Dense small-object counting	CARPK	1

Appendix

Table 3: Task coverage of the 32-type construction. Tasks are grouped by their main visual challenge. “Images” denotes the number of input images per example.

Source Dataset	Annotations	Task IDs
VisDrone2019-MOT	Aerial video frames and object-track annotations	1.1, 4.2, 6.1
TT100K	Street images and traffic-sign annotations	1.3, 2.5
CAMELYON16	Whole-slide images and lesion annotations; slide backgrounds for contrast tasks	1.5, 5.5, 6.6
MVTec AD	Normal images, annotated defect images, and defect patches	1.6, 2.3, 2.8, 3.10, 3.11, 4.3, 5.6, 6.7
CARPK	Aerial parking-lot images and car annotations	1.7, 6.8, 6.9
PubLayNet	Document-page images	2.10

Appendix

Table 4: Sources used to construct the task collection. Task IDs follow Table 3 . Source subsets refer to the upstream datasets, not the final training and evaluation sets.

Field	Content
sample_id	Sample identifier.
prompt	Task question, input layout, and task-specific instructions.
images	Input-image references in the order used by the prompt.
task_type	Task identifier corresponding to Table 3 .
target	Ground-truth answer used for scoring.
metadata	Task-specific construction and audit information.

Appendix

Table 5: Shared sample schema.

Probe	Scope	Decision rule
Centroid / outlier	Matrix-based tasks	Select the transformation nearest to or furthest from the mean of the supplied transformations.
Pair member / singleton	Text-orientation recovery (2.10)	Select an angle from the pair separated by $180^{\circ}$ , or the angle outside that pair.
Central- $x$ / lowest- $y$	Drivable-zone point query (6.4)	Select the most horizontally central point or the lowest point in the image.

Appendix

Table 6: Shortcut probes based on transformation and spatial geometry. These heuristics do not inspect image content.

	Tool successes $k$ out of eight runs
	0	1	2	3	4	5	6	7	8	Total
Constructed examples	3116	1647	1268	995	769	756	715	930	2604	12,800
Incorrect without tools	2603	1429	1026	735	546	449	395	358	669	8,210
the training suite	–	–	1026	735	546	449	395	358	–	3,509
Held-out hard subset	2603	1429	–	–	–	–	–	–	–	4,032
Remaining examples	513	218	242	260	223	307	320	572	2604	5,259

Appendix

Table 7: Rollout outcomes and data selection. The second row contains examples answered incorrectly in the single no-tool run. The final three rows partition the constructed examples.

Available Visual Tools	MosaicBench	Mantis	BLINK	M4Bench	MMIU
Crop only	.539	.806	.624	.588	.583
Full toolset	.633 $\pm$ .011	.816 $\pm$ .012	.628 $\pm$ .010	.618 $\pm$ .014	.572

Appendix

Table 8: Visual tool ablation of Mosaic Agent. We compare the full visual toolset with a restricted setting where only the Crop tool is available.

Model	Setting	MosaicBench	Mantis	BLINK	M4Bench	MMIU
Qwen3-VL-8B	T- $\mathrm{Re}^{2}$	.452 $\pm$ .043	.819 $\pm$ .010	.644 $\pm$ .004	.513 $\pm$ .012	.610
Qwen3-VL-8B	V- $\mathrm{Re}^{2}$	.472 $\pm$ .015	.797 $\pm$ .016	.636 $\pm$ .002	.518 $\pm$ .003	.566
MosaicAgent-8B	T- $\mathrm{Re}^{2}$	.460 $\pm$ .012	.828 $\pm$ .004	.637 $\pm$ .004	.513 $\pm$ .014	.594
MosaicAgent-8B	V- $\mathrm{Re}^{2}$	.633 $\pm$ .011	.816 $\pm$ .012	.628 $\pm$ .010	.618 $\pm$ .014	.572

Appendix

Table 9: Comparison between Qwen3-VL-8B and Mosaic Agent under matched inference settings.

Explore similar work

Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding
Jan 12, 2026Jianghao Yin, Qingbin Li, Kun Sun +10Recent Vision-Language Models Theory-Of-Mind Reasoning
East China Normal University · ByteDance
MUSE: A Unified Agentic Harness for MLLMs
Jun 2, 2026Jianglin Lu, Hailing Wang, Xu Ma +4Multimodal Large Language Models Multimodal Reasoning
Northeastern University
Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning
May 27, 2026Xuanzhao Dong, Wenhui Zhu, Peijie Qiu +11Multimodal Large Language Models Recent Vision-Language Models
Arizona State University, Tempe, AZ, USA · Washington University in St. Louis, St. Louis, MO, USA · Clemson University, Clemson, SC, USA +3

Jan 12, 2026, cs.CVJ/K move · Enter open · S save

Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding

Jianghao Yin, Qingbin Li, Kun Sun, Cheng Ding+9

East China Normal University · ByteDance

Recent Vision-Language Models Theory-Of-Mind Reasoning Cognitive Science Offline Reinforcement Learning Action Expert

Open paper AskPDF arXiv

While Multimodal Large Language Models (MLLMs) excel at single-image understanding, they exhibit significantly degraded performance in multi-image reasoning scenarios. Multi-image reasoning presents fundamental challenges including complex inter-relationships between images and scattered critical information across image sets. Inspired by human cognitive processes, we propose a Cognition-Inspired Meta-Action Framework (CINEMA), which decomposes multi-image reasoning into five structured meta-actions: Global, Focus, Hint, Think, and Answer, explicitly modeling the sequential cognitive steps humans naturally employ. For cold-start training, we introduce a Retrieval-Based Tree Sampling strategy that generates high-quality meta-action trajectories to bootstrap the model with reasoning patterns. During reinforcement learning, we adopt a two-stage paradigm: an exploration phase with Diversity-Preserving Strategy to avoid entropy collapse, followed by an annealed exploitation phase with DAPO to gradually strengthen exploitation. To train our model, we construct a dataset of 56k cold-start and 58k reinforcement learning instances spanning multi-image, multi-frame, and single-image tasks. We conduct extensive evaluations on multi-image reasoning benchmarks, video understanding benchmarks, and single-image benchmarks, achieving competitive state-of-the-art performance on several key benchmarks. Our model surpasses GPT-4o on the MUIR and MVMath benchmarks and notably outperforms specialized video reasoning models on video understanding benchmarks, demonstrating the effectiveness and generalizability of our human cognition-inspired reasoning framework.

Abstract

Explore similar work

Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understanding

MUSE: A Unified Agentic Harness for MLLMs

Mags-RL: Wearing Multimodal LLMs a Magnifying Glass via Agentic Reinforcement Learning For Complex Scene Reasoning