MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation

Bowen Zheng; Jintao Lin; Changheng Yi; Haoran Yang; Binxiao Huang; Chenyang Lei; Rui Liu; Xihui Liu

cs.CVOct 23, 2026

MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation

Authors: Bowen Zheng, Jintao Lin, Changheng Yi, Haoran Yang, Binxiao Huang, Chenyang Lei, Rui Liu, Xihui Liu

Organizations: Nanjing University · The University of Hong Kong · Huawei Research

Abstract

Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.

Figures & tables

Benchmark	Size	Entity Fusion	Multi-Element Extraction	Attribute Transfer	Infographic Manip.	Visual Reasoning	Global Eval.	Atomic Eval.	Metrics
DreamBooth [ 35 ]	75	✓	✗	$\triangle$	✗	✗	✓	✗	CLIP Fidelity
DreamOmni2 [ 48 ]	319	✓	✗	✓	$\triangle$	✗	✓	✗	Score-based
MICo-Bench [ 45 ]	897	✓	✓	✗	✗	✗	✓	✗	Score-based
MultiBanana [ 33 ]	3769	✓	✗	✓	$\triangle$	✗	✓	✗	Score-based
VTEdit-Bench [ 25 ]	24220	✓	✗	$\triangle$	✗	✗	✓	✗	Score-based
MDMR-Bench (Ours)	2390	✓	✓	✓	✓	✓	✓	✓	Checklist-based

Table 1: Comparison with other benchmarks. ✓ denotes dedicated coverage,

\triangle

denotes partial or incidental coverage, and ✗ denotes no coverage.

Figure 4: Benchmark statistics of MDMR-Bench.

Closed-source Models
Model	Following	Consistency	Composition	Minimality	Quality	Overall
GPT-Image-2 [ 32 ]	90.1	87.4	90.8	84.5	93.2	88.8
Nano Banana Pro [ 13 ]	91.2	84.6	83.9	80.4	87.0	85.1
Nano Banana 2 [ 14 ]	88.0	82.0	81.2	72.1	84.1	81.2
Seedream 5.0 [ 5 ]	86.2	74.0	81.1	71.5	84.3	78.8
Seedream 4.0 [ 38 ]	85.3	73.4	80.7	69.7	82.8	77.8

Table 2: Primary leaderboard under the default Gemini-3-Pro judge, in percent. GPT-5.4 results are reported separately in Appendix A.2 .

Model	Entity Fusion	Multi-Element Extraction	Visual Reasoning	Attribute Transfer	Infographic Manip.	Overall
Closed-source Models
GPT-Image-2 [ 32 ]	91.5	89.6	84.8	90.2	87.6	88.7
Nano Banana Pro [ 13 ]	84.9	87.2	85.6	83.3	84.7	85.1
Nano Banana 2 [ 14 ]	82.5	85.6	75.9	82.4	79.8	81.2
Seedream 5.0 [ 5 ]	77.9	81.3	83.6	78.6	72.5	78.8
Seedream 4.0 [ 38 ]	77.9	83.1	83.7	76.6	67.9	77.8

Table 3: Task-level results for eleven models under Gemini-3-Pro. Scores are percentages.

Judge	Agreement Ratio	Pearson $r$	Spearman $r$	Cohen’s $\kappa$
Gemini-3-Pro [ 9 ]	0.853	0.643	0.862	0.642
GPT-5.4 [ 31 ]	0.854	0.621	0.834	0.618
Doubao-1.8 [ 4 ]	0.849	0.617	0.784	0.617
Qwen3-VL-32B [ 1 ]	0.800	0.475	0.725	0.472

Table 4: Correlation between human and VLM judges.

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Closed-source Models
Model	Following	Consistency	Composition	Minimality	Quality	Overall
GPT-Image-2	89.99	81.90	92.04	76.59	95.67	86.37
Nano Banana Pro [ 13 ]	86.99	76.15	86.01	72.04	88.29	81.18
Nano Banana 2 [ 14 ]	85.80	74.34	83.25	65.24	88.00	78.41
Seedream 5.0 [ 5 ]	83.21	66.48	80.67	63.76	85.50	74.73
Seedream 4.0 [ 38 ]	84.38	66.99	81.46	64.10	87.35	75.54

Appendix

Table 5: Evaluation results using GPT-5.4 as the judge.

Task family	Cases
Entity Fusion	70
Attribute Transfer	60
Multi-Element Extraction	20
Infographic Manipulation	30
Visual Reasoning	20
Total	200

Appendix

Table 6: Task distribution of the 200 human-evaluated benchmark cases.

Dimension	Total items	Invalid	Error rate
Following	491	7	1.43%
Consistency	780	11	1.41%
Composition	634	6	0.95%
Minimality	584	9	1.54%
Quality	473	4	0.85%
Total	2,962	37	1.25%

Appendix

Table 7: Checklist validity audit: reported counts by evaluation dimension.

Model	Inputs	Following	Consistency	Composition	Minimality	Quality	Overall
Nano Banana Pro	Real	95.37	88.39	80.19	85.64	86.67	87.01
	Synthetic	95.18	88.68	85.54	80.01	88.48	87.44
Seedream 5.0	Real	91.88	77.16	84.62	78.83	83.37	82.93
	Synthetic	90.78	76.37	87.53	75.76	88.82	83.16

Appendix

Table 8: Paired input-source experiment on 100 cases. Scores are percentages under Gemini-3-Pro.

Model	Run 1	Run 2	Run 3	Mean	Std
Nano Banana Pro	87.2	86.8	84.7	86.2	1.37
Nano Banana 2	85.6	85.7	83.7	85.0	1.12
Seedream 4.0	83.1	83.2	83.2	83.2	0.05
Seedream 5.0	81.3	82.5	84.3	82.7	1.51

Appendix

Table 9: Reported Multi-Element Extraction scores for three independent generation and judge runs, in percent.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

Aug 17, 2026

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task types (e.g., "subject composition"), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators: Anchor (

f

), Disentangle (

g

), Apply (

\oplus

), and Compose (

C

). Any multi-reference prompt can then be represented as a compositional formula over these operators, whose structural complexity is quantified by the number of operator slots. Building on this formulation, we construct TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis for recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies in disentanglement (

g

) and attribute binding (

\oplus

) rather than scene-level composition (

C

), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench

Haoran Wang, Chaofan Ma, Ran Yi +1

Shanghai Jiao Tong University

Image Generation Image Generation Evaluation Multimodal Model Evaluation Compositional Image Generation

Ask PDF

Mar 26, 2026

MACRO: Advancing Multi-Reference Image Generation with Structured Long-Context Data

Generating images conditioned on multiple visual references is critical for real-world applications such as multi-subject composition, narrative illustration, and novel view synthesis, yet current models suffer from severe performance degradation as the number of input references grows. We identify the root cause as a fundamental data bottleneck: existing datasets are dominated by single- or few-reference pairs and lack the structured, long-context supervision needed to learn dense inter-reference dependencies. To address this, we introduce MacroData, a large-scale dataset of 400K samples, each containing up to 10 reference images, systematically organized across four complementary dimensions -- Customization, Illustration, Spatial reasoning, and Temporal dynamics -- to provide comprehensive coverage of the multi-reference generation space. Recognizing the concurrent absence of standardized evaluation protocols, we further propose MacroBench, a benchmark of 4,000 samples that assesses generative coherence across graded task dimensions and input scales. Extensive experiments show that fine-tuning on MacroData yields substantial improvements in multi-reference generation, and ablation studies further reveal synergistic benefits of cross-task co-training and effective strategies for handling long-context complexity. The dataset and benchmark will be publicly released.

Zhekai Chen, Yuqing Wang, Manyuan Zhang +1

HKU MMLab · Meituan

Image Generation Compositional Image Generation Image Generation Evaluation

Ask PDF

Jun 25, 2026

Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existing benchmarks fail to adequately evaluate complex MRIG scenarios, hindering further progress in this area. To better assess model performance on complex MRIG tasks, we introduce OmniRef-Bench, a benchmark that covers complex combinations of reference image types and a large number of reference images. Evaluations on OmniRef-Bench show that mainstream open-source models struggle in complex MRIG scenarios, and their performance deteriorates significantly as the number of mixed-type reference images increases. To address this issue, we propose DyRef, a two-stage training framework. In the first stage, supervised fine-tuning equips the model with the basic capability to handle complex MRIG tasks. In the second stage, we introduce Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS). DAR dynamically adjusts the optimization objective to improve performance when handling a large number of mixed-type reference images. DRS enlarges intra-group reward differences for more effective policy optimization. Experiments demonstrate that DyRef significantly improves the performance of open-source models on OmniRef-Bench and single-image editing benchmarks, demonstrating the effectiveness and generalization capability of our approach.

Wenwang Huang, Yusen Fu, Junjie Wang +6

Harbin Institute of Technology, Shenzhen · Independent Researcher · Shenzhen Loop Area Institute

Image Generation RL for Generative Models Image Generation Evaluation

Ask PDF

Abstract

Image Generation

Explore similar work

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

MACRO: Advancing Multi-Reference Image Generation with Structured Long-Context Data

Scaling Multi-Reference Image Generation with Dynamic Reward Optimization