TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

Haoran Wang; Chaofan Ma; Ran Yi; Lizhuang Ma

cs.CVAug 17, 2026

TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation

Authors: Haoran Wang, Chaofan Ma, Ran Yi, Lizhuang Ma

Organizations: Shanghai Jiao Tong University

Abstract

Despite recent advances in unified multimodal models for multi-reference image generation, existing benchmarks remain organized around predefined task types (e.g., "subject composition"), which are ill-suited to this combinatorial setting and lead to fragmented coverage, uncontrolled complexity, and little diagnostic value. Recognizing that diverse multi-reference tasks share a common set of atomic operations, we adopt a capability-oriented perspective and formalize four operators: Anchor ( $f$ ), Disentangle ( $g$ ), Apply ( $\oplus$ ), and Compose ( $C$ ). Any multi-reference prompt can then be represented as a compositional formula over these operators, whose structural complexity is quantified by the number of operator slots. Building on this formulation, we construct TRACE-Bench, comprising approximately 1,600 evaluation cases across slot counts 1--8, built from 631 formula templates and around 4,000 reference images spanning diverse artistic styles and real-world subjects. The formula structure directly drives an operator-aligned evaluation protocol for per-capability scoring and a diagnostic tree analysis for recursive failure localization. Evaluating 9 leading models reveals insights invisible to holistic scoring: the primary bottleneck lies in disentanglement ( $g$ ) and attribute binding ( $\oplus$ ) rather than scene-level composition ( $C$ ), with even the best model scoring only 0.74 on attribute fidelity. Project page: https://amuseum-whr.github.io/TraceBench

Figures & tables

Figure 1. Representative TRACE-Bench cases. Green/red tags indicate satisfied/failed requirements; red boxes localize failures. Representative multi-reference image-generation cases with reference images, prompts, generated results, and operator-aligned success and failure annotations.

Figure 2. A multi-reference request in TRACE-Bench is progressively resolved through Anchor , Disentangle , Apply , and Compose . A four-stage illustration showing how Anchor, Disentangle, Apply, and Compose progressively construct a multi-reference image-generation result.

Operator	Dimension	Criteria
$f:$ Anchor	Identity	Existence; appearance consistency with the reference
$g:$ Disentangle	Attribute Fidelity	Presence; consistency with the reference source
$\oplus:$ Apply	Binding	Carrier integrity; attribute exclusivity; natural integration
$C:$ Compose	Composition	Coexistence; relation satisfaction; spatial coherence; no duplication or leakage

Table 1. Operator-aligned evaluation dimensions.

Figure 3. Overview of the benchmark construction pipeline. Candidate images are first collected and filtered from multiple sources, then annotated through structured tagging. The tagged pool is then balanced through source-wise sampling and augmented with synthetic data. It is subsequently used for formula-template sampling and prompt construction. A pipeline diagram showing image collection and filtering, structured tagging, balanced sampling and synthetic augmentation, formula-template sampling, and prompt construction.

Figure 4. Representative operator-aligned evaluation examples from TRACE-Bench. Each row shows one benchmark case with its references, brief prompt, compositional formula, readable expression, generated result, and operator-level analysis. The examples illustrate how formula terms are mapped to capability-specific checks and how failures such as identity mismatch, duplication, and leakage-induced mis-binding can be localized. Two benchmark examples showing reference images, prompts, compositional formulas, generated results, and operator-level evaluations that identify identity, duplication, and binding failures.

Model	Anchor ( $\mathbf{f}$ )	Disentangle ( $\mathbf{g}$ )	Apply ( $\boldsymbol{\oplus}$ )	Compose ( $\mathbf{C}$ )	Avg.	CLIP Sim
GPT-Image-1.5	0.7649	0.6890	0.7541	0.9259	0.8118	0.2969
Nano Banana	0.7650	0.6786	0.7631	0.8975	0.7981	0.2867
Nano Banana 2	0.7724	0.7384	0.7989	0.9100	0.8205	0.2944
Nano Banana Pro	0.7488	0.7148	0.7869	0.9214	0.8172	0.2962
Emu3.5	0.6587	0.4982	0.5434	0.7871	0.6561	0.2917
FireRed Image Edit 1.1 ( Super Intelligence Team et al., 2026 ; FireRedTeam, 2026 )	0.6348	0.4703	0.4258	0.7218	0.5889	0.2603

Table 2. Overall results on TRACE-Bench averaged over slots 1–8. Avg. denotes the mean of the four operator-aligned metrics. Best results are in bold and second-best results are highlighted with a gray background.

Figure 5. Representative diagnostic tree analysis case. A diagnostic tree that recursively decomposes a multi-reference generation case into simpler sub-cases to identify where reference-conditioned failures arise.

Figure 6. Anchor performance versus template slot count (left) and reference-image entity count (right). Two line charts comparing anchor scores across template slot counts and across numbers of entities in the reference image for several evaluated models.

Figure 7. Formula abstractions and representative results for virtual try-on (left) and group-photo layout (right). Examples of virtual try-on and group-photo layout showing their formula abstractions, reference images, generated results, and success or failure markers.

Outcome / Source	$f$	$g$	$\oplus$	$C$	Overall
Stable Success	27.9	27.6	38.9	45.2	33.7
Persistent Failure	4.7	13.2	7.3	12.9	9.8
Global Reference	14.0	10.5	9.1	29.0	13.7
Joint Composition	48.8	43.4	41.1	9.7	38.5
Relation / Attribute	4.7	5.3	3.6	3.2	4.4

Table 3. Distribution of diagnostic outcomes across 200 cases.

Figure 8. Benchmark statistics after construction. The left panel shows the distributions of semantic categories and controllable attributes, and the right panel shows the composition of image sources.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Split by	Case basis	Eval. alignment	Hard grounding
OmniContext	Tasks	Subtasks	✗	✗
MultiBanana	Tasks	Reference-count tasks	✗	✗
MICON-Bench	Tasks	Task templates	✗	✓
MacroBench	Tasks	Long-context tasks	✓	✗
TRACE-Bench (ours)	Capabilities	Compositional formulas	✓	✓

Appendix

Table 4. Comparison with existing benchmarks. “Split by” indicates the primary principle used to decompose the benchmark. “Case basis” summarizes the main basis used to construct individual benchmark cases. “Eval. alignment” indicates whether the evaluation protocol is explicitly aligned with the benchmark construction logic. “Hard grounding” indicates whether identifying the intended reference content often requires detailed localization descriptions rather than simple noun phrases.

Slot	Example formula templates
1	$T_{e}\oplus g$ ; $f$
2	$C(f,\,T_{e}\oplus g)$ ; $f\oplus g$ ; $T_{e}\oplus g\oplus g$ ; $C(f,\,f)$
3	$f\oplus g\oplus g$ ; $C(f,\,T_{e}\oplus g)\oplus g_{\mathrm{global}}$ ; $C(f,\,f)\oplus g_{\mathrm{global}}$ ; $C(f,\,T_{e}\oplus g,\,T_{e}\oplus g)$
4	$C(T_{e}\oplus g,\,f,\,f)\oplus g_{\mathrm{global}}$ ; $C(T_{e}\oplus g,\,T_{e}\oplus g,\,f)\oplus g_{\mathrm{global}}$ ; $C(f,\,f\oplus g)\oplus g_{\mathrm{global}}$
5	$C(T_{e}\oplus g,\,f,\,f,\,f)\oplus g_{\mathrm{global}}$ ; $C(C(T_{e}\oplus g,\,f)\oplus g_{\mathrm{rel}},\,f)\oplus g_{\mathrm{global}}$ ; $C(C(T_{e}\oplus g,\,T_{e}\oplus g)\oplus g_{\mathrm{rel}},\,f)\oplus g_{\mathrm{global}}$
6	$C(C(f,\,f)\oplus g_{\mathrm{rel}},\,T_{e}\oplus g\oplus g)\oplus g_{\mathrm{global}}$ ; $C(T_{e}\oplus g,\,f,\,f\oplus g\oplus g)\oplus g_{\mathrm{global}}$

Appendix

Table 5. Example formula templates across different slot levels.

Op.	Target	Binary question
Case formula: $C(f_{1}\oplus g_{1})$
Target object: (’cacao_pod_A’)
$f$	$f_{1}$	Does a food item that is clearly a cacao pod, retaining its rugby-ball shape and vertically grooved surface, exist in the generated image?
$f$	$f_{1}$	Does (’cacao_pod_A’) match the referenced cacao pod in terms of its core identity as a cacao pod?
$g$	$g_{1}$ , IP on cacao_pod_A	Does (’cacao_pod_A’) incorporate the overall fairy-like design, including wings, pose, and magical effects?
$g$	$g_{1}$ , IP on cacao_pod_A	Are recognizable fairy-design cues present in the overall design of (’cacao_pod_A’) ?

Appendix

Table 6. Full checklist example for the cacao-pod case. The Target field records the corresponding operator target or target part in the formula structure, which supports tracing and later grouping.

Judge	Pear. $\uparrow$	Spear. $\uparrow$	MAE $\downarrow$	Agree. $\uparrow$
G3P	0.608	0.613	0.152	86.8%
G25P	0.554	0.537	0.173	85.4%
GPT-5.1	0.569	0.558	0.162	88.1%
GPT-5.4	0.580	0.542	0.156	88.4%
Ensemble	0.662	0.604	0.153	88.6%

Appendix

Table 7. Alignment between VLM judges and human annotations on outputs from 200 sampled benchmark cases. The ensemble averages the four VLM judges.

Figure 10. A formula-level example of diagnostic tree decomposition.

Figure 11. Cumulative pass rate over relative diagnostic-tree depth on 200 Emu3.5 cases.

Figure 12. A diagnostic tree example for a complex multi-reference case. Starting from the full case at $N_{0}$ , the tree is expanded by sequentially applying global-reference stripping, composition flattening, relation simplification, and attribute removal. The resulting branches isolate different potential sources of failure in a structured way.

Figure 13. Overall operator-aligned performance across slot levels.

Benchmark / application task	Typical requirement	TRACE-Bench abstraction
MultiBanana: X Objects	multiple referenced objects	$C(f_{1},f_{2},\ldots,f_{X})$
MultiBanana: X–1 Objects + Local	objects + one local attribute reference	$C(f_{1},\ldots,f_{X-1}\oplus g_{\text{local}})$
MultiBanana: X–1 Objects + Global	objects + one global reference	$C(f_{1},\ldots,f_{X-1})\oplus g_{\text{global}}$
MultiBanana: X–1 Objects + Background	objects + one background reference	$C(f_{1},\ldots,f_{X-1})\oplus g_{\text{background}}$
MICON: Object Composition	combine multiple referenced instances	$C(f_{1},f_{2},\ldots,f_{n})$
MICON: Spatial Composition	compose multiple instances with relation	$C(f_{1},f_{2},\ldots,f_{n})\oplus T_{\text{rel}}$

Appendix

Table 8. Examples of mapping benchmark-defined and application-oriented tasks into the TRACE-Bench formula space.

Figure 14. Breakdown of disentangle performance across fine-grained attribute subtypes. Each axis corresponds to one subtype in the tagging taxonomy.

Figure 15. Additional cross-model qualitative examples from TRACE-Bench. Each row presents one benchmark case, including the reference images, the realized natural-language reference prompt, and outputs from multiple representative models. These examples highlight recurring differences in reference retention, compositional fidelity, and stylistic tendencies across models, especially under complex multi-reference settings.

Image Generation

Image generation studies models that create or edit images from text, examples, layouts, and other structured controls.

See all 463 papers

MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation
Oct 23, 2026Bowen Zheng, Jintao Lin, Changheng Yi +5Image Generation Image Generation Evaluation
Nanjing University · The University of Hong Kong · Huawei Research
StructGen: Disambiguating Multi-Reference Image Generation via Structured Context Modeling
Jul 17, 2026Jianing Peng, Mengyu Wang, Henghui Ding +6Image Generation Compositional Image Generation
Institute of Information Science, Beijing Jiaotong University, Beijing, China · Visual Intelligence + X International Joint Laboratory of the Ministry of Education, Beijing, China · MT Lab, Meitu Inc, Beijing, China +2
VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation
Sep 29, 2026Yuta Oshima, Masakazu Yoshimura, Masahiro Suzuki +2Image Generation Compositional Image Generation
The University of Tokyo

Oct 23, 2026J/K move · Enter open · S save

MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation

Bowen Zheng, Jintao Lin, Changheng Yi, Haoran Yang+4

Nanjing University · The University of Hong Kong · Huawei Research

Image Generation Image Generation Evaluation Multi-Subject Image Generation

Open paper AskPDF arXiv

Image generation models have rapidly advanced, with many models now supporting generation conditioned on multiple reference images and textual instructions. Such models enable a wide range of applications, including entity fusion and attribute transfer across images. However, existing benchmarks for multi-reference image generation rely on limited evaluation dimensions and fail to capture the diverse challenges of the task. To address this limitation, we propose MDMR-Bench, a Multi-Dimensional benchmark for Multi-Reference image generation. MDMR-Bench is organized around representative capability settings and consists of multiple evaluation subsets spanning diverse dimensions of multi-reference image generation, including (1) common entity fusion and attribute transfer, (2) multi-element extraction from a single image, (3) infographic manipulation, and (4) visual reasoning-intensive generation. This design enables systematic analysis of model capabilities and failure modes across different challenge dimensions. Furthermore, to achieve more reliable and interpretable evaluation, we introduce a checklist-based evaluation protocol that decomposes generation quality into fine-grained evaluation aspects. Instead of relying solely on holistic scoring, our framework uses structured checklists composed of both global-level and atomic-level queries, enabling more diagnostic assessment of model performance. By benchmarking both open-source and proprietary models, we demonstrate that MDMR-Bench provides fine-grained insights into model performance across different dimensions. The dataset and code are publicly available.

Abstract

Image Generation

Explore similar work

MDMR-Bench: A Multi-Dimensional Benchmark for Multi-Reference Image Generation

StructGen: Disambiguating Multi-Reference Image Generation via Structured Context Modeling

VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation