LongBanana: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis
Authors: Haoxiang Cao, Yuxuan Zhang, Penghui Du, Bo Li, Jiajiong Cao, Yingying Fan, Zimeng Wu, Yifu Luo, Mingzhe Zheng, Junwen Miao, Huaisong Zhang, Yang Yao, Yu Wang, Dongfu Jiang, Ping Nie, Wenhu Chen, Chaoqun Wang, Changqian Yu, Kelsey R. Allen
Organizations: South China Normal University · KlingAI Research · University of British Columbia · Vector Institute · University of Maryland · Tsinghua University · HKUST · Carnegie Mellon University · HKU · University of Waterloo · Canada CIFAR AI Chair
We introduce LongBanana, to our knowledge the first expert-designed and expert-verified benchmark for long-context multi-reference image synthesis in the age of agent-driven image generation. The task requires retrieving dispersed visual evidence, rejecting distractors, binding entities and relations, and composing one coherent final image from multiple references. LongBanana contains 90 carefully designed tasks organized by a 3-by-3 taxonomy; each task includes about 32 reference images, 20 constraints, and 50 evaluation items, yielding 4,670 expert-specified atomic evaluation items in total. Its graph-grounded evaluation design instantiates each task with reference images and an entity-relation graph, then converts node, edge, and whole-graph obligations into checklist items for Content Fidelity, Relational Binding, and Global Constraint Satisfaction. Reviewed expert labels include a 15% double-labeled subset with 89.4% exact agreement and 0.83 quadratic weighted kappa. We further introduce LongBanana-Gen, an agent-harness framework for long-horizon image generation; experiments show that minimal agent harnesses improve over direct generation, while a ReAct-Plan-Exec-Reflect (RPR) workflow yields harness-dependent gains by strengthening evidence grounding and repair. Together, LongBanana provides an auditable testbed for measuring dense visual evidence use and a shared target for developing stronger agent-driven image-generation systems. Code and data are available at the project page: https://long-banana.com/.
Figures & tables
Figure 1: LongBanana task overview. (a) Each case pairs a heterogeneous reference set and structured synthesis prompt with the generated image under evaluation. (b) LongBanana -Eval converts graph-grounded checklist items into dimension-level scores and auditable evidence traces. The score labels Content, Relations, and Global denote Content Fidelity, Relational Binding, and Global Constraint Satisfaction, respectively.
Figure 2: The 3×3 taxonomy used to organize LongBanana . Rows group cases into Scene Composition, Visual Appearance, and Reference Reasoning, while columns expose subclass-level demands used for diagnostic reporting. Each subclass contributes 10 cases.
Figure 3: LongBanana construction pipeline. (1) Scenario Construction : experts design scenarios aligned with the task taxonomy. (2) Graph Construction : an LLM structures each scenario into entities, attributes, relations, and whole-scene constraints. (3) Prompt & Reference Synthesis : the graph guides the creation of the target instruction and visual references. (4) Checklist & Curation : graph constraints are translated into evidence-linked checklist items, and human curators review the instruction, references, and checklist for consistency, visual feasibility, and verifiability.
Figure 4: LongBanana -Eval scoring pipeline. The parser assembles item-specific reference evidence and routes each check to one of three rubric critics. Critics inspect the evidence, call verification tools as needed, and return item labels with supporting traces. Item labels are averaged to produce dimension-level and overall scores.
Judge
Pearson
Spearman
Kendall
VLM-as-judge
0.8413
0.8248
0.6437
LongBananaEval (agentic checklist judge)
0.8707
0.8602
0.6727
Table 1: Judge–human agreement on the fully covered 90-case LongBanana audit pool. Correlations compare each judge’s checklist-average output-level Overall score with the reviewed human Overall score; higher values indicate closer alignment.
Image generator= Nano Banana2
Scene Composition
Visual Appearance
Reference Reasoning
Overall
Strategy ↓
SCC
RGC
LAC
GAC
LAB
AHC
SEL
TSR
AGG
Avg.
P@70%
Direct generation with Nano Banana 2
FP-Direct
69.6
61.8
62.0
88.7
77.5
71.1
56.4
73.4
55.1
68.4
24.4
PD-Direct
69.9
61.9
62.6
98.8
87.0
72.1
68.5
67.2
83.6
74.6
35.6
Agent harness + Nano Banana 2
Hermes-Agent
70.1
61.5
57.9
93.2
87.2
65.4
72.1
75.1
83.1
73.9
43.3
Table 2: Harness and workflow comparison on the held-out 90-case LongBanana benchmark. The foundation image model is fixed to Nano Banana 2, and the compared component is the generation harness/workflow. Subclass columns and Overall Avg. report the unweighted checklist-average score (0–100); P@70% is the difficulty-weighted task pass rate. Higher is better for all reported metrics. Boldface and underlining mark the best and second-best values in each column.
Strategy = OpenClaw + RPR
Scene Composition
Visual Appearance
Reference Reasoning
Overall
Image generator ↓
SCC
RGC
LAC
GAC
LAB
AHC
SEL
TSR
AGG
Avg.
P@70%
Seedream 4
60.0
52.6
51.2
92.3
82.0
46.6
50.9
67.8
68.7
63.6
18.9
GPT Image 1
54.3
47.1
49.3
94.0
87.2
47.4
54.0
62.1
60.5
61.8
17.8
[2][2] Nano Banana 2
76.1
64.5
64.1
97.1
88.6
75.6
79.8
73.6
86.4
78.4
47.8
Table 3: Foundation image-model comparison on the same 90-case LongBanana benchmark. The generation harness/workflow is fixed to OpenClaw + RPR, and the compared component is the foundation image model. Subclass columns and Overall Avg. use the same unweighted checklist-average score as Table 2 (0–100); P@70% is the difficulty-weighted task pass rate. Higher is better for all reported metrics. Boldface and underlining mark the best and second-best values in each column.
Appendix figures & tables23 assets
Supplementary material from the paper’s appendix.
Appendix
Suite
Code
Subclass
Dominant evaluative demand
Held-out cases
Suite I
SCC
Semantic Coverage & Consistency
Preserve all required entities, attributes, and semantic elements from the reference set.
10
Suite I
RGC
Relational Graph Consistency
Satisfy required interactions, spatial relations, and topology-level constraints among referenced entities.
10
Suite I
LAC
Layout Arrangement Consistency
Render the correct global composition, scale, occlusion, and foreground/background organization.
10
Suite II
GAC
Global Appearance Conditioning
Apply scene-level appearance cues such as style, palette, lighting, or atmosphere to the whole image.
10
Suite II
LAB
Local Appearance Binding
Bind different appearance references to specific entities, regions, or materials without cross-talk.
10
Suite II
AHC
Appearance Harmonization & Coherence
Merge multiple appearance references into a visually coherent and physically plausible final image.
10
Appendix
Table 4: The 3×3 task taxonomy of LongBanana . All tasks require synthesizing a final image from multiple references; subclasses differ in their dominant evaluative demand rather than output modality. LongBanana contains 10 cases per subclass, for 90 cases in total.
Step
Input / operation
Why it is needed
Quality control
Scenario authoring
Expert scenario description grounded in realistic storyboard, product, or scene-composition use cases
Ensures each task is visually concrete and long-horizon rather than a random prompt
Screen for reference density, scene feasibility, and taxonomy fit
Graph construction
Entity and attribute nodes, relation and binding edges, and whole-graph constraints extracted from the scenario
Provides the source of truth for prompt construction and checklist generation
Human audit removes ambiguous entities, weak relations, and unverifiable constraints
Reference generation
Clean reference images for objects, backgrounds, styles, local attributes, and optional distractors
Forces systems to use visual evidence rather than text shortcuts
Discriminative attributes are visible in references and omitted from text when possible
Hard-negative insertion
At least 30 cases include two or more irrelevant but visually plausible references
Tests resistance to distractors, a common failure amplifier in agentic workflows
Checklists contain forbidden-use items and annotators mark distractor leakage
Checklist derivation
Atomic requirements generated from graph nodes, graph edges, whole-graph constraints, counts, and forbidden content
Converts complex synthesis into auditable ordinal decisions
Items must be reference-dependent, visually verifiable, and non-overlapping where possible
Reported-evaluation isolation
Case identifiers and quality-check records kept separate from the reported benchmark
Keeps benchmark design checks separate from reported results
Human labels and scorer outputs are reported for the fixed 90-case benchmark
Appendix
Table 5: Construction rationale for LongBanana . Each step has an explicit input, operation, purpose, and quality-control check so that benchmark cases are grounded in auditable reference-based synthesis requirements.
Statistic
Value
Major suites
3
Subclasses
9
Cases per subclass
10
Total cases
90
Typical reference count per case
approximately 32
Typical cross-reference constraints per case
20+
Appendix
Table 6: LongBanana benchmark statistics by design. Entries describe the fixed benchmark specification rather than model outcomes.
Benchmark
Date
Human scale
Label QA / agreement
Code open?
Dense multi-ref?
TIEdit
2026-03
20 raters; 307K ratings
SRCC/PLCC/KRCC
✗
✗
CREval
2026-03
18 raters; > 200 cases
not central
✓
✗
HumanAligned-MLLM-J.
2026-02
25 raters; 100 tasks
ICC(2,k)
✓
✗
GEditBench v2
2026-03
experts; 3,506 pairs
accuracy
✓
✗
EditHF-1M
2026-03
pros; 29.1M pairs
not reported
✓
✗
Co-EditBench
2025-09
34 raters; 800 cases
SROCC/PLCC/KLCC
✗
✗
Appendix
Table 7: Comparison of LongBanana to representative image-editing and image-synthesis benchmarks. LongBanana is positioned as a high-density, expert-verifiable benchmark for long-horizon multi-reference synthesis with explicit expert-label quality-control reporting.
Quality check
Scope
Reported value
Held-out test cases
90 cases
90
Audited generation conditions
all reported generation conditions
10
Audited generated outputs
90×10 outputs
900
Benchmark checklist items
90 cases total
4,670
Reviewed human checklist labels
4,670×10 conditions
46,700 labels
Independently double-labeled subset
15% of reviewed labels
7,005 labels
Appendix
Table 8: Expert annotation quality for the 90-case LongBanana benchmark. Human reviewers label all 4,670 benchmark checklist items for all 10 reported generation conditions, yielding 46,700 reviewed output-level checklist labels. The table reports the quality-control records behind those human labels, including independent double labeling, adjudication, audit, and schema validation.
Comparison
Corrected labels
Mean score delta
Max score delta
Rank Kendall τ
Top-1 changed?
Primary → Reviewed
4.5%
1.2 pp
2.8 pp
0.96
No
Reviewed → Adjudicated
1.8%
0.5 pp
1.4 pp
0.99
No
Pre-audit → Post-audit
1.3%
0.3 pp
0.9 pp
1.00
No
Primary → Final
5.7%
1.5 pp
3.1 pp
0.95
No
Appendix
Table 9: Score stability under label correction for the ten reported generation conditions. The pre-audit to post-audit correction rate uses the full 46,700-label pool and includes expanded checks beyond the random audit sample. Checklist-average score changes are expressed in percentage points; Kendall τ and Top-1 compare condition rankings before and after revision.
Figure 5: Human checklist annotation platform examples for count-oriented items. Top: a count_check item asks whether the target image contains exactly two luxury handbags, with the relevant handbag references, prompt, item question, scoring guidance, submitted score, and first-review approval visible in the same audit record. Bottom: a count_check item asks whether the full target image contains exactly four specified people, again tying the target output, references, item text, label submission and review result to a single checklist record.
Figure 6: Human checklist annotation platform example for a relation item. The relation_check screen asks whether the warrior ally stands beside the protagonist, requiring the reviewer to compare the target image with the relevant character references and apply the predefined ternary labels. The stored record includes metadata, category, guidance text, submitted score, and first-review approval, supporting strict post-hoc audit of each checklist decision.
Figure 7: An example from the Relational Graph Consistency (RGC) subclass, with the input reference images, prompt, and a representative generated result.
Figure 8: Partial evaluation checklist for an example from the Relational Graph Consistency (RGC) subclass.
Figure 9: An example from the Layout Arrangement Consistency (LAC) subclass, with the input reference images, prompt, and a representative generated result.
Figure 10: Partial evaluation checklist for an example from the Layout Arrangement Consistency (LAC) subclass.
Figure 11: An example from the Local Appearance Binding (LAB) subclass, with the input reference images, prompt, and a representative generated result.
Figure 12: Partial evaluation checklist for an example from the Local Appearance Binding (LAB) subclass.
Figure 13: An example from the TSR task, with the input reference images, prompt and a representative generated result.
Figure 14: Partial evaluation checklist for an example from the TSR task.
Figure 15: An example from the AGG task, with the input reference images, prompt and a representative generated result.
Figure 16: Partial evaluation checklist for an example from the AGG task.
Metric
VLM-as-judge
LongBananaEval
Change
Exact ternary agreement
50.99%
66.81%
+15.82 pp
Within-one agreement
55.91%
75.19%
+19.28 pp
Opposite-end 0↔2 error
44.09%
24.81%
−19.28 pp
Appendix
Table 10: Full-pool agreement with 46,700 reviewed human checklist labels from 900 generated outputs across 10 generation conditions. Higher is better for exact ternary and within-one agreement; lower is better for opposite-end error. Changes are LongBananaEval minus VLM-as-judge, in percentage points (pp).
Figure 17: Radar profiles for the two primary result comparisons. Left: strategy comparison with Nano Banana 2 fixed as the image generator. Right: image-generator comparison with OpenClaw + RPR fixed as the harness. Axes are LongBanana taxonomy subclasses; values are the checklist-average subclass scores (0–100) reported in the main tables.
Figure 18: Agent-orchestrator workflow used in LongBanana -Gen. ReAct retrieves and reasons over reference evidence, Plan–Exec decomposes synthesis into executable subgoals, and Reflect inspects drafts to guide revision.
Scene Composition
Visual Appearance
Reference Reasoning
Overall ↑
Strategy
SCC ↑
RGC ↑
LAC ↑
GAC ↑
LAB ↑
AHC ↑
SEL ↑
TSR ↑
AGG ↑
Avg. ↑
P@70% ↑
OpenClaw
58.6
51.6
47.8
95.4
83.3
65.6
59.6
69.4
72.5
67.1
40.0
Plan–Exec only
56.6
45.7
46.5
96.5
88.3
63.7
59.0
58.4
66.8
64.6
35.6
ReAct only
54.5
48.1
41.5
92.0
71.8
52.1
62.0
57.0
72.8
61.3
36.7
Reflect only
58.7
46.4
50.6
96.4
79.8
52.0
59.7
64.4
75.7
64.9
44.4
OpenClaw + RPR
64.3
54.5
50.3
95.9
83.8
66.6
72.6
62.0
79.6
70.0
47.8
Appendix
Table 11: OpenClaw agent-generation component ablation on the 90-case LongBanana benchmark with Nano Banana 2 fixed as the image generator. Subclass scores and Overall Avg. are difficulty-weighted task scores (0–100), with difficulty calibrated across the five configurations shown; P@70% is the corresponding task pass rate. Boldface and underlining mark the best and second-best values in each column.
Comparison
Score / diff.
Bootstrap 95% CI
Paired p
Minimal harness vs. RPR, with Nano Banana 2 fixed as backend
Hermes-Agent: RPR − minimal
+1.7 pp
[+0.1, +3.7] pp
0.0239
OpenCode: RPR − minimal
+0.5 pp
[-1.3, +2.1] pp
0.1823
OpenClaw: RPR − minimal
+2.3 pp
[+0.6, +4.2] pp
0.0032
Image backend comparison, with OpenClaw + RPR fixed as the harness
Nano Banana 2 Overall
78.4
[76.7, 80.1]
–
Appendix
Table 12: Checklist-average generation summary with case-level uncertainty and paired-inference diagnostics. Scores use the same 0–100 scale as the main generation tables; mean differences are computed from the reported overall scores and expressed in percentage points (pp).
Generating images conditioned on multiple visual references is critical for real-world applications such as multi-subject composition, narrative illustration, and novel view synthesis, yet current models suffer from severe performance degradation as the number of input references grows. We identify the root cause as a fundamental data bottleneck: existing datasets are dominated by single- or few-reference pairs and lack the structured, long-context supervision needed to learn dense inter-reference dependencies. To address this, we introduce MacroData, a large-scale dataset of 400K samples, each containing up to 10 reference images, systematically organized across four complementary dimensions -- Customization, Illustration, Spatial reasoning, and Temporal dynamics -- to provide comprehensive coverage of the multi-reference generation space. Recognizing the concurrent absence of standardized evaluation protocols, we further propose MacroBench, a benchmark of 4,000 samples that assesses generative coherence across graded task dimensions and input scales. Extensive experiments show that fine-tuning on MacroData yields substantial improvements in multi-reference generation, and ablation studies further reveal synergistic benefits of cross-task co-training and effective strategies for handling long-context complexity. The dataset and benchmark will be publicly released.
Agentic image generation pairs an image generator with a search agent that retrieves references from the web, extending generation beyond the generator's training data. Real-world requests, however, often concern entities that appeared only days ago. We call this setting Image Generation in the Wild. In this setting, the web offers limited visual evidence, search results are crowded with easily confused entities, and the target is rarely shown on its own. To study this setting, we build an automated pipeline that synthesizes prompts from recent headlines and keeps only those whose real search results are highly confusing, yielding requests that are hard to solve yet natural to ask. With it, we construct Savanna, a benchmark of 500 prompts across 16 subcategories. On Savanna, even harness frameworks built on the frontier models retrieve a correct reference for only about half of the prompts. We then propose Locate-Corroborate-Isolate, a paradigm in which the agent first pins down the target's identity through text search, then verifies candidate images against multiple sources, and finally crops a clean reference for the generator. Trained on 15K trajectories that follow this paradigm, Ranger-8B achieves a search score of 67.2, outperforming harnesses equipped with frontier models. It also achieves the best generation quality with every generator we test, and surpasses prior agentic methods on other image generation benchmarks. We release the pipeline, benchmark, training data and model at https://github.com/RUCAIBox/Agentic-ImageGen-Wild.
Yifan Li, Bowen Qin, Kun Zhou +8
Renmin University of China · Tencent · Peking University
Unified multimodal models provide a natural and promising architecture for understanding diverse and complex real-world knowledge while generating high-quality images. However, they still rely primarily on frozen parametric knowledge, which makes them struggle with real-world image generation involving long-tail and knowledge-intensive concepts. Inspired by the broad success of agents on real-world tasks, we explore agentic modeling to address this limitation. Specifically, we present Unify-Agent, a unified multimodal agent for world-grounded image synthesis, which reframes image generation as an agentic pipeline consisting of prompt understanding, multimodal evidence searching, grounded recaptioning, and final synthesis. To train our model, we construct a tailored multimodal data pipeline and curate 143K high-quality agent trajectories for world-grounded image synthesis, enabling effective supervision over the full agentic generation process. We further introduce FactIP, a benchmark covering 12 categories of culturally significant and long-tail factual concepts that explicitly requires external knowledge grounding. Extensive experiments show that our proposed Unify-Agent substantially improves over its base unified model across diverse benchmarks and real world generation tasks, while approaching the world knowledge capabilities of the strongest closed-source models. As an early exploration of agent-based modeling for world-grounded image synthesis, our work highlights the value of tightly coupling reasoning, searching, and generation for reliable open-world agentic image synthesis.
Shuang Chen, Quanxin Shou, Hangting Chen +16
University of California, Los Angeles · Tencent Hunyuan · The Hong Kong University of Science and Technology +1