AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation

Yuta Oshima; Ku Onoda; Yusuke Iwasawa; Masahiro Suzuki; Yutaka Matsuo; Hiroki Furuta

cs.CVSep 28, 2026

AutoRef: Harness Optimization for Agentic Multi-Reference Image Generation

Authors: Yuta Oshima, Ku Onoda, Yusuke Iwasawa, Masahiro Suzuki, Yutaka Matsuo, Hiroki Furuta

Organizations: The University of Tokyo

Abstract

Recent image generation models can take multiple reference images as input and combine them into a new image. However, multi-reference image generation remains challenging: models may omit or duplicate subjects from the references, or produce images in which multiple subjects appear unnaturally pasted. Recent work has proposed image generation agents that combine image generation models, reasoning models, and a harness, which is an executable program that specifies how reference images are interpreted, how generation is performed, how outputs are diagnosed, and how the final image is selected. In multi-reference generation, however, references play different roles and outputs must satisfy many criteria at once, such as fidelity to each reference and the naturalness of the whole image, so many parts of the harness could be improved, from how references are processed to how outputs are diagnosed. This makes it hard to predict which changes will improve performance and by how much, and good harnesses difficult to design by hand; indeed, human-written harnesses vary widely in performance. We therefore propose AutoRef, which optimizes the harness automatically while keeping both models frozen: a coding agent iteratively rewrites the harness code. AutoRef separates the tasks whose feedback informs proposals from the tasks used to select candidates, and continues the search from a beam of the top-ranked harnesses on the selection tasks. Using this procedure, we discover AutoRef-Harness, which improves the open-weight FLUX.2 [klein] 4B from 5.72 to 7.37 on held-out four-reference tasks of the MultiBanana benchmark, matching or exceeding proprietary models including Nano Banana Pro and GPT-Image-1.5. Without re-optimization, the same harness also improves results when the generator, number of references, benchmark, evaluator, or reasoning model differs from those used in the search.

Figures & tables

Figure 1: AutoRef-Harness (+ Ours), optimized on four-reference MultiBanana tasks with FLUX.2 [klein] 4B, ( a ) enables FLUX.2 [klein] 4B to match or exceed proprietary models on the held-out test split and transfers unchanged to ( b ) unseen three- and five-reference settings and ( c ) generators of different scales and families.

Figure 2: AutoRef , one iteration. A harness is a program that calls a frozen reasoning model $M_{\theta}$ and a frozen image generator $G_{\phi}$ ; the search rewrites this program. The proposer, a coding agent $P$ , reads the current beam and the search history $\mathcal{L}$ and writes $K=4$ new harnesses as code. On $D_{\mathrm{train}}$ (pink), everything from every candidate, rejected ones included — scores, evaluator rationales, execution trajectories, and generated images — enters $\mathcal{L}$ , where $P$ may inspect it. $D_{\mathrm{val}}$ (purple) only ranks the four and keeps $B=2$ ; $P$ is told which two survived but never their scores. The survivors form the next beam; their parents do not compete again.

Method	Gen.	Object	Local	Global	Background	Avg.
Proprietary Models
GPT-Image-1.5	1	6.84	8.12	7.22	6.11	7.07
Nano Banana Pro	1	6.73	7.98	7.70	6.39	7.20
Seedream 4.5	1	6.65	7.54	7.83	6.10	7.03
Open Models
OmniGen2	1	3.56	4.35	3.64	3.43	3.75

Table 1: MultiBanana ( Oshima et al., 2026 ) with 4 references, held-out test split. Cells are the mean of the five evaluation metrics (1–10) per task type and Avg. the mean of the four. Gen. is images drawn per task, a measured mean over the split rather than a nominal maximum. Best open image generators per column in bold, second best underlined. † : AutoRef-Harness discovered with FLUX.2 [klein] 4B on the four-reference MultiBanana setting.

		3 references					5 references
Method	Gen.	Object	Local	Global	Backg.	Avg.	Object	Local	Global	Backg.	Avg.
Proprietary Models
GPT-Image-1.5	1	8.42	8.14	7.84	7.03	7.86	5.68	8.48	6.20	6.02	6.60
Nano Banana Pro	1	8.27	8.09	7.62	6.03	7.50	5.53	8.41	7.03	5.88	6.71
Seedream 4.5	1	6.88	8.43	7.68	6.67	7.41	6.02	8.16	6.19	6.04	6.60
Open Models

Table 2: MultiBanana with 3 and 5 references. AutoRef-Harness largely preserves its performance gains when transferred to unseen reference counts. Best open image generators per column in bold, second best underlined. † : AutoRef-Harness discovered with FLUX.2 [klein] 4B on the four-reference MultiBanana setting.

Figure 3: Qualitative comparison on held-out four-reference MultiBanana tasks. Each reference is outlined in a distinct color, and the phrase in the instruction that refers to it is shown in the same color. Baseline models often omit, duplicate, or misplace references, or copy and paste them unnaturally, whereas AutoRef-Harness preserves each reference and integrates all four into a coherent scene.

		SINGLE		MULTIPLE			SCENE
Method	Gen.	Char.	Obj.	Char.	Obj.	C. $+$ O.	Char.	Obj.	C. $+$ O.	Avg.
Proprietary Models
GPT-Image-1.5	1	9.56	9.70	9.32	9.46	9.26	9.69	9.39	8.93	9.41
Nano Banana Pro	1	9.63	9.42	9.46	9.19	9.02	9.35	8.39	8.20	9.08
Seedream 4.5	1	9.45	9.50	9.09	9.39	9.09	9.35	8.66	8.23	9.09
Open Models

Table 3: OmniContext ( Wu et al., 2026a ) , unseen during the search and scored by its GPT-4.1 evaluator. Cells are the geometric mean of prompt following and subject consistency (0–10) per task type, and Avg. the mean of the eight; Char.: character, Obj.: object, C.

+

O.: character and object. Best open image generators per column in bold, second best underlined. † : AutoRef-Harness discovered with FLUX.2 [klein] 4B on the four-reference MultiBanana setting.

Figure 4: ( Left ) Reasoning-model transfer. AutoRef-Harness (+ Ours) improves FLUX.2 [klein] 4B with both GPT-5.5 and the open-weight Qwen3-VL-32B. ( Middle ) Comparison with human-written harnesses. AutoRef-Harness outperforms Best-of-3, GEMS, IPR, and Idea2Img. ( Right ) Comparison of harness optimization methods. AutoRef discovers stronger harnesses than Meta-Harness and Greedy Search; even its second-best discovered harness outperforms the final harnesses produced by both alternative search procedures.

	§ 5.1	§ 5.2	§ 5.3	§ 5.4	Avg.
Full harness	✓	✓	✓	✓	7.37
Leave one out	✓	✗	✓	✓	7.01 ( $-$ 0.36)
	✓	✓	✗	✓	6.99 ( $-$ 0.38)
	✓	✓	✓	✗	6.93 ( $-$ 0.44)
Grounding only	✓	✗	✗	✗	6.91 ( $-$ 0.46)
Selection only	✗	✗	✗	✓	6.15 ( $-$ 1.22)

Table 4: Component ablation on the held-out test split. Columns are the components of § 5.1 – 5.4 ; parentheses give the change from the full harness.

Figure 5: Win, tie, and loss rates (%) of AutoRef-Harness (FLUX.2 [klein] 4B) against each baseline in pairwise human evaluation.

Appendix figures & tables18 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Identifier
Open image models
OmniGen2 ( Wu et al., 2026a )	OmniGen2/OmniGen2
DreamOmni2 ( Xia et al., 2026 )	xiabs/DreamOmni2
BAGEL ( Deng et al., 2025 )	ByteDance-Seed/BAGEL-7B-MoT
FLUX.2 [klein] 4B ( Black Forest Labs, 2026 )	black-forest-labs/FLUX.2-klein-4B
FLUX.2 [klein] 9B ( Black Forest Labs, 2026 )	black-forest-labs/FLUX.2-klein-9B

Appendix

Table 5: Model versions used in this work. Identifiers are Hugging Face repositories or API model names. Exact revisions are pinned in the released code.

Figure 6: The AutoRef search that produced AutoRef-Harness. The two initial harnesses and the 20 candidates written over five iterations, each with its

J_{D_{\mathrm{val}}}

score; each candidate has an edge to its parent. Checked boxes were kept in the beam; the pink path is the lineage that ends in AutoRef-Harness (

\star

), and edit marks candidates that repair a draft by editing it. Candidate names are those given by the proposer.

Figure 7: Search progress on

D_{\mathrm{val}}

. Best validation score found so far at each iteration by AutoRef, Greedy Search, and Meta-Harness, with the same proposer, models, and evaluator. Meta-Harness does not select on

D_{\mathrm{val}}

; its candidates are scored on it only for this figure.

Figure 8: Execution trajectory of AutoRef-Harness on a held-out four-reference task. Steps 1–5 show the input, the identification of the scene reference, the two drafts, failure-aware selection, and complaint-directed revision. Quoted text with a blue bar is a prompt sent to the reasoning model $M_{\theta}$ , and the text below it is the model’s response; boxes labeled “image model” mark calls to the generator $G_{\phi}$ . Prompts and responses are abridged but not otherwise modified. Each reference’s frame color matches that of its number in the text.

Method	Gen.	Inst.	Ref.	BG	Real.	Qual.	Avg.
GPT-Image-1.5	1	6.58	7.63	6.49	6.79	7.88	7.07
Nano Banana Pro	1	6.94	7.74	6.62	6.78	7.94	7.20
Seedream 4.5	1	6.62	7.78	6.38	6.73	7.66	7.03
OmniGen2	1	3.08	3.36	3.16	3.87	5.26	3.75
DreamOmni2	1	3.36	3.57	3.10	3.91	5.22	3.83
BAGEL	1	2.83	3.13	2.66	3.22	3.92	3.15

Appendix

Table 6: MultiBanana with 4 references, held-out test split (

n{=}133

), by evaluation metric. Each metric is averaged over the four task types (object, local, global, background), and Avg. is the mean of the five metrics, as in the Avg. column of Tables 1 and 2 . Gen. is images drawn per task. Inst.: instruction alignment; Ref.: reference consistency; BG: background–subject match; Real.: physical realism; Qual.: visual quality. Best open image generators per column in bold, second best underlined.

Method	Gen.	Inst.	Ref.	BG	Real.	Qual.	Avg.
GPT-Image-1.5	1	7.22	8.31	7.56	7.84	8.35	7.86
Nano Banana Pro	1	6.79	7.91	7.15	7.68	7.99	7.50
Seedream 4.5	1	6.95	8.47	6.58	7.06	8.00	7.41
OmniGen2	1	4.55	5.15	4.51	5.46	6.58	5.25
DreamOmni2	1	4.75	5.20	4.67	5.56	6.64	5.36
BAGEL	1	3.77	4.14	3.36	4.16	5.05	4.10

Appendix

Table 7: MultiBanana with 3 references (

n{=}96

), by evaluation metric. Columns as in Table 6 .

Method	Gen.	Inst.	Ref.	BG	Real.	Qual.	Avg.
GPT-Image-1.5	1	6.73	7.45	5.65	5.96	7.20	6.60
Nano Banana Pro	1	6.82	7.50	5.66	6.21	7.36	6.71
Seedream 4.5	1	6.64	7.36	5.80	6.02	7.19	6.60
OmniGen2	1	2.84	2.66	2.59	3.20	4.60	3.18
DreamOmni2	1	2.54	3.10	2.70	3.17	4.08	3.12
BAGEL	1	3.00	2.73	2.17	2.61	3.57	2.82

Appendix

Table 8: MultiBanana with 5 references (

n{=}96

), by evaluation metric. Columns as in Table 6 .

	Object	Local	Global	Background	Avg.
Full harness	7.27	7.87	7.64	6.70	7.37
$-$ diverse drafts	7.07	7.37	7.14	6.47	7.01
$-$ complaint revision	7.24	7.01	7.33	6.38	6.99
$-$ failure-aware selection	6.79	7.39	7.55	6.00	6.93
Grounding only	7.21	6.72	7.21	6.50	6.91
Selection only	6.11	6.78	6.14	5.56	6.15

Appendix

Table 9: Component ablation on the held-out test split, by task type. Rows as in Table 4 ; Avg. is the mean over the four task types, as reported there. All rows draw three images per task except Grounding only and Generator only, which draw one.

Figure 9: Harness optimization is compatible with fine-tuning. On Qwen-Image-Edit-2511, AutoRef-Harness outperforms DyRef alone, and applying it to the DyRef-trained model yields a further improvement. Ours denotes AutoRef-Harness.

		SINGLE						MULTIPLE
		Character			Object			Character			Object			Char. + Obj.
Method	Gen.	PF	SC	Overall	PF	SC	Overall	PF	SC	Overall	PF	SC	Overall	PF	SC	Overall
GPT-Image-1.5	1	9.80	9.33	9.56	9.80	9.60	9.70	9.67	9.00	9.32	9.67	9.27	9.46	9.27	9.27	9.26
Nano Banana Pro	1	9.53	9.73	9.63	9.53	9.33	9.42	9.73	9.20	9.46	9.40	9.00	9.19	9.00	9.07	9.02
Seedream 4.5	1	9.60	9.33	9.45	9.47	9.60	9.50	9.20	9.13	9.09	9.67	9.13	9.39	9.13	9.07	9.09
OmniGen2	1	8.20	8.93	8.51	6.87	6.40	5.73	7.13	5.87	6.30	7.67	5.87	6.58	7.80	7.67	7.71

Appendix

Table 10: OmniContext, SINGLE and MULTIPLE task types, 15 tasks each: prompt following, subject consistency, and their geometric mean behind Table 3 . Gen. is images drawn per task. Best open image generators per column in bold, second best underlined.

		SCENE
		Character			Object			Char. + Obj.
Method	Gen.	PF	SC	Overall	PF	SC	Overall	PF	SC	Overall
GPT-Image-1.5	1	10.00	9.40	9.69	9.53	9.27	9.39	8.80	9.13	8.93
Nano Banana Pro	1	9.73	9.00	9.35	8.07	8.87	8.39	7.93	8.53	8.20
Seedream 4.5	1	9.87	8.87	9.35	8.60	8.80	8.66	8.13	8.47	8.23
OmniGen2	1	7.27	6.80	6.93	6.47	5.87	6.10	7.53	6.73	7.03

Appendix

Table 11: OmniContext, SCENE task types. Columns as in Table 10 .

Figure 10: AutoRef-Harness helps Qwen-Image-Edit-2511 at three references but not at five. Outputs of Qwen-Image-Edit-2511 and FLUX.2 [klein] 4B, each without and with AutoRef-Harness, on three-reference (top two rows) and five-reference (bottom two rows) tasks. The number below each output is its evaluator score (1–10).

Figure 11: Object composition. Held-out four-reference tasks that compose all subjects into one scene.

Figure 12: Local attribute transfer. Held-out four-reference tasks in which an attribute of one reference (e.g., a pose, a garment, a texture, or a text style) is applied to another.

Figure 13: Background and global style. Held-out four-reference tasks in which one reference sets the background or the style of the whole image.

Figure 14: Three references. MultiBanana tasks with three references, a count not used during the search.

Figure 15: Five references. MultiBanana tasks with five references, a count not used during the search.

Figure 16: OmniContext. Tasks with two or three references, from a benchmark not used during the search.

Explore similar work

Jun 25, 2026cs.CV

Scaling Multi-Reference Image Generation with Dynamic Reward Optimization

While personalized image generation has achieved remarkable progress, multi-reference image generation (MRIG) remains a challenging task. Most existing benchmarks fail to adequately evaluate complex MRIG scenarios, hindering further progress in this area. To better assess model performance on complex MRIG tasks, we introduce OmniRef-Bench, a benchmark that covers complex combinations of reference image types and a large number of reference images. Evaluations on OmniRef-Bench show that mainstream open-source models struggle in complex MRIG scenarios, and their performance deteriorates significantly as the number of mixed-type reference images increases. To address this issue, we propose DyRef, a two-stage training framework. In the first stage, supervised fine-tuning equips the model with the basic capability to handle complex MRIG tasks. In the second stage, we introduce Difficulty-aware Advantage Reweighting (DAR) and Discriminative Reward Scaling (DRS). DAR dynamically adjusts the optimization objective to improve performance when handling a large number of mixed-type reference images. DRS enlarges intra-group reward differences for more effective policy optimization. Experiments demonstrate that DyRef significantly improves the performance of open-source models on OmniRef-Bench and single-image editing benchmarks, demonstrating the effectiveness and generalization capability of our approach.

Wenwang Huang, Yusen Fu, Junjie Wang +6

Harbin Institute of Technology, Shenzhen · Independent Researcher · Shenzhen Loop Area Institute

Multi-Reference Image Generation Multi-Image Editing Benchmarks Image Generation

Ask PDF

Jul 17, 2026cs.CV

StructGen: Disambiguating Multi-Reference Image Generation via Structured Context Modeling

Multi-reference image generation aims to synthesize images by integrating attributes from multiple reference images under textual instructions. As the number of references increases, the task necessitates complex semantic comprehension, such as correctly associating attributes with the intended subjects and planing out coherent spatial arrangement between subjects and their environments. Existing approaches, which rely solely on natural language instruction, often fail to capture these complex intentions precisely, leading to semantic misalignment and inconsistent generation. We identify two key factors behind these limitations: natural language instructions are often verbose and ambiguous, and high-quality multi-reference data is scarce. To address these issues, we propose StructGen, which employs a structured, dictionary-like format to encode multiple reference images, thereby enabling explicit and unambiguous specification of generation intentions. To support this design, we construct a structured dataset based on high-quality real images and develop a corresponding training framework, along with a dedicated benchmark for challenging multi-reference scenarios. Extensive experiments on both public benchmarks and our proposed benchmark demonstrate that StructGen consistently outperforms existing methods on both semantic alignment and detailed reference-generation consistency, especially under complex instructions with multiple references. The code is available at https://jianingpeng0382.github.io/StructGen/

Jianing Peng, Mengyu Wang, Henghui Ding +6

Institute of Information Science, Beijing Jiaotong University, Beijing, China · Visual Intelligence + X International Joint Laboratory of the Ministry of Education, Beijing, China · MT Lab, Meitu Inc, Beijing, China +2

Multi-Reference Image Generation Multimodal Generation

Ask PDF

Sep 29, 2026cs.CV

VIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation

Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues. However, existing benchmarks do not evaluate the joint setting in which multiple references must be composed under multiple and heterogeneous visual-instruction images. To address this gap, we introduce VIF-Bench, a benchmark of 1,241 tasks designed to assess the edge of model capabilities in this joint setting by covering: (i) multi-reference generation (up to 7) under multiple heterogeneous visual instructions (up to 6), (ii) cases where reference images can potentially compete with visual instructions (e.g., a strongly posed subject vs. a target pose), and (iii) controlled comparison of visual instructions with text descriptions at different levels of specificity. Using these capabilities, we uncover three findings: (1) models face an adherence-artifact trade-off: once models reach stronger visual instruction adherence, stronger adherence tends to coincide with more instruction artifacts in generated images, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute (e.g., a neon-lit subject under a light-direction instruction), most consistently for light and wind, and (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text; when using text, a moderate level of detail works better than an exhaustive description. VIF-Bench is released as an open benchmark to establish a basis for fair comparison in controllable multi-reference image generation.

Yuta Oshima, Masakazu Yoshimura, Masahiro Suzuki +2

The University of Tokyo

Multi-Reference Image Generation Multimodal Generation Image Generation Visual Representations

Ask PDF