cs.CVSep 29, 2026
SaveVIF-Bench: Evaluating Visual Instruction Following in Multi-Reference Image Generation
Organizations: The University of Tokyo
Abstract
Recent multimodal image generation models can take multiple images and textual instructions as input, enabling reference-based generation guided not only by text but also by visual instructions such as layouts, arrows, and pose cues. However, existing benchmarks do not evaluate the joint setting in which multiple references must be composed under multiple and heterogeneous visual-instruction images. To address this gap, we introduce VIF-Bench, a benchmark of 1,241 tasks designed to assess the edge of model capabilities in this joint setting by covering: (i) multi-reference generation (up to 7) under multiple heterogeneous visual instructions (up to 6), (ii) cases where reference images can potentially compete with visual instructions (e.g., a strongly posed subject vs. a target pose), and (iii) controlled comparison of visual instructions with text descriptions at different levels of specificity. Using these capabilities, we uncover three findings: (1) models face an adherence-artifact trade-off: once models reach stronger visual instruction adherence, stronger adherence tends to coincide with more instruction artifacts in generated images, (2) visual instruction adherence tends to be lower on tasks whose reference images carry a salient state of the controlled attribute (e.g., a neon-lit subject under a light-direction instruction), most consistently for light and wind, and (3) for models that can understand visual instructions, it is often better to provide visual constraints directly rather than describe them in text; when using text, a moderate level of detail works better than an exhaustive description. VIF-Bench is released as an open benchmark to establish a basis for fair comparison in controllable multi-reference image generation.
Figures & tables
Figure 1: Overview of VIF-Bench . VIF-Bench covers challenges in multi-reference, multi-visual-instruction settings, including multiple references (up to 7) with multiple visual instructions (up to 6) and potential reference–visual-instruction conflicts, such as lighting conflicts (bottom middle) or pose conflicts (bottom right). VIF-Bench covers various types of visual instructions, including layout, 3D orientation, pose, and wind/light direction.
| Benchmark | #Size | #Refs | #VIs | Reference–VI Conflict | Metrics |
| Without visual instructions | |||||
| DreamBooth ( Ruiz et al., 2022 ) | 75 | 1 | – | ✗ | CLIP, DINO |
| OmniContext ( Wu et al., 2025b ) | 400 | 3 | – | ✗ | GPT (3 dim.) |
| DreamOmni2 ( Xia et al., 2025b ) | 319 | 4 | – | ✗ | Gemini, Doubao ( ByteDance, 2025 ) |
| MultiBanana ( Oshima et al., 2026b ) | 3,769 | 8 | – | ✗ | GPT, Gemini (5 dim.) |
| With visual instructions | |||||
Table 1: Comparison among major benchmarks for reference-based image generation, editing, and visual-instruction following. #Refs denotes the maximum number of reference images provided within a task, and #VIs denotes the maximum number of explicit visual-instruction images that can be combined within a single task. Reference–VI Conflict indicates whether the benchmark explicitly identifies potential conflict cases in which attributes implied by a reference image may compete with the corresponding visual instruction. VIF-Bench jointly evaluates multiple references and heterogeneous visual instructions while explicitly identifying reference–visual-instruction conflict. DreamOmni3 only studies scribble-guided editing.
Figure 2: ( Left ) Per-category counts of reference images in VIF-Bench, broken down into three families: Main Reference (i.e., person, animal, object, text), Sub Reference (i.e., clothes, grooming, facial expression, surface), and Scene Context (scene style, background). Combining multiple real-image datasets with synthetic images from image-generation models lets us populate not only the common Main Reference categories but also Sub Reference categories that are scarce in general-purpose datasets (e.g., grooming, facial expression, and object surface), yielding a balanced pool across the taxonomy. ( Right ) Examples of reference images and visual instructions sampled from VIF-Bench. We prepare various types of visual instructions, such as layout, light arrow, wind arrow, orientation, and human pose.
Figure 3: Statistics of the VIF-Bench evaluated set (1,241 tasks). (a) Distribution of the number of reference images per task. (b) Distribution of the number of visual-instruction images per task. (c) Adoption counts of visual instructions by kind. (d) Word cloud of the textual instructions. It primarily consists of terms that describe a wide range of object categories and words indicating spatial directions.
| Model | Text Instruction Following | Reference Consistency | Visual Instruction Adherence | Visual Instruction Cleanliness | Scene Coherence | Visual Quality | Avg. |
| GPT-Image-1.5 | 6.79 | 8.12 | 4.57 | 9.39 | 7.84 | 8.88 | 7.60 |
| + multi-step | 6.53 | 7.29 | 4.43 | 9.80 | 8.24 | 8.89 | 7.53 |
| Nano Banana Pro | 6.07 | 8.27 | 5.67 | 6.27 | 7.01 | 8.48 | 6.96 |
| + multi-step | 6.85 | 7.37 | 5.09 | 8.37 | 8.03 | 8.69 | 7.40 |
| GPT-Image-1 | 6.51 | 7.84 | 4.35 | 9.69 | 7.99 | 8.90 | 7.55 |
| Nano Banana | 6.40 | 8.39 | 5.23 | 7.09 | 7.13 | 8.55 | 7.13 |
Table 2: Overall VIF-Bench scores for each generator under the six evaluation criteria. We additionally report results for the agentic multi-step variants of GPT-Image-1.5 and Nano Banana Pro.
Figure 4: ( Left ) Two-regime adherence–artifact pattern across generators. The models separate into two distinct clusters (dashed ellipses): open-weight models group tightly at low adherence, whereas closed models form a separate cluster at higher adherence. Within the closed cluster, however, stronger adherence tends to coincide with reduced cleanliness, suggesting a trade-off between the two objectives. ( Right ) Average VIF-Bench score on tasks containing each visual-instruction type. Visual instructions adopted in VIF-Bench are sensitive to improvements within model families.
Figure 5: ( Left ) Scores versus the number of reference images for four representative generators. More references degrade all three criteria and drive the open-weight models to near the floor in Reference Consistency and Visual Instruction Adherence, whereas the closed models largely retain Reference Consistency; among the closed models, Nano Banana Pro retains much of its Adherence while GPT-Image-1.5 drops sharply. ( Right ) Scores versus the number of visual-instruction images. In contrast to the number of references, Adherence changes far less with more visual instructions and depends mostly on the generator itself, while additional visual instructions mainly lower Text Instruction Following.
Figure 6: ( Left ) Visual Instruction Adherence on conflict and no-conflict subsets for Orientation, Light, Wind, and Pose. Conflict generally reduces adherence, with particularly pronounced gaps for Light and Wind; for Pose, the reduction is visible only for generators with non-trivial adherence. ( Right ) Comparison between visual instructions (VIs) and text-converted instructions at different levels of granularity for Nano Banana Pro. VI and TI Dense approximately match information content and primarily differ in modality; TI Medium and TI Sparse progressively remove information. All conditions are evaluated against the original VI, so the comparison among text variants measures recovery of the original constraint under increasing textual abstraction.
Figure 7: Qualitative examples of VIF-Bench. ( Left ) The adherence–artifact trade-off among closed models: layout drift in GPT-Image-1.5 and residual instructions in Nano Banana Pro. ( Right ) Reference–visual-instruction conflict in pose or lighting results in ignorance of visual instructions.
| Judge | Pearson | Spearman |
| GPT-5 | 0.78 | 0.75 |
| Gemini 2.5 | 0.74 | 0.71 |
| Qwen3-VL | 0.73 | 0.70 |
| Human | 0.80 | 0.78 |
Table 3: Correlation between human and VLM judges on average scores for the 168-image human-evaluation subset. Qwen3-VL-32B is included as an open-source judge.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
| Model | Text Instruction Following | Reference Consistency | Visual Instruction Adherence | Visual Instruction Cleanliness | Scene Coherence | Visual Quality | Avg. |
| GPT-Image-1.5 | 8.27 | 9.34 | 7.90 | 9.48 | 8.87 | 9.74 | 8.93 |
| + multi-step | 7.88 | 8.75 | 7.19 | 9.85 | 8.85 | 9.69 | 8.70 |
| Nano Banana Pro | 7.34 | 9.30 | 7.74 | 6.60 | 7.69 | 9.46 | 8.02 |
| + multi-step | 7.58 | 8.51 | 7.08 | 8.42 | 8.43 | 9.55 | 8.26 |
| GPT-Image-1 | 8.01 | 9.27 | 7.40 | 9.81 | 8.95 | 9.72 | 8.86 |
| Nano Banana | 7.62 | 9.48 | 7.76 | 7.46 | 7.93 | 9.52 | 8.29 |
Table 4: Overall VIF-Bench scores for each generator under the six evaluation criteria, evaluated by Qwen3-VL. We additionally report results for the agentic multi-step variants of GPT-Image-1.5 and Nano Banana Pro.
Figure 8: Examples of converting visual instructions (VIs) into text instructions (TIs) at three levels of granularity. From Dense to Sparse, information conveyed by the VI is progressively abstracted or removed. For example, Orientation is simplified from explicit yaw/pitch values to “front-right, level” and then to “the viewer’s right,” while Pose is compressed from a detailed joint-level description to a short phrase capturing only its main configuration.
Figure 9: Reference adoption counts on the VIF-Bench evaluated set (1,241 tasks). Each bar reports how many times a reference type is used, grouped into Main Reference, Sub Reference, and Scene Context.
| Model | IQ | IF | SF | Avg. |
| Show-o ( Xie et al., 2024 ) | 0.764 | 0.616 | 0.462 | 0.614 |
| OmniGen ( Xiao et al., 2024 ) | 0.730 | 0.532 | 0.438 | 0.567 |
| ACE ( Han et al., 2025 ) | 0.740 | 0.655 | 0.528 | 0.641 |
| ChatDiT ( Huang et al., 2024a ) | 0.811 | 0.713 | 0.574 | 0.699 |
| Claude + SD 2.1 ( Rombach et al., 2022 ) | 0.812 | 0.726 | 0.572 | 0.703 |
| Claude + SD 3 ( Esser et al., 2024 ) | 0.876 | 0.817 | 0.658 | 0.784 |
Table 5: Comparison with prior methods on MultiRef ( Chen et al., 2025a ) . IQ, IF, and SF denote Image Quality, Instruction Following, and Source Fidelity, respectively.
Figure 10: Example of MultiRef benchmark. These tasks are almost fully solvable by advanced models such as Nano Banana Pro and GPT-Image-1.5, and each task uses at most a single visual-instruction image.
Figure 11: Full per-generator analysis of reference–visual-instruction conflict. For each visual-instruction type (columns), we compare conflict (hatched) and no-conflict subsets across all eight generators on the six evaluation criteria and their average (rows). On Visual Instruction Adherence (third row), conflict consistently reduces scores for Orientation, Light, and Wind, with substantially larger gaps for Light and Wind, whereas the effect for Pose is more model-dependent.
Figure 12: ( Left ) Full results as a function of the number of reference images for all generators. We merge tasks with five or more reference images into the 5+ bin. For every generator, all criteria except Visual Instruction Cleanliness are lower with five or more references than with one. Cleanliness decreases for the Nano Banana models but increases for the open-weight generators, whose adherence approaches the floor. ( Right ) Full results as a function of the number of visual-instruction images for all generators. Tasks with four or more visual instructions are merged into the 4+ bin, which contains few tasks. The effect is weaker than that of the number of reference images.
Figure 13: Full comparison of visual instructions (VIs) and text-converted instructions at three levels of granularity for Nano Banana Pro and GPT-Image-1.5. Nano Banana Pro achieves its highest VI/TI Instruction Adherence with the original VI, whereas GPT-Image-1.5 benefits from text conversion and performs best with TI Medium. Other criteria generally improve as the constraints are relaxed; in particular, Visual Instruction Cleanliness increases when VI images are removed. We evaluate all conditions against the original task and its original VIs.
Figure 14: Qualitative example of VIF-Bench.
Figure 15: Qualitative example of VIF-Bench.
| Model | Text Instr. | Ref. Consist. | Vis. Adher. | Vis. Clean. | Scene | Quality | Avg. |
| GPT-Image-1 | 6.39 / 6.63 | 8.08 / 7.59 | 4.16 / 4.54 | 9.74 / 9.63 | 7.62 / 8.37 | 8.95 / 8.86 | 7.49 / 7.60 |
| GPT-Image-1.5 | 6.70 / 6.89 | 8.45 / 7.79 | 4.22 / 4.91 | 9.41 / 9.37 | 7.51 / 8.16 | 8.99 / 8.77 | 7.55 / 7.65 |
| Nano Banana Pro | 6.12 / 6.02 | 8.82 / 7.72 | 5.52 / 5.82 | 6.30 / 6.24 | 7.17 / 6.84 | 8.83 / 8.13 | 7.13 / 6.80 |
| Nano Banana | 6.42 / 6.38 | 8.84 / 7.93 | 4.93 / 5.54 | 7.13 / 7.05 | 7.11 / 7.15 | 8.81 / 8.28 | 7.21 / 7.06 |
| Qwen-Image-2511 | 2.79 / 3.23 | 3.42 / 3.34 | 2.55 / 2.44 | 7.82 / 7.73 | 5.02 / 6.22 | 7.41 / 7.26 | 4.84 / 5.04 |
| DreamOmni2 | 2.62 / 3.09 | 4.03 / 3.74 | 2.25 / 2.39 | 5.55 / 5.59 | 4.53 / 6.13 | 7.97 / 7.75 | 4.49 / 4.78 |
Table 6: Generator scores evaluated separately by GPT-5 and Gemini 2.5 Flash. Each entry reports GPT-5 / Gemini .
| Metric | Gemini–Human | GPT-5–Human | Qwen3-VL–Human | Human–Human |
| Text Instr. Follow. | 0.69 / 0.67 | 0.72 / 0.69 | 0.64 / 0.69 | 0.63 / 0.63 |
| Reference Consist. | 0.73 / 0.72 | 0.73 / 0.71 | 0.67 / 0.69 | 0.75 / 0.78 |
| Visual Instr. Adher. | 0.53 / 0.60 | 0.76 / 0.72 | 0.61 / 0.63 | 0.75 / 0.74 |
| Visual Instr. Clean. | 0.70 / 0.72 | 0.71 / 0.72 | 0.71 / 0.72 | 0.76 / 0.78 |
| Scene Coherence | 0.48 / 0.45 | 0.53 / 0.55 | 0.51 / 0.50 | 0.58 / 0.59 |
| Visual Quality | 0.49 / 0.57 | 0.57 / 0.63 | 0.49 / 0.59 | 0.57 / 0.60 |
Table 7: Correlation between human and VLM judges. Each entry reports PLCC / SRCC on the 168-image human-evaluation subset. Qwen3-VL-32B is included as an open-source judge.
| Score source | AP v1 PLCC | AP v1 SRCC | AP v2 PLCC | AP v2 SRCC | |
| GPT-5 Visual Quality | 168 | 0.566 | 0.506 | 0.680 | 0.598 |
| Gemini Visual Quality | 168 | 0.472 | 0.382 | 0.551 | 0.468 |
| Qwen3-VL-32B Visual Quality | 168 | 0.540 | 0.438 | 0.640 | 0.524 |
| Human Visual Quality | 168 | 0.415 | 0.380 | 0.497 | 0.435 |
Table 8: Correlation between Visual Quality scores and LAION Aesthetic Predictor (AP) ( LAION-AI, 2022 ) v1/v2 over 168 generated images.
| Geometric metric | PLCC | SRCC | |
| Bounding-box IoU | 168 | 0.357 | 0.435 |
| Center-in-region rate | 168 | 0.508 | 0.551 |
Table 9: Correlation of detector-based geometric metrics ( Liu et al., 2023 ) with human ratings over 168 layout examples.
| Model | #Tasks | CLIP div. | LPIPS div. | Avg. Score | |
| GPT-Image-1.5 | 1 | 3 | 0.066 | 0.412 | 8.55 |
| GPT-Image-1.5 | 2 | 3 | 0.042 | 0.381 | 8.58 |
| GPT-Image-1.5 | 3 | 3 | 0.079 | 0.438 | 8.49 |
| GPT-Image-1.5 | 4 | 3 | 0.124 | 0.505 | 6.03 |
| Nano Banana Pro | 1 | 3 | 0.078 | 0.481 | 7.74 |
| Nano Banana Pro | 2 | 3 | 0.142 | 0.542 | 7.40 |
Table 10: Diversity across repeated generations from identical references and instructions. We generate five outputs for each task. Higher CLIP and LPIPS distances indicate greater variation.
| Benchmark | #Size | #Refs | #VIs | Reference–VI Conflict | Metrics |
| Without visual instructions | |||||
| EditBench ( Wang et al., 2023 ) | 240 | 1 | – | ✗ | CLIP ( Radford et al., 2021 ) |
| EditVal ( Basu et al., 2023 ) | 648 | 1 | – | ✗ | CLIP, VLM, manual |
| EmuEdit ( Sheynin et al., 2024 ) | 3,055 | 1 | – | ✗ | L1, CLIP, DINO ( Caron et al., 2021 ) |
| MagicBrush ( Zhang et al., 2023a ) | 1,053 | 1 | – | ✗ | L1, L2, CLIP, DINO |
| AnyEdit ( Yu et al., 2025 ) | 1,250 | 1 | – | ✗ | L1, CLIP, DINO |
Table 11: Further comparison among major benchmarks for reference-based image generation, editing, and visual-instruction following.