YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

Shengyu Wang; Tengyue Nong; Junming Liu; Haosong Peng; Siyu Li; Ruiqi Lei; Fushuo Huo; Yuhao Pan

cs.CVNov 3, 2026

YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

Authors: Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li, Ruiqi Lei, Fushuo Huo, Yuhao Pan,

Organizations: The Hong Kong University of Science and Technology · Independent Researcher · The University of Melbourne · The Chinese University of Hong Kong, Shenzhen ·

Abstract

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.

Figures & tables

Figure 1: YANAMI: the first large-scale multimodal dialogue dataset and benchmark for stateful interaction.

Figure 2: Representative examples from YANAMI-400K. The four columns present evolving multimodal dialogues in four interaction scenarios: (a) multi-image fusion, (b) image editing, (c) world modeling, and (d) visual reasoning. Across turns, subsequent responses retrieve, maintain, and update the accumulated dialogue state. Colored text highlights six key capabilities.

Figure 3: Overview of the YANAMI data pipeline. The pipeline comprises four stages: (a) Data Preparation, (b) Data Construction, (c) Automatic Verification, and (d) Human Verification.

Index	Model	Size	Fusion	Editing	Modeling	Reasoning	Overall
1	GPT-Image-2 ( OpenAI, 2026b )	–	86.78	90.54	81.40	73.46	83.05
2	Nano Banana 2 ( Google, 2026a )	–	84.73	87.32	74.74	76.93	80.93
3	Step1X-v1p2 ( Liu et al., 2025 )	12B	56.08	63.36	51.66	48.00	54.77
4	Qwen-Edit ( Wu et al., 2025a )	20B	49.66	54.70	28.14	37.72	42.55
5	FLUX.2 [dev] ( Labs, 2025 )	32B	72.44	74.54	65.54	56.68	67.30
6	UniPic 3.0 ( Wei et al., 2026a )	20B	63.44	70.66	56.96	48.82	59.97

Table 2: Results on YANAMI-Bench across four interaction scenarios. denotes models supporting image generation/editing only; denotes models supporting both understanding and generation. denotes concatenated image inputs; denotes placing all available images at the beginning; and denotes interleaved image–text inputs. Red , blue , and yellow cells highlight the best results among closed-source models, open-source image-editing models, and UMMs. Underlined results denote the second-best performance within each model group.

Model	Fus.	Edi.	Mod.	Rea.	Avg.
Show-o2	74.00	64.25	79.50	54.00	67.94
BAGEL	85.75	76.50	85.25	78.25	81.44
BAGEL + Ours	96.25 $\uparrow$	98.00 $\uparrow$	95.50 $\uparrow$	93.00 $\uparrow$	95.69 $\uparrow$

Table 3: Selected average DSU results. An upward arrow indicates an improvement over BAGEL. Full results are provided in Appendix D.2 .

Panel of Figure 6 — Figure 6: Analysis of stateful multi-turn multimodal interaction. (a) Measuring the understanding-realization gap, lower is better; (b) comparing performance at the first and last image-generation turns under GT-History, to assess the effect of increasingly accumulated context and state dependencies; (c) comparing GT-History with Self-History on dialogues to measure error accumulation when self-generated images are used as subsequent visual context.

Model	Understanding			GenEval
Model	POPE	MME-P	MMBench	Single Obj.	Two Obj.	Count.	Color	Position	Attri.	Overall
Emu3 ( Wang et al., 2024 )	85.2	1243.8	58.5	0.99	0.81	0.42	0.80	0.49	0.45	0.66
Show-o ( Xie et al., 2025 )	80.0	1097.2	–	0.98	0.80	0.66	0.84	0.31	0.50	0.68
Janus-Pro-7B ( Chen et al., 2025d )	87.4	1567.1	79.2	0.99	0.89	0.59	0.90	0.79	0.66	0.80
MetaQuery-XL ( Pan et al., 2025 )	–	1685.2	83.5	–	–	–	–	–	–	0.80
Ovis-U1 ( Wang et al., 2025a )	–	1508.0	77.8	0.98	0.98	0.90	0.92	0.79	0.75	0.89

Table 4: Comparison of different models on visual understanding tasks and GenEval. An upward arrow indicates an improvement over BAGEL.

Model	BG	Color	Mat.	Motion	Port.	Style	Add	Remove	Replace	Text	Tone	Avg.
AnyEdit ( Yu et al., 2025 )	4.31	4.25	2.64	0.67	1.90	1.95	3.72	3.75	3.23	0.77	4.21	2.85
InstructPix2Pix ( Brooks et al., 2023 )	3.94	5.40	3.52	1.27	2.62	4.39	3.07	1.50	3.48	1.13	5.10	3.22
OmniGen ( Xiao et al., 2025 )	5.23	5.93	5.44	3.12	3.17	4.88	6.33	6.35	5.34	4.31	4.96	5.01
Step1X-Edit ( Liu et al., 2025 )	7.03	6.26	6.46	3.66	5.23	7.24	7.17	6.42	7.39	7.40	6.62	6.44
Step1X-Edit-v1p2 ( Liu et al., 2025 )	–	–	–	–	–	–	–	–	–	–	–	7.24
OmniGen2 ( Wu et al., 2025c )	6.99	6.66	4.88	2.55	3.66	6.08	7.09	6.60	6.65	4.49	6.03	5.57

Table 5: Comparison of fine-tuned BAGEL and other models on GEdit-EN-full benchmark. An upward arrow indicates an improvement over BAGEL.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Scale	Domain	Open Data	Multi-turn Format	Hybrid Evaluation	Explicit Dialogue-State Management	Irreducible to Single-Turn
InstructPix2Pix ( Brooks et al., 2023 )	$\sim$ 450K editing pairs	E	Yes	No	No	No	No
AnyEdit ( Yu et al., 2025 )	2.5M editing pairs	E	Yes	No	No	No	No
CoMM ( Chen et al., 2025c )	227K documents	U/G	Yes	No	No	No	No
Zebra-CoT ( Li et al., 2025 )	182,384 samples	U/G	Yes	No	No	No	No
Pico-Banana-400K ( Qian et al., 2026 )	400K editing pairs; 72K multi-turn	E	Yes	Partial	No	No	No
WEAVE-100K ( Chow et al., 2026b )	100,750 dialogues	U/G/E	Yes	Yes	Yes	No	No

Appendix

Table 6: Comparison with representative multimodal datasets. U, G, and E denote visual understanding, image generation, and image editing, respectively. Multi-turn Format indicates that data are organized as multi-turn user–assistant interactions; Hybrid Evaluation indicates a unified protocol combining complementary evaluation signals rather than merely reporting multiple independent metrics. Explicit Dialogue-State Management is marked Yes only when a dataset explicitly supervises the establishment, maintenance, invalidation, or update of the currently valid multimodal state; generic history conditioning, context tracking, or a claim of “stateful dialogue” alone does not qualify. Irreducible to Single-Turn indicates that target turns cannot be reformulated as conventional single-turn tasks simply by directly supplying the relevant historical images. Partial denotes implicit or subset-level coverage. Scale follows the unit reported by each work and is therefore not directly comparable across all rows. ‡ ChatUMM reports approximately 40B consumed training tokens but does not report the number of dialogues, images, or samples. § No official ChatUMM training-data release was publicly available as of 2026-09-24; its categorization is therefore based on the construction procedure and examples reported in the paper. Although ChatUMM describes its interactions as stateful, its published pipeline primarily converts standard single-turn tasks into dialogues through historical dependencies and distractor turns, without explicit dialogue-state supervision. Once the relevant historical input is supplied directly, the reported target turns can generally be reformulated as conventional single-turn generation, editing, or composition tasks.

Figure 7: Human-verification interface for YANAMI-Bench. Annotators can browse the 100 benchmark dialogues in each interaction scenario, inspect the complete multi-turn context and visual outputs, and review or revise the corresponding atomic evaluation criteria. The interface also records whether a sample is approved or requires image refinement, supporting the exhaustive verification and correction process used to construct the final benchmark.

Corpus and Sequence Statistics		Dataset Distributions
Statistic	Value	Images per Dialogue	Dialogues	Share
Total Dialogues	402,639	$<4$	80,771	20.06%
Total Images	2,047,236	$=4$	106,841	26.54%
Total Dialogue Turns	2,643,291	$=5$	76,358	18.96%
Total User Messages	2,643,291	$\geq 4$	321,868	79.94%
Total Assistant Messages	2,643,291	$\geq 5$	215,027	53.40%

Appendix

Table 7: Detailed statistics of YANAMI-400K. Each dialogue turn contains one user message and one assistant response; Total Messages therefore counts both sides of the interaction. Scenario shares are rounded percentages corresponding to the training-set distribution shown in Fig. 4 .

Scenario	Dialogues	Images	Dialogue Turns	Understanding QAs	Atomic Checks	Avg. Checks
Multi-Image Fusion	100	400	777	400	1,890	18.90
Image Editing	100	499	754	400	3,408	34.08
World Modeling	100	467	598	400	1,141	11.41
Visual Reasoning	100	517	1,044	400	2,028	20.28
Total	400	1,883	3,173	1,600	8,467	21.17

Appendix

Table 8: Detailed statistics of YANAMI-Bench by interaction scenario. Images include all reference and target images contained in the benchmark dialogues. Understanding QAs comprise Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating questions, with one question of each type per dialogue. Atomic Checks counts the independently verifiable visual criteria used in the benchmark evaluation and excludes capability-specific checks; Avg. Checks reports the average number of evaluated atomic checks per dialogue.

Model	Generation mechanism	Visual-input handling	Principal inference settings
BAGEL ( Deng et al., 2025 ) and +YANAMI-400K	Unified MoT for understanding and generation	Native interleaved image–text sequence; no image concatenation	50 steps; text/image CFG $4.0/1.5$ ; parallel CFG; timestep shift $3.0$
Emu3.5 ( Cui et al., 2025 )	Unified autoregressive generation with discrete visual tokens	Native interleaved image–text sequence; no image concatenation	Image/text guidance $3.0/1.0$ ; image top- $k$ 10,240; top- $p$ $1.0$
OmniGen2 ( Wu et al., 2025c )	Unified diffusion pipeline	Separate visual inputs; references are concatenated into a grid only when more than three are present	50 steps; text guidance $4.0$ ; resolution alignment disabled
Ovis-U1 ( Wang et al., 2025a )	Unified MLLM with a diffusion generation head	Ordered image placeholders; multiple references are concatenated into a grid	50 steps; text/image CFG $5.0/1.5$
InternVL-U ( Tian et al., 2026 )	Unified VLM with diffusion-based generation	Multiple references are concatenated into a grid	50 steps; deterministic per-sample seed
Janus-Pro ( Chen et al., 2025d )	Autoregressive generation with discrete visual tokens	Interleaved inputs for understanding; generation uses the flattened dialogue text without reference-image conditioning	576 visual tokens per image; CFG $5.0$ ; greedy decoding

Appendix

Table 9: Baseline inference configurations used on YANAMI-Bench. Only settings that materially affect input organization or output generation are reported; all models otherwise follow the shared evaluation protocol described in the text.

Model and System Configuration		Optimization and Sequence Configuration
Setting	Value	Setting	Value
Base model	BAGEL-7B-MoT	Learning rate	$1\times 10^{-5}$
Decoder layer	Qwen2MoTDecoderLayer	Scheduler	Constant
Compute resources	$6\times$ NVIDIA A800 80GB; $16\times$ NVIDIA A100 80GB	Warm-up steps	100
Launch configuration	1 node, 6 processes	CE / MSE loss weights	0.25 / 1.0
Sharding strategy	FSDP FULL_SHARD	Gradient accumulation steps	1

Appendix

Table 10: Training configuration. Settings used to fine-tune BAGEL-7B-MoT on YANAMI-400K, including the model and system setup, optimization hyperparameters, and sequence-length settings.

Model	Fusion	Editing	Modeling	Reasoning	Avg.
OmniGen2 ( Wu et al., 2025c )	46.75	50.00	65.00	42.75	51.13
Janus-Pro ( Chen et al., 2025d )	68.00	41.75	69.25	51.50	57.63
Show-o2 ( Xie et al., 2026a )	74.00	64.25	79.50	54.00	67.94
Ovis-U1 ( Wang et al., 2025a )	83.75	25.25	65.75	55.25	57.50
Emu3.5 ( Cui et al., 2025 )	80.00	47.50	85.50	71.00	71.00
InternVL-U ( Tian et al., 2026 )	82.75	73.00	88.75	63.00	76.88

Appendix

Table 11: Dialogue-state understanding results across the four interaction scenarios. Fusion, Editing, Modeling, and Reasoning denote Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, respectively. Each scenario score is the arithmetic mean of RR, CR, CT, and CU, and Avg. is the macro-average across the four scenarios. All results are reported on a 0–100 scale, and higher values are better.

Index	Model	Size	RR	CR	CT	CU	DSU Avg.
1	OmniGen2 ( Wu et al., 2025c )	7B	15.0	66.0	28.0	62.0	42.75
2	Janus-Pro ( Chen et al., 2025d )	7B	30.0	71.0	41.0	64.0	51.50
3	Show-o2 ( Xie et al., 2026a )	7B	18.0	81.0	43.0	74.0	54.00
4	Ovis-U1 ( Wang et al., 2025a )	3B	34.0	70.0	42.0	75.0	55.25
5	Emu3.5 ( Cui et al., 2025 )	34B	34.0	87.0	71.0	92.0	71.00
6	InternVL-U ( Tian et al., 2026 )	4B	25.0	90.0	53.0	84.0	63.00

Appendix

Table 12: Dialogue-state understanding results for Visual Reasoning. RR, CR, CT, and CU denote Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating, respectively. DSU Avg. is the arithmetic mean of the four capability scores. All results are reported on a 0–100 scale, and higher values are better. An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	Modality	Format	DSU Avg.	SC	VQ	Overall Avg.
1	GPT-Image-2 ( OpenAI, 2026b )	–			–	75.2	70.8	73.46
2	Nano Banana 2 ( Google, 2026a )	–			–	80.7	71.3	76.93
3	Step1X-v1p2 ( Liu et al., 2025 )	12B			–	54.0	39.0	48.00
4	Qwen-Edit ( Wu et al., 2025a )	20B			–	39.4	35.2	37.72
5	FLUX.2 [dev] ( Labs, 2025 )	32B			–	49.0	68.2	56.68
6	UniPic 3.0 ( Wei et al., 2026a )	20B			–	56.1	37.9	48.82

Appendix

Table 13: Understanding and visual-realization results for Visual Reasoning. SC and VQ denote State Consistency and Visual Quality, respectively. All component and aggregate results are reported on a 0–100 scale; VQ scores are converted from their original 0–10 scale by multiplying by 10. Overall Avg. is computed using the weighting defined in Appendix B.3 . Image-generation-only models do not provide DSU scores; a dash denotes an unsupported or unavailable component. Modality and input-format icons follow the conventions in Table 2 . An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	RR	CR	CT	CU	DSU Avg.
1	OmniGen2 ( Wu et al., 2025c )	7B	73.0	44.0	44.0	39.0	50.00
2	Janus-Pro ( Chen et al., 2025d )	7B	18.0	55.0	62.0	32.0	41.75
3	Show-o2 ( Xie et al., 2026a )	7B	70.0	60.0	75.0	52.0	64.25
4	Ovis-U1 ( Wang et al., 2025a )	3B	32.0	25.0	35.0	9.0	25.25
5	Emu3.5 ( Cui et al., 2025 )	34B	2.0	66.0	81.0	41.0	47.50
6	InternVL-U ( Tian et al., 2026 )	4B	67.0	83.0	87.0	55.0	73.00

Appendix

Table 14: Dialogue-state understanding results for Image Editing. RR, CR, CT, and CU denote Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating, respectively. DSU Avg. is the arithmetic mean of the four capability scores. All results are reported on a 0–100 scale, and higher values are better. An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	Modality	Format	DSU Avg.	SC	VQ	Overall Avg.
1	GPT-Image-2 ( OpenAI, 2026b )	–			–	93.6	85.9	90.54
2	Nano Banana 2 ( Google, 2026a )	–			–	91.1	81.6	87.32
3	Step1X-v1p2 ( Liu et al., 2025 )	12B			–	64.0	62.4	63.36
4	Qwen-Edit ( Wu et al., 2025a )	20B			–	53.7	56.2	54.70
5	FLUX.2 [dev] ( Labs, 2025 )	32B			–	73.3	76.4	74.54
6	UniPic 3.0 ( Wei et al., 2026a )	20B			–	68.3	74.2	70.66

Appendix

Table 15: Understanding and visual-realization results for Image Editing. SC and VQ denote State Consistency and Visual Quality, respectively. All component and aggregate results are reported on a 0–100 scale; VQ scores are converted from their original 0–10 scale by multiplying by 10. Overall Avg. is computed using the weighting defined in Appendix B.3 . Image-generation-only models do not provide DSU scores; a dash denotes an unsupported or unavailable component. Modality and input-format icons follow the conventions in Table 2 . An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	RR	CR	CT	CU	DSU Avg.
1	OmniGen2 ( Wu et al., 2025c )	7B	78.0	65.0	48.0	69.0	65.00
2	Janus-Pro ( Chen et al., 2025d )	7B	68.0	47.0	93.0	69.0	69.25
3	Show-o2 ( Xie et al., 2026a )	7B	70.0	75.0	100.0	73.0	79.50
4	Ovis-U1 ( Wang et al., 2025a )	3B	58.0	56.0	81.0	68.0	65.75
5	Emu3.5 ( Cui et al., 2025 )	34B	73.0	76.0	99.0	94.0	85.50
6	InternVL-U ( Tian et al., 2026 )	4B	81.0	87.0	97.0	90.0	88.75

Appendix

Table 16: Dialogue-state understanding results for World Modeling. RR, CR, CT, and CU denote Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating, respectively. DSU Avg. is the arithmetic mean of the four capability scores. All results are reported on a 0–100 scale, and higher values are better. An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	Modality	Format	DSU Avg.	SC	VQ	Overall Avg.
1	GPT-Image-2 ( OpenAI, 2026b )	–			–	79.2	84.7	81.40
2	Nano Banana 2 ( Google, 2026a )	–			–	71.6	79.4	74.74
3	Step1X-v1p2 ( Liu et al., 2025 )	12B			–	56.7	44.1	51.66
4	Qwen-Edit ( Wu et al., 2025a )	20B			–	27.1	29.7	28.14
5	FLUX.2 [dev] ( Labs, 2025 )	32B			–	60.9	72.5	65.54
6	UniPic 3.0 ( Wei et al., 2026a )	20B			–	62.2	49.1	56.96

Appendix

Table 17: Understanding and visual-realization results for World Modeling. SC and VQ denote State Consistency and Visual Quality, respectively. All component and aggregate results are reported on a 0–100 scale; VQ scores are converted from their original 0–10 scale by multiplying by 10. Overall Avg. is computed using the weighting defined in Appendix B.3 . Image-generation-only models do not provide DSU scores; a dash denotes an unsupported or unavailable component. Modality and input-format icons follow the conventions in Table 2 . An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	RR	CR	CT	CU	DSU Avg.
1	OmniGen2 ( Wu et al., 2025c )	7B	31.0	63.0	60.0	33.0	46.75
2	Janus-Pro ( Chen et al., 2025d )	7B	45.0	79.0	72.0	76.0	68.00
3	Show-o2 ( Xie et al., 2026a )	7B	48.0	71.0	90.0	87.0	74.00
4	Ovis-U1 ( Wang et al., 2025a )	3B	84.0	78.0	88.0	85.0	83.75
5	Emu3.5 ( Cui et al., 2025 )	34B	41.0	91.0	95.0	93.0	80.00
6	InternVL-U ( Tian et al., 2026 )	4B	66.0	84.0	89.0	92.0	82.75

Appendix

Table 18: Dialogue-state understanding results for Multi-Image Fusion. RR, CR, CT, and CU denote Reference Resolution, Context Retrieval, Constraint Tracking, and Correction Updating, respectively. DSU Avg. is the arithmetic mean of the four capability scores. All results are reported on a 0–100 scale, and higher values are better. An upward arrow indicates an improvement over BAGEL.

Index	Model	Size	Modality	Format	DSU Avg.	SC	VQ	Overall Avg.
1	GPT-Image-2 ( OpenAI, 2026b )	–			–	90.3	81.4	86.78
2	Nano Banana 2 ( Google, 2026a )	–			–	89.7	77.3	84.73
3	Step1X-v1p2 ( Liu et al., 2025 )	12B			–	61.2	48.4	56.08
4	Qwen-Edit ( Wu et al., 2025a )	20B			–	48.5	51.4	49.66
5	FLUX.2 [dev] ( Labs, 2025 )	32B			–	74.8	68.9	72.44
6	UniPic 3.0 ( Wei et al., 2026a )	20B			–	63.6	63.2	63.44

Appendix

Table 19: Understanding and visual-realization results for Multi-Image Fusion. SC and VQ denote State Consistency and Visual Quality, respectively. All component and aggregate results are reported on a 0–100 scale; VQ scores are converted from their original 0–10 scale by multiplying by 10. Overall Avg. is computed using the weighting defined in Appendix B.3 . Image-generation-only models do not provide DSU scores; a dash denotes an unsupported or unavailable component. Modality and input-format icons follow the conventions in Table 2 . An upward arrow indicates an improvement over BAGEL.

Model	First	Last	$\Delta$
BAGEL original	0.543	0.518	$-0.025$
BAGEL + YANAMI-400K	0.703	0.634	$-0.068$
Emu3.5	0.657	0.650	$-0.007$
OmniGen2	0.568	0.539	$-0.029$
Ovis-U1	0.609	0.481	$-0.128$
InternVL-U	0.573	0.508	$-0.065$

Appendix

Table 20: Complete first- versus last-generation-turn results under GT-History. Only dialogues containing at least two generation turns are included. Higher scores are better, and

\Delta

denotes Last minus First.

Model	CK $\downarrow$	CU $\downarrow$	Overall $\downarrow$
BAGEL original	0.5667	0.5922	0.5795
BAGEL + YANAMI-400K	0.4158	0.3270	0.3714
Emu3.5	0.3765	0.3889	0.3827
OmniGen2	0.5188	0.5126	0.5157
Ovis-U1	0.4835	0.4609	0.4722
InternVL-U	0.5478	0.5707	0.5592

Appendix

Table 21: Understanding–realization gaps computed on shared subsets that all evaluated models understand correctly. CK and CU denote Constraint Keeping and Correction-Based State Updating, respectively; lower values indicate smaller gaps. Overall is the macro-average of CK and CU.

Panel of Figure 8 — Figure 8: Complete qualitative example for Image Editing. Across seven turns, the dialogue progressively replaces and modifies visual elements while preserving the active appearance constraints, incorporating a user correction, and retrieving earlier requirements for the final composition. The left and right panels show the first and second halves of the same dialogue, respectively.

Panel of Figure 9 — Figure 9: Complete qualitative example for Multi-Image Fusion. The dialogue combines information from multiple visual references while tracking the intended setting, object placement, and persistent restrictions, culminating in a jointly composed visual output. The left and right panels show the first and second halves of the same dialogue, respectively.

Panel of Figure 10 — Figure 10: Complete qualitative example for World Modeling. The interaction requires the model to maintain the scene layout, update an initially incorrect depth relation, and render a new viewpoint while preserving the corrected spatial state and other persistent scene properties. The left and right panels show the first and second halves of the same dialogue, respectively.

Panel of Figure 11 — Figure 11: Complete qualitative example for Visual Reasoning. The model follows a multi-step action plan, revises the plan after a user correction, generates intermediate visual states, and subsequently retrieves those states to describe the completed sequence. The left and right panels show the first and second halves of the same dialogue, respectively.

Multimodal Dataset

See all 60 papers

Jun 8, 2026

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

In recent years, unified multimodal models (UMMs) have emerged to support both understanding and generation within a single framework. Mastering dynamic, multi-turn interleaved image-text dialogues is a crucial task for UMMs in real-world applications. However, existing benchmarks fail to evaluate this important task, as they are often limited to single-turn or static settings, and typically overlook exposure bias in multi-turn interactions. To bridge this gap, we propose IMUG-Bench, a comprehensive benchmark for multi-turn interleaved image-text dialogue of UMMs that jointly evaluates their understanding and generation capabilities. Our IMUG-Bench comprises three classes: Static Spatial, Temporal Causal, and Hybrid, covering 3,113 samples and 12,034 interaction turns. It also includes dynamic understanding questions, thereby supporting evaluation that better reflects real-world multi-turn interaction scenarios. Large-scale experiments on IMUG-Bench systematically evaluate mainstream open-source and closed-source UMMs, revealing their capability boundaries and failure modes, and uncovering pronounced exposure bias on the generation side in multi-turn interactions. We further explore several test-time scaling strategies, including Chain-of-Thought, Self-Verification, and Best-of-N Sampling, which effectively improve generation accuracy and mitigate exposure bias in generation tasks. These findings provide insights into enhancing the robustness and multi-turn interaction capability of future UMMs.

Lingyi Meng, Zecong Tang, Haoran Li +12

Zhejiang University · Institute of Automation, Chinese Academy of Sciences · The University of Hong Kong +1

Unified Multimodal Models Multimodal Generation Multi-Turn Dialogue Evaluation Exposure Bias Multimodal Model Evaluation

Ask PDF

Aug 18, 2026

Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges

Conversational AI is moving beyond isolated text prompts toward sustained, multimodal interaction. In real conversations, users clarify goals, revise requests, interrupt responses, switch topics, and introduce new evidence while expecting systems to preserve context across turns. This makes multi-turn dialogue a distinct challenge requiring systems to maintain and update memory, ground responses across modalities, tools, and external knowledge, and adapt across languages and cultures. This study reviews multi-turn conversational AI across text-only dialogue, AudioLLMs and speech-native systems, multimodal and omni-modal systems, and tool-augmented agents. We organize the literature around datasets and benchmarks, modeling paradigms, training strategies, evaluation setups, and cross-cutting challenges. Our analysis shows that support for multiple modalities has advanced faster than the ability to sustain coherent interaction across a session. Despite stronger capabilities to perceive, speak, and act across modalities, current systems still struggle with persistent memory, cross-turn grounding, full-duplex interaction, robust evaluation, and cultural alignment. We conclude with a research agenda for systems that can remember, revise, ground, speak, listen, act, and adapt across turns, modalities, and cultures. (https://github.com/faiza-sfa/multiturn-conversational-ai-survey)

Syeda Faiza Ahmed Sara, Zien Sheikh Ali, Hunzalah Hassan Bhatti +2

Qatar Computing Research Institute, HBKU, Qatar

Conversational Agents Multimodal Agents Multi-Turn Dialogue Evaluation Multimodal Grounding Full-Duplex Spoken Dialogue Systems

Ask PDF

Mar 13, 2026

MIBench: Evaluating LMMs on Multimodal Interaction

In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways between modalities are referred to as "multimodal interaction". How well a model handles various multimodal interactions largely characterizes its multimodal ability. In this paper, we introduce MIBench, a comprehensive benchmark designed to evaluate the multimodal interaction capabilities of Large Multimodal Models (LMMs), which formulates each instance as a (con_v , con_t, task) triplet with contexts from vision and text, necessitating that LMMs employ correct forms of multimodal interaction to effectively complete the task. MIBench assesses models from three key aspects: the ability to source information from vision-centric or text-centric cues, and the ability to generate new information from their joint synergy. Each interaction capability is evaluated hierarchically across three cognitive levels: Recognition, Understanding, and Reasoning. MIBench comprises over 10,000 vision-text context pairs spanning 32 distinct tasks. Evaluation of state-of-the-art LMMs show that: (1) LMMs' ability on multimodal interaction remains constrained, despite the scaling of model parameters and training data; (2) they are easily distracted by textual modalities when processing vision information; (3) they mostly possess a basic capacity for multimodal synergy; and (4) natively trained multimodal models show noticeable deficits in fundamental interaction ability. We expect that these observations can serve as a reference for developing LMMs with more enhanced multimodal ability in the future.

Yu Miao, Zequn Yang, Yake Wei +5

Gaoling School of Artificial Intelligence, Renmin University of China, Beijing, China · Beijing Key Laboratory of Research on Large Models and Intelligent Governance, Beijing, China · Beihang University, Beijing, China +1

Multimodal Large Language Models Multimodal Model Evaluation Multimodal Understanding Multimodal Reasoning

Ask PDF

Abstract

Multimodal Dataset

Explore similar work

IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generation

Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challenges

MIBench: Evaluating LMMs on Multimodal Interaction