What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation

Sieun Hyeon; Yejoon Lee; Mintaek Lim; Woojin Kim; Jaeik Kim; Jaeyoung Do

cs.CVSep 29, 2026

What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation

Authors: Sieun Hyeon, Yejoon Lee, Mintaek Lim, Woojin Kim, Jaeik Kim, Jaeyoung Do

Organizations: AIDAS Laboratory, 1ECE & 2IPAI, Seoul National University

Abstract

Omnimodal evaluation should go beyond independent text, image, and speech production: individually plausible outputs may not express a coherent shared event. We introduce Omni-StoryBench, a story-grounded omnimodal benchmark evaluating whether models can coherently continue stories across image, narration, and speech. Each instance provides a current storybook page and structured next-page conditions, requiring models to generate the next illustration, narration, and spoken character utterance. Omni-StoryBench contains 900 rigorously validated story transitions from openly licensed children's books, with ground-truth next-page references and speech metadata. We evaluate systems with modality-specific metrics and consistency-centered LLM-as-a-judge rubrics for context preservation, condition following, reference consistency, and cross-modal coherence. Across 32 baseline configurations spanning orchestration, semi-orchestration, and native any-to-any paradigms, we find orchestration with strong VLM planning most reliable, while current native omnimodal models often struggle with output completeness and controllability. Our analysis shows text-side performance is associated with image and speech quality, but image generation and visual continuity form the clearest observed bottleneck among the evaluated configurations. These results position Omni-StoryBench as a system-level benchmark measuring coherent omnimodal generation beyond isolated modality quality.

Figures & tables

Figure 1: Overview of Omni-StoryBench . Given the current scene image, narration text, and structured generation conditions, models generate the next scene’s image, narration, and spoken character utterance. The generated triplet is evaluated against the structured conditions and ground-truth references for modality-specific quality and cross-modal consistency. The top output illustrates an any-to-any generation, whereas the bottom output demonstrates an orchestration approach.

Figure 2: Overview of Baseline Systems. In the Orchestration approach, the backbone model only outputs text. In Semi-orchestration , the backbone can only generate two out of the three modalities (text, image, and speech), so a separate expert module is connected to generate the remaining modality. Finally, Any-to-Any refers to an architecture where a single backbone model is capable of directly generating all three modalities.

Figure 3: Omni-StoryBench ranking by the seven-metric Total Average. Each metric is averaged over examples with its required outputs available, and each LLM-judge score averages three judges. BERTScore, CLIP Similarity, and Speech Metadata Accuracy are scaled from 0-1 to 10 and averaged. Generation failures are reported separately in Figure 4 .

Figure 4: Number of orchestration JSON errors and missing outputs per modality. A test case may contribute to multiple modality segments. Unlisted backbones have no recorded generation failures.

Table 5

Table 6

Figure 5: Comparison of Text Evaluation Metrics. Large-VLM orchestration scores highly under LLM judging, but shows smaller gains in BERTScore.

Figure 6: Image LLM-judge scores by category and baseline paradigm.

Appendix figures & tables34 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 7: Modality-wise LLM-judge score distributions, averaged over three judges per modality.

Panel of Figure 8 — Figure 8: An Omni-StoryBench example from Play Me the Harmonica . The input contains the current page image, current narration, book-level metadata, and structured next-page condition. The ground truth contains the next page image, narration text, speech utterance, and speech metadata.

Panel of Figure 10 — Figure 10: Web interface human annotators used for benchmark dataset filtering.

Figure 11: Web interface used by human annotators to audit and revise the benchmark dataset.

Role	Checkpoint identifier	Backend	Context limit	Max. output tokens
Text	Qwen/Qwen3-30B-A3B-Instruct-2507	vLLM	8192	1024
	ByteDance-Seed/Seed-OSS-36B-Instruct	vLLM	8192	1024
	nvidia/Llama-3_3-Nemotron-Super-49B-v1_5	vLLM	8192	1024
Image	Qwen/Qwen3-VL-32B-Instruct	vLLM	20000	1024
	OpenGVLab/InternVL3_5-38B-Instruct	vLLM	20000	1024
	LGAI-EXAONE/EXAONE-4.5-33B	vLLM	20000	1024

Appendix

Table 6: Judge checkpoints and context limits. Qwen3 JSON repair is applied after judging to nonempty responses that fail schema parsing across all four evaluation types.

Statistic	Value	$p$
Pearson correlation	0.538	0.001
Spearman correlation	0.704	6.9e-06
BERTScore range (0–10)	8.02–9.08	–
Text LLM-judge range	3.30–9.39	–

Appendix

Table 7: Agreement between BERTScore and the text LLM judge across 32 configurations. BERTScore is scaled to 0–10 only for the range comparison; correlations are unchanged. Text judge scores are the equal-weight mean of three judges. Missing or invalid evaluations are excluded from the corresponding score averages (valid-only).

Score paired with DreamSim distance	$r$	$p$	$N$
CLIP similarity	-0.982	4.0e-23	32
Image LLM judge (three-judge mean)	-0.910	5.3e-13	32
Image modality score	-0.959	6.7e-18	32
Integrated LLM judge (three-judge mean)	-0.728	2.3e-06	32
Total Average	-0.809	2.1e-08	32

Appendix

Table 8: Pearson correlations between mean DreamSim distance and system-level scores across 32 configurations on the current dataset. Lower DreamSim distance indicates greater perceptual similarity to the ground-truth next-page image. DreamSim averages exclude missing images; judge scores use the three-judge mean with the same valid-only aggregation as the main analysis. Bold indicates nominal

p<0.05

(two-sided).

Metric paired with integrated judge	$r$	$p$	$N$
Text modality score	0.987	3.7e-25	32
Image modality score	0.871	8.8e-11	32
Speech modality score	0.870	1.0e-10	32
DreamSim distance	-0.728	2.3e-06	32

Appendix

Table 9: Pearson correlations between integrated LLM-judge scores and modality-specific scores across 32 configurations on the current dataset. Integrated scores average the three judges with valid-only aggregation. DreamSim is the mean distance over valid generated images; lower values are better. Bold indicates nominal

p<0.05

(two-sided).

Partial correlation with integrated judge	Partial $r$	$p$	$N$
Text modality score $\mid$ Image , Speech	0.921	5.7e-13	32
Image modality score $\mid$ Text , Speech	0.637	1.5e-04	32
Speech modality score $\mid$ Text , Image	0.123	0.519	32

Appendix

Table 10: Partial correlations between integrated judge scores and each modality score, controlling for the other two modality scores, across 32 configurations. Judge scores are averaged equally across three judges using valid-only score averages. Bold indicates nominal two-sided

p<0.05

Integrated-judge category	Mean	IQR	Loss share	Lowest count
Metadata alignment	8.26	1.22	18.1%	0
Multimodal continuity	7.59	1.55	25.1%	0
Condition satisfaction	7.61	2.12	24.8%	0
GT semantic consistency	6.91	1.92	32.1%	32

Appendix

Table 11: Integrated-judge category diagnostics across 32 configurations. Loss share is computed from

10-\mathrm{score}

within the four integrated categories. Bold marks the strongest bottleneck signal in each column. Each category score averages three judges equally, using valid-only score averages.

Integrated-judge category	Text	Image	Speech
Metadata alignment	0.82	0.47	0.41
Multimodal continuity	0.91	0.69	0.20
Condition satisfaction	0.93	0.53	-0.12
GT semantic consistency	0.95	0.77	-0.10

Appendix

Table 12: Partial correlations between each integrated-judge category and modality scores, controlling for the other two modalities, across 32 configurations. Category scores average three judges equally using valid-only score averages. Bold indicates nominal two-sided

p<0.05

;

p

-values are omitted for compactness.

Figure 12: Category-level diagnostics of integrated omnimodal evaluation, using the equal-weight average of three judges and valid-only score averages. (a) Mean category scores within each paradigm. (b) Partial correlations with each modality score after controlling for the other two modalities.

Figure 13: Total Average under valid-only and zero-filled scoring for the 32 configurations, using the equal-weight three-judge average. Each point represents one configuration; the diagonal marks equal scores.

Figure 14: Total Average ranking of the 32 configurations under zero-filled scoring, using the equal-weight three-judge average and the same seven-component aggregation as the main analysis. Missing or invalid entries contribute zero in each component’s fixed denominator of 900 examples.

Paradigm	$N$	Valid-only	Zero-filled	Change
Orchestration (Large)	12	7.614	7.608	-0.006
Orchestration (Small)	6	6.737	6.726	-0.011
Semi-orchestration (TTS experts)	4	5.428	4.719	-0.709
Semi-orchestration (image experts)	6	6.797	6.674	-0.122
Any-to-any	4	5.792	5.613	-0.179

Appendix

Table 13: Mean Total Average by paradigm under valid-only and zero-filled scoring. The three judges are averaged equally. Changes are zero-filled minus valid-only scores.

	Experts		Missing candidates			Total Average
Backbone	Image	TTS	T	I	S	Valid-only	Zero-filled	$\Delta$
Orchestration (large VLMs)
GPT-5.4	D	P	0	0	0	7.5305	7.5302	-0.0003
GPT-5.4	K	P	0	0	0	7.6986	7.6983	-0.0003
GPT-5.4	D	V	0	0	0	7.5426	7.5397	-0.0029
GPT-5.4	K	V	0	0	0	7.7123	7.7094	-0.0029

Appendix

Table 14: Candidate availability and Total Average under valid-only and zero-filled scoring for all 32 configurations. Missing counts refer to unavailable text (T), image (I), and speech (S) candidates among 900 examples, independently of judge validity; they are not all planning failures and may overlap across modalities. Scores use the equal-weight three-judge average and the same seven-component Total Average as the main analysis. Zero filling also includes invalid evaluation records, so missing-candidate counts alone do not determine the score change.

\Delta

is zero-filled minus valid-only, calculated before rounding. External image expert: D=FLUX.1-dev, K=FLUX.1-Kontext, N=Nitro-T-1.2B; TTS: P=Parler-large, V=VoxCPM2; – means no external expert.

Diagnostic	Text	Image	Speech
Weakest-modality count	9	14	9
Mean weakest-modality rank gap (pp)	2.47	6.55	2.62
IQR of modality score	1.22	1.86	0.55
Max Total (Bottom Tercile)	6.94	6.55	6.72

Appendix

Table 15: Zero-filled modality-bottleneck diagnostics across 32 evaluated systems. Missing or invalid evaluations contribute zero in the fixed 900-sample denominator. Bold marks the strongest diagnostic signal in each row.

	Valid-only		Zero-filled
Association	$r$	$p$	$r$	$p$
All configurations ( $N=32$ )
Text–Image $\mid$ Speech	0.626	$1.7\times 10^{-4}$	0.560	0.001
Text–Speech $\mid$ Image	0.758	$8.0\times 10^{-7}$	0.812	$3.0\times 10^{-8}$
Image–Speech $\mid$ Text	-0.122	0.515	-0.058	0.756
Semi-orchestration and any-to-any ( $N=14$ )

Appendix

Table 16: Partial correlations between modality scores under valid-only and zero-filled scoring. The variable after

\mid

is controlled. The restricted group includes all 14 semi-orchestration and any-to-any configurations.

	Valid-only		Zero-filled
Modality and controls	$r$	$p$	$r$	$p$
Text $\mid$ Image, Speech	0.921	$5.7\times 10^{-13}$	0.831	$1.4\times 10^{-8}$
Image $\mid$ Text, Speech	0.637	$1.5\times 10^{-4}$	0.378	0.039
Speech $\mid$ Text, Image	0.123	0.519	0.342	0.065

Appendix

Table 17: Partial correlations of the integrated judge score with each modality score, controlling for the other two modalities (

N=32

configurations). The scoring policy applies to both the integrated score and the modality scores.

Track	Submitted	Skipped/unscored	Scored	Category scores
Text	189	2	187	748
Image	168	0	168	672
Speech	151	0	151	604
Integrated	189	8	181	724
Total	697	10	687	2,748

Appendix

Table 18: Human-evaluation counts. Each scored evaluation supplies four rubric-category scores. Integrated evaluation is also called Joint in the study records.

Track	Spearman $\rho$	Kendall $\tau_{b}$	Pearson $r$	LOTO $\rho$ range
Text	0.863	0.677	0.919	0.820–0.876
Image	0.882	0.714	0.928	0.866–0.918
Speech	0.724	0.559	0.793	0.665–0.800
Integrated	0.884	0.712	0.906	0.849–0.902
Four-track composite	0.945	0.806	0.957	0.934–0.955

Appendix

Table 19: Agreement between human ratings and the current three-judge means across 32 system configurations. Leave-one-transition-out (LOTO) removes each human-study transition in turn while holding the full-benchmark automated comparator fixed. Systems without remaining human ratings are omitted in that repetition: individual-track analyses retain 30–32 systems and the composite retains 28–32. LOTO ranges are sensitivity ranges, not confidence intervals.

Paradigm tier	Systems	Human	Three-judge
Orchestration, large VLM	12	7.84	7.83
Orchestration, small VLM	6	6.73	6.49
Semi-orchestration	10	5.14	5.76
Native any-to-any	4	4.61	4.90

Appendix

Table 20: Paradigm-tier means for the human study’s four-track rubric composite. These values are distinct from the seven-component Total Average in the main experiments.

Track	Human ordinal $\alpha$ range	Judge subset/full Spearman $\rho$
Text	0.46–0.72	0.946
Image	0.72–0.84	0.966
Speech	0.26–0.35	0.840
Integrated	0.48–0.64	0.932

Appendix

Table 21: Two complementary diagnostics. The

\alpha

range is ordinal Krippendorff inter-annotator agreement across the four rubric categories within each track. The final column is the system-ranking correlation between the current three-judge scores on the 10-transition window and on all 900 transitions; it is not a human–judge correlation.

Score	A–B	A–C	B–C
Text	0.990 / 0.930	0.969 / 0.874	0.966 / 0.851
Image	0.969 / 0.899	0.977 / 0.899	0.975 / 0.903
Speech	0.730 / 0.542	0.404 / 0.360	0.595 / 0.429
Integrated	0.909 / 0.754	0.956 / 0.839	0.935 / 0.810
Four-track composite	0.972 / 0.883	0.985 / 0.927	0.990 / 0.940
Total Average	0.986 / 0.919	0.993 / 0.948	0.992 / 0.948

Appendix

Table 22: System-level agreement across the 32 configurations. Each cell reports Spearman’s

\rho

/ Kendall’s

\tau_{b}

. A, B, and C are the three judge panels defined in the text. The four-track composite contains only rubric scores; Total Average also contains three automatic metrics shared across panels.

Track	Pair	Paired observations	Spearman $\rho$	MAE
Text	A–B	28,194	0.873	1.011
Text	A–C	28,191	0.872	1.017
Text	B–C	28,191	0.888	1.433
Image	A–B	28,177	0.864	0.994
Image	A–C	28,177	0.806	1.078
Image	B–C	28,177	0.803	0.933

Appendix

Table 23: Agreement on paired configuration–example observations where both judges have valid scores. Scores are the means of the four rubric criteria on the 1–10 scale. MAE is the mean absolute score difference. These descriptive statistics pool configuration–example observations; repeated transitions and shared generation modules mean that the rows are not independent samples.

ID	Backbone	I/S	Total	95% score interval	Rank	Rank interval
C10	Claude-Opus-4.7	K/P	7.749	[7.711, 7.786]	1	[1, 2]
C12	Claude-Opus-4.7	K/V	7.729	[7.693, 7.765]	2	[1, 4]
C04	GPT-5.4	K/V	7.712	[7.676, 7.748]	3	[2, 4]
C02	GPT-5.4	K/P	7.699	[7.661, 7.735]	4	[2, 4]
C08	GLM-4.6V	K/V	7.655	[7.617, 7.691]	5	[5, 7]
C06	GLM-4.6V	K/P	7.631	[7.591, 7.670]	6	[5, 8]

Appendix

Table 24: Current valid-only totals and paired-bootstrap percentile intervals. Rows follow the point-estimate ranking; C01–C32 identify the configurations in the order of the full results table in Appendix G. I/S denotes image/speech experts: D=FLUX.1-dev, K=FLUX.1-Kontext, N=Nitro-T-1.2B, P=Parler-large, V=VoxCPM2, and –=no external expert. Score intervals and rank intervals are marginal, not simultaneous confidence sets.

Comparison set	Pairs	95% CI excludes zero	Holm rejections
All pairs	496	476	462
Adjacent point-estimate ranks	31	17	8
Large orchestration vs. remaining	240	240	240

Appendix

Table 25: Paired comparisons of Total Average. The last column uses normal-approximation tests with paired-bootstrap standard errors and one Holm family comprising all 496 pairs; subset rows do not receive separate corrections.

$N$	Association	$r$	95% transition interval
32	Text–Image	0.824	[0.815, 0.832]
32	Text–Speech	0.880	[0.855, 0.898]
32	Image–Speech	0.693	[0.661, 0.717]
32	Text–Image $\mid$ Speech	0.626	[0.583, 0.669]
32	Text–Speech $\mid$ Image	0.758	[0.717, 0.787]
32	Image–Speech $\mid$ Text	-0.122	[-0.182, -0.054]

Appendix

Table 26: Paired-transition bootstrap intervals for modality associations over fixed sets of configurations. The variable after

\mid

is controlled. These are unadjusted percentile intervals conditional on the evaluated configurations, not intervals over independently sampled model families.

Component	Range	SD	Share (%)
BERTScore $\times 10$	8.020–9.081	0.153	1.62
Text judge	3.295–9.389	1.820	28.76
CLIP $\times 10$	5.827–8.513	0.604	7.36
Image judge	2.028–6.808	1.623	25.29
Speech metadata $\times 10$	4.190–5.864	0.371	4.63
Speech judge	3.286–6.987	0.678	9.60

Appendix

Table 27: Observed component ranges, sample SDs, and covariance shares of Total Average variance across the 32 current configurations. Automatic metrics are scaled to 0–10. Unrounded shares sum to 100%; they describe covariance, not independent or causal contributions.

Aggregation	$\rho$	$\tau_{b}$	First	Top five
Equal seven	1.000	1.000	Claude/K/P	5/5
Modality balanced	0.995	0.972	Claude/K/P	5/5
Automatic only	0.905	0.754	Dynin-Omni	3/5
Judges only	0.982	0.919	Claude/K/V	4/5
Z-score mean	0.990	0.927	Claude/K/P	4/5
Min–max mean	0.993	0.940	Claude/K/P	4/5

Appendix

Table 28: Sensitivity relative to equal weighting of seven components. The top-five column counts shared configurations. Claude=Claude-Opus-4.7; GPT=GPT-5.4; K=FLUX.1-Kontext; P=Parler-large; V=VoxCPM2. M1 and M2 are defined in the text. These are weighting and length-adjustment comparisons, not bootstrap intervals.

Backbone	I	S	BERT	CLIP	Meta	$J_{T}$	$J_{I}$	$J_{S}$	$J_{J}$	Total
GPT-5.4	D	P	0.882	0.732	0.584	9.386	5.788	6.752	8.797	7.530
GPT-5.4	K	P	0.882	0.772	0.584	9.384	6.489	6.752	8.879	7.699
GPT-5.4	D	V	0.882	0.732	0.566	9.387	5.787	6.987	8.830	7.543
GPT-5.4	K	V	0.882	0.772	0.566	9.389	6.499	6.987	8.909	7.712
GLM-4.6V	D	P	0.882	0.707	0.565	9.217	5.995	6.522	8.773	7.436
GLM-4.6V	K	P	0.882	0.771	0.565	9.217	6.652	6.522	8.847	7.631

Appendix

Table 29: Valid-only results for all 32 configurations. BERTScore, CLIP similarity, and speech metadata accuracy (Meta) use their native 0–1 scale.

J_{T},J_{I},J_{S},J_{J}

are text, image, speech, and integrated judge scores, respectively; each averages the three judge-specific valid means equally. Total is

(10\,\mathrm{BERT}+10\,\mathrm{CLIP}+10\,\mathrm{Meta}+J_{T}+J_{I}+J_{S}+J_{J})/7

, calculated before rounding. Image expert (I): D=FLUX.1-dev, K=FLUX.1-Kontext, N=Nitro-T-1.2B. Speech expert (S): P=Parler-large, V=VoxCPM2; – indicates no external expert.

Backbone	Valid	Mean characters
Ground truth	900	106.65
GPT-5.4	900	226.43
GLM-4.6V	900	219.19
Claude-Opus-4.7	900	244.69
InternVL3.5-4B	900	240.08
Qwen3.5-4B	900	146.14

Appendix

Table 30: Narration length supporting the text-metric comparison. Length is the number of Unicode characters after trimming surrounding whitespace, not the number of tokens. Valid means a nonempty generated text candidate, independently of judge validity. Configurations sharing a backbone are collapsed only after verifying identical text and validity for all 900 examples. Counts describe one shared set of 900 outputs, not pooled copies. The ground-truth row uses all 900 next-page narrations.

Paradigm	$N$	Gender	Speed	Pitch	Emotion
Orchestration (Large)	12	85.7	58.7	48.2	33.2
Orchestration (Small)	6	79.8	55.4	46.5	31.1
Semi-orch. (TTS experts)	4	65.4	55.7	47.0	35.9
Semi-orch. (image experts)	6	66.0	46.8	52.1	50.9
Any-to-any	4	66.5	49.9	48.7	38.4
All configurations	32	76.0	54.4	48.5	37.1

Appendix

Table 31: Speech metadata classifier accuracy (%). Each attribute is first averaged over valid classifier predictions within a configuration, then configurations are weighted equally within each row;

N

is the number of configurations. All configurations weights the 32 configurations equally, not the five paradigms. Valid zero-accuracy predictions remain included. Classifier coverage is independent of LLM-judge coverage. These are classifier agreement rates with target labels, not human perceptual accuracy.

Backbone	Image expert	Valid	Missing	Mean $\downarrow$	SD
Dynin-Omni	–	900	0	0.2945	0.1279
Claude-Opus-4.7	FLUX.1-Kontext	900	0	0.3895	0.1046
GLM-4.6V	FLUX.1-Kontext	900	0	0.4007	0.1114
EMOVA-7B	FLUX.1-Kontext	876	24	0.4008	0.1212
GPT-5.4	FLUX.1-Kontext	900	0	0.4014	0.1108
Qwen3.5-4B	FLUX.1-Kontext	900	0	0.4105	0.1101

Appendix

Table 32: DreamSim distance on current generated images, using valid images only (lower is better). SD is the population standard deviation over valid examples. All 32 configurations are represented by 24 rows: configurations differing only in TTS are merged only when their image hashes, reference hashes, missingness, and per-example distances are identical. Valid and missing counts in each row sum to 900 slots. Missing images are excluded rather than assigned zero. DreamSim is a separate diagnostic and is not included in Total Average.

Explore similar work

Mar 25, 2026cs.CL

OmniACBench: A Benchmark for Evaluating Context-Grounded Acoustic Control in Omni-Modal Models

Most testbeds for omni-modal models assess multimodal understanding via textual outputs, leaving it unclear whether these models can properly speak their answers. To study this, we introduce OmniACBench, a benchmark for evaluating context-grounded acoustic control in omni-modal models. Given a spoken instruction, a text script, and an image, a model must read the script aloud with an appropriate tone and manner. OmniACBench comprises 3,559 verified instances covering six acoustic features: speech rate, phonation, pronunciation, emotion, global accent, and timbre. Extensive experiments on eight models reveal their limitations in the proposed setting, despite their strong performance on prior textual-output evaluations. Our analyses show that the main bottleneck lies not in processing individual modalities, but in integrating multimodal context for faithful speech generation. Moreover, we identify three common failure modes-weak direct control, failed implicit inference, and failed multimodal grounding-providing insights for developing models that can verbalize responses effectively.

Seunghee Kim, Bumkyu Park, Kyudan Jung +5

Hanyang University · Seoul National University · KAIST AI +1

Omni-Modal Multimodal Benchmarks Multimodal Understanding Acoustic

Ask PDF

May 11, 2026cs.CV

Omni-Persona: Systematic Benchmarking and Improving Omnimodal Personalization

While multimodal large language models have advanced across text, image, and audio, personalization research has remained primarily vision-language, with unified omnimodal benchmarking that jointly covers text, image, and audio still limited, and lacking the methodological rigor to account for absent-persona scenarios or systematic grounding studies. We introduce Omni-Persona, the first comprehensive benchmark for omnimodal personalization. We formalize the task as cross-modal routing over the \emph{Persona Modality Graph}, encompassing 4 task groups and 18 fine-grained tasks across

{\sim}750

items. To rigorously diagnose grounding behavior, we propose \emph{Calibrated Accuracy (

\mathrm{Cal}

)}, which jointly rewards correct grounding and appropriate abstention, incorporating absent-persona queries within a unified evaluation framework. On our dedicated experiments, three diagnostic findings emerge: (i) open-source models show a consistent audio-vs-visual grounding gap that RLVR partially narrows via dense rule-based supervision; (ii) answerable recall and parameter scale are incomplete diagnostics, since strong recall can coexist with absent-persona hallucination and larger models do not always achieve higher

\mathrm{Cal}

, exposing calibration as a separate evaluation axis; and (iii) SFT is bounded by the difficulty of constructing annotated ground-truth supervision at scale, while RLVR generalizes more consistently through outcome-level verifiable feedback yet drifts toward conservative behavior and lower generation quality under our reward design. Omni-Persona thus serves as a diagnostic framework that surfaces the pitfalls of omnimodal personalization, guiding future post-training and reward design.

Yeongtak Oh, Dongwook Lee, Sangkwon Park +2

Department of Electrical and Computer Engineering, Seoul National University · Interdisciplinary Program in Artificial Intelligence, Seoul National University · Department of Artificial Intelligence, University of Seoul

Multimodal Benchmarks Personalization Visual Hallucinations

Ask PDF

Aug 11, 2026cs.AI

Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence

Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, but a spoken answer still leaves the agent visually absent. We introduce \textbf{Ex-Omni-2D}, a framework that answers a multimodal query with coordinated text, personalized speech, and reference-conditioned video. The dialogue model first writes a structured \textit{Visual Thought Plan} (VTP) for scene, emotion, and motion, then generates the response text and multi-codebook speech units. These speech units are decoded into audio and aligned with video frames, giving the speech and avatar modules a common timing signal while allowing them to learn from different data sources. The video module is trained as a full-sequence Teacher conditioned on reference appearance, VTP semantics, and frame-aligned speech units. We further explore to distill it into a few-step block-causal \emph{Streaming Student}; its Prefix Streaming mechanism carries the previous clean latent into the next chunk and is analyzed as a partial mitigation for late-chunk subject drift. At

400\times720

720\times400

, the four-step four-GPU Student provides incremental output with lower startup latency than the full-sequence Teacher.

Haoyu Zhang, Zhipeng Li, Xiaoying Tang +2

The Chinese University of Hong Kong, Shenzhen · LIGHTSPEED · Independent Researcher

Omni-Modal Omni Dialogue

Ask PDF