TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Yibo Hu; Yu Qian; Mao Gu; Yingfan Tao; Yuhao Chen; Yongdong Luo; Zhuoqun Liu; Meiguang Jin

cs.AIAug 21, 2026

TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Authors: Yibo Hu, Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Meiguang Jin,

Organizations: Taobao & Tmall Group of Alibaba

Abstract

E-commerce live streaming requires omni-modal understanding of noisy, temporally extended streams, where product facts are distributed across speech, video frames, product images, overlaid text, and user queries. We present TLive-Omni, an omni-modal understanding model tailored to live-commerce scenarios. It maps image, video, audio, and text inputs into a unified representation space. For long-form live streaming analysis, we introduce Per-vGrid, a timestamped token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens to facilitate temporal alignment. We design a three-stage supervised training recipe that progressively develops live-commerce understanding, from omni-modal perception to instruction-following responses. We then propose Faithful-RFT, a reinforcement fine-tuning stage that further improves answer faithfulness and expression quality while meeting real-time demands, scoring final responses directly with task-verifiable feedback rather than optimizing for reasoning-style exploration during rollout. Moreover, TLive-Omni is supported by a scenario-oriented atomic capability taxonomy and a compact data production engine that converts live-commerce audio, image, and video streams into training signals for speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc. For scalable training, a synchronized length-grouped sampler reduces padding while preserving comparable workloads across workers, while a lightweight dynamic sampling strategy regenerates rollout groups with near-zero reward variance to maintain meaningful relative advantages for GRPO. Experiments on e-commerce live streaming benchmarks demonstrate strong performance across live-commerce domain tasks, together with excellent generalization on general benchmarks.

Figures & tables

Figure 1: The architectural overview of TLive-Omni. It is built upon a Qwen3.5 backbone and extended with the AuT audio encoder through a lightweight audio aligner. The inset illustrates the Per-vGrid token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens.

Figure 2: TLive-Omni three-stage SFT data construction framework. Modality-specific generation and quality control transform audio, image, and video sources into task-grounded supervision for the three-stage SFT recipe.

Closed-source Omni models
Model	Params	Live-Commerce ASR	Speaker-Attributed ASR	Audio Description		Audio QA
Model	Params	CER $\downarrow$	cpWER $\downarrow$	Acc. $\uparrow$	Hal. $\downarrow$	Acc. $\uparrow$
Gemini 2.5 Flash	-	16.30	17.14	65.21	26.19	76.28
Gemini 2.5 Pro	-	11.48	12.17	81.10	14.16	82.85
Gemini 3 Flash	-	15.18	19.04	68.27	26.17	74.68
Gemini 3 Pro	-	12.09	11.67	85.07	10.92	88.62

Table 1: Live-commerce audio evaluation covering live-commerce ASR, speaker-attributed ASR, audio description and question answering. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source Omni models
Model	Params	Visual Grounding		Text Understanding
Model	Params	Live AP $\uparrow$	Prod AP $\uparrow$	Loc. F1 $\uparrow$	Rec. NED $\downarrow$	Cls. Acc. $\uparrow$
Gemini 2.5 Flash	-	61.08	28.81	20.52	43.28	51.21
Gemini 2.5 Pro	-	51.98	32.63	31.60	27.82	61.86
Gemini 3 Flash	-	80.38	65.67	61.11	16.25	69.11
Gemini 3 Pro	-	73.80	58.83	68.60	9.72	76.86

Table 2: Live-commerce image evaluation covering product visual grounding and text understanding. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source Omni models
Model	Params	TG	Dense Caption		Video QA	Shot Understanding
Model	Params	mIoU $\uparrow$	Acc. $\uparrow$	Hal. $\downarrow$	Acc. $\uparrow$	Layout $\uparrow$	Shot Size $\uparrow$	Camera $\uparrow$	Content $\uparrow$
Gemini 2.5 Flash	-	76.50	54.60	10.97	88.21	80.00	46.80	84.20	68.60
Gemini 2.5 Pro	-	76.22	41.95	16.88	92.62	85.20	50.80	76.00	70.40
Gemini 3 Flash	-	77.43	32.21	20.76	89.64	76.80	45.70	78.50	71.60
Gemini 3 Pro	-	77.90	37.80	20.99	84.36	80.40	43.40	75.70	74.80

Table 3: Live-commerce video evaluation covering temporal grounding, dense video caption, video question answering, and shot understanding. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	MMMU	MathVista	DynaMath	VLMsAreBlind	MMBench	RealWorldQA	MMStar	SimpleVQA
Gemini 2.5 Flash	-	76.3	75.3	69.7	75.9	86.6	75.7	75.8	59.2
Gemini 2.5 Pro	-	80.9	77.7	78.5	78.5	88.4	76.0	78.5	66.9
Gemini 3 Pro	-	87.2	87.9	85.1	–	93.7	83.3	83.1	73.2
GPT-4o	-	70.7	63.8	54.4	–	86.0	–	–	–
GPT-5 (minimal)	-	74.4	50.9	74.0	53.4	81.3	77.3	65.2	56.7

Table 4: General image benchmark results on MMMU, MathVista, DynaMath, VLMsAreBlind, MMBench, RealWorldQA, MMStar, and SimpleVQA. MMBench results are reported on the EN-DEV-v1.1 split. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	Hallusion	AI2D	OCRBench	CC-OCR	CharXiv(RQ)	RefCOCO	ERQA	EmbSpatial
Gemini 2.5 Flash	-	59.1	87.7	86.4	74.8	60.1	–	–	–
Gemini 2.5 Pro	-	60.9	90.0	87.2	76.8	62.9	–	50.3	73.3
Gemini 3 Pro	-	68.6	94.1	90.4	79.0	81.4	84.1	70.5	61.2
GPT-4o	-	–	82.6	84.3	–	–	–	–	–
GPT-5 (minimal)	-	53.7	84.1	78.7	66.1	57.8	–	42.0	75.1

Table 5: General image benchmark results on HallusionBench, AI2D, OCRBench, CC-OCR, CharXiv(RQ), RefCOCO, ERQA, and EmbSpatialBench. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	MVBench	MLVU	Video-MME	LongVideoBench	LVBench	MMVU	VideoMMMU
Gemini 2.5 Flash	-	–	77.8	75.6	–	62.2	68.2	65.2
Gemini 2.5 Pro	-	65.8	81.2	80.6	–	69.0	72.2	79.4
Gemini 3 Pro	-	74.1	83.0	87.7	76.7	76.2	77.5	87.6
GPT-4o	-	–	–	71.9	–	–	–	–
GPT-5 (minimal)	-	64.6	78.3	77.3	–	–	68.1	61.6

Table 6: General video benchmark results on MVBench, MLVU, Video-MME, LongVideoBench, LVBench, MMVU, and VideoMMMU. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	Charades-TL	ActivityNet-TL	QVHighlights-TL
Gemini 2.5 Flash	-	48.6	52.5	64.3
Gemini 2.5 Pro	-	52.8	58.1	70.4
GPT-4o	-	41.8	40.4	52.1
GPT-5 (minimal)	-	40.5	42.9	56.8
Open-source VLM models

Table 7: Temporal grounding results on TimeLens-Bench, reported as mIoU on Charades-TL, ActivityNet-TL, and QVHighlights-TL. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	AVUT	WorldSense	VideoHolmes	DailyOmni	OmniVideoBench	FutureOmni
Gemini 2.5 Flash	-	65.4	50.9	–	–	–	55.6
Gemini 3.1 Pro	-	85.6	65.5	–	82.7	–	–
Qwen3.5-Omni Flash	-	81.4	57.9	–	81.8	–	–
Open-source Omni models
video-SALMONN 2+	3B	66.2	48.3	42.2	67.7	–	–

Table 8: General Omni benchmark results on AVUT, WorldSense, VideoHolmes, DailyOmni, OmniVideoBench, and FutureOmni. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Panel of Figure 3 — Figure 3: Qualitative live-commerce examples of TLive-Omni.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Keyword	Qwen3-ASR-Flash		Qwen3-Omni		TLive-Omni-4B		TLive-Omni-9B
Count	Recall $\uparrow$	CER $\downarrow$	Recall $\uparrow$	CER $\downarrow$	Recall $\uparrow$	CER $\downarrow$	Recall $\uparrow$	CER $\downarrow$
0	51.88	6.33	53.44	6.59	49.12	6.69	48.62	6.54
50	75.69	5.40	85.44	7.64	79.06	5.41	81.00	5.18
100	76.94	6.59	81.56	7.34	77.75	5.45	80.31	5.19
200	68.00	5.73	78.69	8.13	76.56	5.56	79.50	5.30
300	67.12	5.89	76.94	7.65	76.06	5.61	78.31	5.36

Appendix

Table 9: In-Context ASR results under different keyword-list sizes. Recall denotes keyword recall, and CER denotes character error rate. For each keyword-list size, best values are shown in bold and second-best values are underlined .

Benchmark(s)	Prompt Template
MMMU, MMBench, MMStar, AI2D	{question} Options: A. {option_a} / B. {option_b} / C. {option_c} / D. {option_d} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $LETTER’ (without quotes) where LETTER is one of the options.
VLMsAreBlind, RealWorldQA, ERQA, EmbSpatialBench	Hint: {hint} Question: {question} Options: A. {option_a} / B. {option_b} / ... Please select the correct answer from the options above.
MathVista	{hint} {question} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $ANSWER’ (without quotes) where$ ANSWER is your final answer.
CharXiv, SimpleVQA	{question} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $ANSWER’ (without quotes) where$ ANSWER is your final answer.
DynaMath	## Question {question} ## Answer Instruction Please provide an answer to the question outlined above. Your response should adhere to the following JSON format, which includes two keys: ‘solution’ and ‘short answer’. The ‘solution’ key can contain detailed steps needed to solve the question, and the ‘short answer’ key should provide a concise response. Provide the corresponding choice option in the ‘short answer’ key, such as ‘A’, ‘B’, ‘C’, or ‘D’. Example of expected JSON response format: {‘‘solution’’: ‘‘[Detailed step-by-step explanation]’’, ‘‘short answer’’: ‘‘[Concise Answer]’’}
HallusionBench	{question} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: Yes/No’ (without quotes).

Appendix

Table 13

Multimodal Understanding

Multimodal understanding studies how AI systems interpret and reason across text, images, video, audio, and other signals.

See all 478 papers

Sep 14, 2026

Omni-Streaming Thinking

Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.

Enjun Du, Siyi Liu, Ziyu Zheng +4

1The University of Hong Kong · 2The Hong Kong University of Science and Technology (Guangzhou) · University of Sussex +2

Audio-Visual Understanding Streaming Video Understanding Multimodal Reasoning Multimodal Hallucination

Ask PDF

May 25, 2026

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

While streaming omni-video understanding demands continuous perception and proactive, real-time interaction, this crucial area remains largely under-explored. Current omni-modal methods are inherently designed for offline settings, limiting their applicability in streaming scenarios due to two fundamental flaws. First, they lack robust mechanisms to manage continuously growing audio-visual context over long horizons and cannot autonomously initiate responses at opportune moments. Second, existing benchmarks are predominantly confined to offline, single-turn question answering, failing to capture continuous, multi-turn streaming interactions. To bridge these gaps, we propose StreamOV, a novel Streaming Omni-Video understanding framework for efficient online audio-visual reasoning with bounded memory and proactive response triggering. Specifically, StreamOV introduces a multimodal evidence-guided long-short term memory that condenses historical audio-visual context into compact informative evidence under a fixed budget. It further employs a hidden-state-driven trigger to decide when to respond, avoiding explicit silence-token generation and external routers. We also curate SOVBench, the first comprehensive benchmark for online, multi-turn omni-modal evaluation. Extensive experiments show that StreamOV achieves state-of-the-art performance across diverse streaming and omni-video benchmarks, demonstrating its effectiveness for both online and offline video understanding.

Ming Xie, Zizheng Huang, Xudong Tan +6

Shanghai Innovation Institute · Fudan University · Nanjing University +2

Audio-Visual Understanding Streaming Video Understanding Multimodal Memory Multimodal QA

Ask PDF

May 26, 2026

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants

We introduce OmniInteract, a streaming benchmark for real-time omnimodal large language models evaluated through native online inference over audio-visual streams. Unlike offline video understanding or text-prompted streaming QA, OmniInteract preserves the original audio-visual stream and requires models to process it online, without access to future content. User queries and ambient sounds are embedded in the audio track, requiring models to detect multimodal triggers, decide when to respond, and answer while the stream unfolds. OmniInteract contains 250 videos with 1,430 temporally grounded response slots: 1,062 1Q1A slots across real-time, proactive, and nested scenarios, and 368 1QnA slots for continuous task monitoring and step guidance. Each slot includes a trigger, response window, and target answer. We evaluate response correctness, timing, invalid outputs, interruption handling, and context continuity using Interaction-Aware Quality-Timeliness F1, Interruption Diagnostic Suite, and Nested Chain Completion Score. Experiments show that current models remain weak in streaming interaction, with the best overall IA-QTF1 reaching only 0.368 and the best 1QnA IA-QTF1 only 0.052. Further study on mathematical reasoning in full-duplex settings shows that offline capability does not necessarily transfer to online interaction. Code and datasets will be made publicly accessible at https://github.com/Lucky-Lance/OmniInteract.

Xudong Lu, Xueying Li, Annan Wang +8

CUHK MMLab · SJTU · NTU +3

Multimodal Agents Multimodal Understanding Multimodal Foundation Models Streaming Video Understanding Multimodal Model Evaluation Full-Duplex Spoken Dialogue Systems+1

Ask PDF

Abstract

Multimodal Understanding

Explore similar work

Omni-Streaming Thinking

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants