TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Yibo Hu; Yu Qian; Mao Gu; Yingfan Tao; Yuhao Chen; Yongdong Luo; Zhuoqun Liu; Meiguang Jin

cs.AIAug 21, 2026

TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Authors: Yibo Hu, Yu Qian, Mao Gu, Yingfan Tao, Yuhao Chen, Yongdong Luo, Zhuoqun Liu, Meiguang Jin,

Organizations: Taobao & Tmall Group of Alibaba

Abstract

E-commerce live streaming requires omni-modal understanding of noisy, temporally extended streams, where product facts are distributed across speech, video frames, product images, overlaid text, and user queries. We present TLive-Omni, an omni-modal understanding model tailored to live-commerce scenarios. It maps image, video, audio, and text inputs into a unified representation space. For long-form live streaming analysis, we introduce Per-vGrid, a timestamped token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens to facilitate temporal alignment. We design a three-stage supervised training recipe that progressively develops live-commerce understanding, from omni-modal perception to instruction-following responses. We then propose Faithful-RFT, a reinforcement fine-tuning stage that further improves answer faithfulness and expression quality while meeting real-time demands, scoring final responses directly with task-verifiable feedback rather than optimizing for reasoning-style exploration during rollout. Moreover, TLive-Omni is supported by a scenario-oriented atomic capability taxonomy and a compact data production engine that converts live-commerce audio, image, and video streams into training signals for speech recognition, speaker analysis, product visual grounding, text recognition, temporal grounding, video dense caption, and omni-modal QA, etc. For scalable training, a synchronized length-grouped sampler reduces padding while preserving comparable workloads across workers, while a lightweight dynamic sampling strategy regenerates rollout groups with near-zero reward variance to maintain meaningful relative advantages for GRPO. Experiments on e-commerce live streaming benchmarks demonstrate strong performance across live-commerce domain tasks, together with excellent generalization on general benchmarks.

Figures & tables

Figure 1: The architectural overview of TLive-Omni. It is built upon a Qwen3.5 backbone and extended with the AuT audio encoder through a lightweight audio aligner. The inset illustrates the Per-vGrid token organization that groups each video grid with its temporally corresponding audio within explicit boundary tokens.

Figure 2: TLive-Omni three-stage SFT data construction framework. Modality-specific generation and quality control transform audio, image, and video sources into task-grounded supervision for the three-stage SFT recipe.

Closed-source Omni models
Model	Params	Live-Commerce ASR	Speaker-Attributed ASR	Audio Description		Audio QA
Model	Params	CER $\downarrow$	cpWER $\downarrow$	Acc. $\uparrow$	Hal. $\downarrow$	Acc. $\uparrow$
Gemini 2.5 Flash	-	16.30	17.14	65.21	26.19	76.28
Gemini 2.5 Pro	-	11.48	12.17	81.10	14.16	82.85
Gemini 3 Flash	-	15.18	19.04	68.27	26.17	74.68
Gemini 3 Pro	-	12.09	11.67	85.07	10.92	88.62

Table 1: Live-commerce audio evaluation covering live-commerce ASR, speaker-attributed ASR, audio description and question answering. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source Omni models
Model	Params	Visual Grounding		Text Understanding
Model	Params	Live AP $\uparrow$	Prod AP $\uparrow$	Loc. F1 $\uparrow$	Rec. NED $\downarrow$	Cls. Acc. $\uparrow$
Gemini 2.5 Flash	-	61.08	28.81	20.52	43.28	51.21
Gemini 2.5 Pro	-	51.98	32.63	31.60	27.82	61.86
Gemini 3 Flash	-	80.38	65.67	61.11	16.25	69.11
Gemini 3 Pro	-	73.80	58.83	68.60	9.72	76.86

Table 2: Live-commerce image evaluation covering product visual grounding and text understanding. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source Omni models
Model	Params	TG	Dense Caption		Video QA	Shot Understanding
Model	Params	mIoU $\uparrow$	Acc. $\uparrow$	Hal. $\downarrow$	Acc. $\uparrow$	Layout $\uparrow$	Shot Size $\uparrow$	Camera $\uparrow$	Content $\uparrow$
Gemini 2.5 Flash	-	76.50	54.60	10.97	88.21	80.00	46.80	84.20	68.60
Gemini 2.5 Pro	-	76.22	41.95	16.88	92.62	85.20	50.80	76.00	70.40
Gemini 3 Flash	-	77.43	32.21	20.76	89.64	76.80	45.70	78.50	71.60
Gemini 3 Pro	-	77.90	37.80	20.99	84.36	80.40	43.40	75.70	74.80

Table 3: Live-commerce video evaluation covering temporal grounding, dense video caption, video question answering, and shot understanding. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	MMMU	MathVista	DynaMath	VLMsAreBlind	MMBench	RealWorldQA	MMStar	SimpleVQA
Gemini 2.5 Flash	-	76.3	75.3	69.7	75.9	86.6	75.7	75.8	59.2
Gemini 2.5 Pro	-	80.9	77.7	78.5	78.5	88.4	76.0	78.5	66.9
Gemini 3 Pro	-	87.2	87.9	85.1	–	93.7	83.3	83.1	73.2
GPT-4o	-	70.7	63.8	54.4	–	86.0	–	–	–
GPT-5 (minimal)	-	74.4	50.9	74.0	53.4	81.3	77.3	65.2	56.7

Table 4: General image benchmark results on MMMU, MathVista, DynaMath, VLMsAreBlind, MMBench, RealWorldQA, MMStar, and SimpleVQA. MMBench results are reported on the EN-DEV-v1.1 split. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	Hallusion	AI2D	OCRBench	CC-OCR	CharXiv(RQ)	RefCOCO	ERQA	EmbSpatial
Gemini 2.5 Flash	-	59.1	87.7	86.4	74.8	60.1	–	–	–
Gemini 2.5 Pro	-	60.9	90.0	87.2	76.8	62.9	–	50.3	73.3
Gemini 3 Pro	-	68.6	94.1	90.4	79.0	81.4	84.1	70.5	61.2
GPT-4o	-	–	82.6	84.3	–	–	–	–	–
GPT-5 (minimal)	-	53.7	84.1	78.7	66.1	57.8	–	42.0	75.1

Table 5: General image benchmark results on HallusionBench, AI2D, OCRBench, CC-OCR, CharXiv(RQ), RefCOCO, ERQA, and EmbSpatialBench. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	MVBench	MLVU	Video-MME	LongVideoBench	LVBench	MMVU	VideoMMMU
Gemini 2.5 Flash	-	–	77.8	75.6	–	62.2	68.2	65.2
Gemini 2.5 Pro	-	65.8	81.2	80.6	–	69.0	72.2	79.4
Gemini 3 Pro	-	74.1	83.0	87.7	76.7	76.2	77.5	87.6
GPT-4o	-	–	–	71.9	–	–	–	–
GPT-5 (minimal)	-	64.6	78.3	77.3	–	–	68.1	61.6

Table 6: General video benchmark results on MVBench, MLVU, Video-MME, LongVideoBench, LVBench, MMVU, and VideoMMMU. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	Charades-TL	ActivityNet-TL	QVHighlights-TL
Gemini 2.5 Flash	-	48.6	52.5	64.3
Gemini 2.5 Pro	-	52.8	58.1	70.4
GPT-4o	-	41.8	40.4	52.1
GPT-5 (minimal)	-	40.5	42.9	56.8
Open-source VLM models

Table 7: Temporal grounding results on TimeLens-Bench, reported as mIoU on Charades-TL, ActivityNet-TL, and QVHighlights-TL. A dash denotes an undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Closed-source models
Model	Params	AVUT	WorldSense	VideoHolmes	DailyOmni	OmniVideoBench	FutureOmni
Gemini 2.5 Flash	-	65.4	50.9	–	–	–	55.6
Gemini 3.1 Pro	-	85.6	65.5	–	82.7	–	–
Qwen3.5-Omni Flash	-	81.4	57.9	–	81.8	–	–
Open-source Omni models
video-SALMONN 2+	3B	66.2	48.3	42.2	67.7	–	–

Table 8: General Omni benchmark results on AVUT, WorldSense, VideoHolmes, DailyOmni, OmniVideoBench, and FutureOmni. A dash denotes an unreported result or undisclosed parameter count. The Best results among open-source models are marked in bold , while the second-best results are in underlined .

Panel of Figure 3 — Figure 3: Qualitative live-commerce examples of TLive-Omni.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

Keyword	Qwen3-ASR-Flash		Qwen3-Omni		TLive-Omni-4B		TLive-Omni-9B
Count	Recall $\uparrow$	CER $\downarrow$	Recall $\uparrow$	CER $\downarrow$	Recall $\uparrow$	CER $\downarrow$	Recall $\uparrow$	CER $\downarrow$
0	51.88	6.33	53.44	6.59	49.12	6.69	48.62	6.54
50	75.69	5.40	85.44	7.64	79.06	5.41	81.00	5.18
100	76.94	6.59	81.56	7.34	77.75	5.45	80.31	5.19
200	68.00	5.73	78.69	8.13	76.56	5.56	79.50	5.30
300	67.12	5.89	76.94	7.65	76.06	5.61	78.31	5.36

Appendix

Table 9: In-Context ASR results under different keyword-list sizes. Recall denotes keyword recall, and CER denotes character error rate. For each keyword-list size, best values are shown in bold and second-best values are underlined .

Benchmark(s)	Prompt Template
MMMU, MMBench, MMStar, AI2D	{question} Options: A. {option_a} / B. {option_b} / C. {option_c} / D. {option_d} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $LETTER’ (without quotes) where LETTER is one of the options.
VLMsAreBlind, RealWorldQA, ERQA, EmbSpatialBench	Hint: {hint} Question: {question} Options: A. {option_a} / B. {option_b} / ... Please select the correct answer from the options above.
MathVista	{hint} {question} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $ANSWER’ (without quotes) where$ ANSWER is your final answer.
CharXiv, SimpleVQA	{question} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: $ANSWER’ (without quotes) where$ ANSWER is your final answer.
DynaMath	## Question {question} ## Answer Instruction Please provide an answer to the question outlined above. Your response should adhere to the following JSON format, which includes two keys: ‘solution’ and ‘short answer’. The ‘solution’ key can contain detailed steps needed to solve the question, and the ‘short answer’ key should provide a concise response. Provide the corresponding choice option in the ‘short answer’ key, such as ‘A’, ‘B’, ‘C’, or ‘D’. Example of expected JSON response format: {‘‘solution’’: ‘‘[Detailed step-by-step explanation]’’, ‘‘short answer’’: ‘‘[Concise Answer]’’}
HallusionBench	{question} Think step by step before answering. The last line of your response should be of the following format: ‘Answer: Yes/No’ (without quotes).

Appendix

Table 13

Multimodal Understanding

Multimodal understanding studies how AI systems interpret and reason across text, images, video, audio, and other signals.

See all 478 papers

Omni-Streaming Thinking
Sep 14, 2026Enjun Du, Siyi Liu, Ziyu Zheng +4Audio-Visual Understanding Streaming Video Understanding
1The University of Hong Kong · 2The Hong Kong University of Science and Technology (Guangzhou) · University of Sussex +2
StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering
May 25, 2026Ming Xie, Zizheng Huang, Xudong Tan +6Audio-Visual Understanding Streaming Video Understanding
Shanghai Innovation Institute · Fudan University · Nanjing University +2
OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants
May 26, 2026Xudong Lu, Xueying Li, Annan Wang +8Multimodal Agents Multimodal Understanding
CUHK MMLab · SJTU · NTU +3

Sep 14, 2026J/K move · Enter open · S save

Omni-Streaming Thinking

Enjun Du, Siyi Liu, Ziyu Zheng, Jingyu Li+3

1The University of Hong Kong · 2The Hong Kong University of Science and Technology (Guangzhou) · University of Sussex+2

Audio-Visual Understanding Streaming Video Understanding Multimodal Reasoning Multimodal Hallucination

Open paper AskPDF arXiv

Streaming omni-modal models must decide what and when to answer from the video chunks and synchronized audio observed so far. Visual cues often support an interpretation before an utterance or sound event is complete. If that interpretation enters memory as a fact, later reasoning can keep relaying it even after audio contradicts it. We call this failure premature cross-modal commitment. We propose Omni-Streaming Thinking (OST), which generates structured outputs that include evidence observed so far, forecasts of future evidence, and claims based on this evidence. Each claim is initially marked as pending and linked to a future verification interval. Audio and visual evidence are stored separately, and OST checks a claim against the evidence from the specified modality at the end of the verification interval. When contradictory evidence is detected, a refutation process reduces the influence of the claim and its dependent states, and then guides a state update using the new evidence. An answer gate decides whether the answer-critical claims meet the conditions for giving a response. Using a frozen Qwen3-Omni-30B-A3B-Instruct backbone with lightweight adaptation, OST outperforms the strongest open baselines on five streaming and audio-visual benchmarks by more than 10% relative on average. We also introduce OST-DiagBench, which holds video fixed and edits audio to test agreement, absence, contradiction, coexistence, and subtitle-speech conflict. OST reaches d-prime = 2.95, compared with at most 1.38 for open baselines, while reducing vision-induced auditory hallucinations.

Abstract

Multimodal Understanding

Explore similar work

Omni-Streaming Thinking

StreamOV: Streaming Omni-Video Understanding via Evidence-Guided Memory and Response Triggering

OmniInteract: Benchmarking Real-World Streaming Interaction for Real-Time Omnimodal Assistants