Click-to-Ask: An AI Live Streaming Assistant with Offline Copywriting and Online Interactive QA

Ruizhi Yu; Keyang Zhong; Peng Liu; Qi Wu; Haoran Zhang; Yanhao Zhang; Chen Chen; Haonan Lu

cs.CVMar 19, 2026

Click-to-Ask: An AI Live Streaming Assistant with Offline Copywriting and Online Interactive QA

Authors: Ruizhi Yu, Keyang Zhong, Peng Liu, Qi Wu, Haoran Zhang, Yanhao Zhang, Chen Chen, Haonan Lu

Organizations: East China normal University, Shanghai, China · Sun Yat-sen University, Guangzhou, China · OPPO AI Center, Shenzhen, China · Shenzhen Institutes of Advanced Technology, Shenzhen, China

Abstract

Live streaming commerce has become a prominent form of broadcasting in the modern era. To facilitate more efficient and convenient product promotions for streamers, we present Click-to-Ask, an AI-driven assistant for live streaming commerce with complementary offline and online components. The offline module processes diverse multimodal product information, transforming complex inputs into structured product data and generating compliant promotional copywriting. During live broadcasts, the online module enables real-time responses to viewer inquiries by allowing streamers to click on questions and leveraging both the structured product information generated by the offline module and an event-level historical memory maintained in a streaming architecture. This system significantly reduces the time needed for promotional preparation, enhances content engagement, and enables prompt interaction with audience inquiries, ultimately improving the effectiveness of live streaming commerce. On our collected dataset of TikTok live stream frames, the proposed method achieves a Question Recognition Accuracy of 0.913 and a Response Quality score of 0.876, demonstrating considerable potential for practical application. The video demonstration can be viewed here: https://www.youtube.com/shorts/mWIXK-SWhiE.

Figures & tables

Figure 1. Framework of Click-to-Ask. The upper half illustrates the offline copywriting module, including product information integration, copywriting generation, prohibited-term purification, and the structured product database. The lower half shows the online interactive Q&A module on the left, responsible for real-time bullet-chat responses, and the event-based history memory module on the right, which asynchronously segments the video stream and accelerates historical caption generation.

Baseline	RQ						QRA
Baseline	electronics	appliances	clothing	food	others	mean	QRA
Qwen2.5-VL 7B	0.361	0.333	0.758	0.774	0.500	0.536	0.512
+GRPO	0.479	0.349	0.808	0.758	0.628	0.599	0.619
+visual prompts	0.493	0.420	0.875	0.750	0.600	0.620	0.591
+visual prompts +GRPO	0.872	0.835	1.000	0.823	0.887	0.876	0.913

Table 1. Performance comparison of different methods on the test set.

Figure 2. Some visual results on mobile devices.

E-Commerce

E-commerce studies digital marketplaces, including product discovery, recommendations, pricing, transactions, and AI-assisted shopping and operations.

See all 69 papers

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams
Sep 23, 2026Shujian Gao, Jiamei Yan, Yuchen Yang +6Human-AI Interaction Proactive Assistance
Institute of Trustworthy Embodied AI, Fudan University · ByteDance TikTok · Shanghai Innovation Institution +1
TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming
Aug 21, 2026Yibo Hu, Yu Qian, Mao Gu +6Multimodal Understanding Streaming Video Understanding
Taobao & Tmall Group of Alibaba
Ottilie: A Socially Intelligent Virtual Host for Sales-Driven Live Commerce
May 14, 2026Yuyan ChenE-Commerce Conversational Agents
Cornell University

Sep 23, 2026J/K move · Enter open · S save

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

Shujian Gao, Jiamei Yan, Yuchen Yang, Penghao Zhou+5

Institute of Trustworthy Embodied AI, Fudan University · ByteDance TikTok · Shanghai Innovation Institution+1

Human-AI Interaction Proactive Assistance LLM Agents Multimodal Agents Mixed-Initiative Interaction

Open paper AskPDF arXiv

Livestreams are long-lasting interactive environments where audiovisual content, viewer activity, host behavior, and platform signals evolve together, creating assistance needs that emerge from the stream itself. We introduce \liveassistant, a framework for mixed-initiative, role-conditioned assistance that formulates livestream interaction as four coupled decisions: whether to act, when to act, whom to address, and what to communicate. At each 10-second interval, one autoregressive policy consumes native audio and video with synchronized comments, gifts, viewer dynamics, and room metadata, then selects OBS, MEM, or ANS. OBS remains silent, MEM records a private semantic update, and ANS specifies a recipient, task, and grounded message. To support this task, we build a trajectory engine that reconstructs real livestream sessions into structured causal supervision, yielding over 320 hours of optimization trajectories and a human-reviewed benchmark of 275 clips and 13,812 decision intervals. We train the policy with Marker-Aware Multiturn Supervised Fine-Tuning (MA-MSFT), which strengthens sparse structured decisions, followed by Streaming Multiturn GSPO (SM-GSPO), which optimizes self-generated trajectories with turn- and trajectory-level credit. On the held-out benchmark, \liveassistant reaches 71.14 state accuracy, 72.67 recipient accuracy, and 58.41 task accuracy, with consistent gains over representative streaming and general multimodal baselines. Together, the formulation, benchmark, and training framework establish livestream assistance as selective participation in a shared social stream.

Abstract

E-Commerce

Explore similar work

Live Assistant: Learning Whether, When, and Whom to Assist in Real-World Live Social Streams

TLive-Omni: An Omni-Modal Understanding Model for E-Commerce Live Streaming

Ottilie: A Socially Intelligent Virtual Host for Sales-Driven Live Commerce