cs.CVAug 21, 2026
SaveOmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs
Organizations: Nanjing University · Nankai University · University of Waterloo
Abstract
Recent omni-modal large language models (Omni-LLMs) show great potential as real-time video assistants, which continuously perceive environments and guide users to achieve specific goals. Unlike traditional passive video understanding, interactive assistants should actively combine visual states, user goals, and prior knowledge to provide effective help. Evaluating this is rather challenging, as the model's unpredictable response dynamically changes the user's subsequent actions, which static offline datasets cannot accommodate. To address this bottleneck, we introduce OmniAssistBench. To solve the issue of diverging interaction paths where the same user goal can be achieved through various methods, we provide models with predefined priors derived from the source video, requiring them to guide users along the exact same routes. Since real interaction videos are rare, we construct the dataset by reverse-engineering existing Internet videos. We deduce logical user goals and segment the videos into multi-turn clips to simulate continuous interactions. This rigorous pipeline required over 1000 expert person-hours to build the dataset. Results show that the proprietary Gemini-3-Pro reaches 66.4 out of the max point of 100, while the open-source Qwen3-Omni-Instruct achieves 51.2. Although current models generally understand user inputs, they frequently provide incorrect or incomplete answers. Specifically, they struggle with visual prompts (e.g., hand gestures), fail to maintain historical context during multi-turn interactions, and fail to delay response until the target event. Results indicate substantial room for improvement before models can become reliable assistants.
Figures & tables
Figure 1: (a) In traditional video understanding evaluation, model’s responses do not change the content of the test video. (b) Assistant models may guide the user to achieve one goal through different paths ( e.g . , Both Response 1 and Response 2 shown in the figure are valid, but test data only contains the consequent video of Response 1.)(c) To address this problem of path diversity, we manually filters for videos with certain plots and deduces prior knowledge from the video content to enforce a fixed interaction path. Based on the paths, experts then design detailed interaction turns and edit the videos accordingly to build human-assistant interaction recordings.
Figure 2: Task construction of OmniAssistBench. Based on the two-tier framework, the benchmark consists of 7 major tasks, 16 sub-tasks and 3 real world cases. Abbreviations in the outermost ring denote the corresponding sub-task names.
Figure 3: Distribution of video topics of OmniAssistBench. Basic tier contains more videos from movies due to the Social Perception task, as emotion labels of movie characters can be less ambiguous. Advanced tier data has a balanced topic distribution covering most common daily scenarios.
| Task | # Vid. | # QA | Turns | Avg. Vid. len (s) |
| Social Perception | 67 | 67 | 1 | 49.46 |
| Temporal Perception | 47 | 47 | 1 | 77.03 |
| Referential Perception | 49 | 49 | 1 | 40.86 |
| Non-audio Prompt Following | 30 | 76 | 2 | 149.84 |
| Context-aware Response | 20 | 20 | 1 | 119.62 |
| Proactive Response | 32 | 133 | 4 | 209.03 |
Table 1: Statistics of the dataset grouped by second-level task categories, including video counts, QA pairs, turns of interaction, and video duration metrics. Note that the number of turns denotes the annotation target rather than an exact average, as actual rounds may vary slightly depending on the video content.
Figure 4: The 4-step data construction process of OmniAssistBench, taking the Multitask Tracking (MT) task as an example. The designing of user goals and heavy video editing make it difficult to replace human labor with MLLM-based automatic pipeline.
Figure 5: Examples of 5 typical tasks in OmniAssistBench: Gesture-based Prompt Following (GPF), OCR-based Prompt Following (GPF), Context-aware Response (CR), Multitask Tracking (MT), and Multi Event-triggered Response (MER). Descriptions of the input videos are denoted as Italic .
Figure 6: Examples of key plots and questions of the 3 Real World Cases. Plot descriptions are denoted as italic .
Figure 7: Evaluation results comparison of 4 representative models at sub-task level. Abbreviations of task names are defined in Fig. 2 .
| Model | Overall Avg. | Basic Interactive Understanding Tasks | ||||||||||||||
| Social Perception | Temporal Perception | Referential Perception | Non-audio Prompt | Basic Avg. | ||||||||||||
| II | AI | CE | Avg. | ER | AO | DC | Avg. | AR | LR | Avg. | GPF | OPF | Avg. | |||
| Proprietary Models | ||||||||||||||||
| Gemini-3-Pro | 66.4 | 71.8 | 62.8 | 66.2 | 67.0 | 61.2 | 74.6 | 52.6 | 62.6 | 72.6 | 66.4 | 69.8 | 55.0 | 66.6 | 57.4 | 63.6 |
| Gemini-2.5-Pro | 64.6 | 77.2 | 59.0 | 69.6 | 68.8 | 62.6 | 72.0 | 55.0 | 63.0 | 74.0 | 78.2 | 76.0 | 54.8 | 65.4 | 56.8 | 65.4 |
| Doubao-Seed-2.0-lite | 57.3 | 64.5 | 54.3 | 50.0 | 55.9 | 56.3 | 78.7 | 51.3 | 61.7 | 72.6 | 65.5 | 69.4 | 29.5 | 57.3 | 35.0 | 53.2 |
Table 2: Performance comparison on Basic Interactive Understanding tasks. Overall Avg. denotes the average score on the overall benchmark. Best scores are marked in bold. Abbreviations of task names are defined in Fig. 2 . All original 0-5 scores have been normalized to 0-100 point.
| Model | Overall Avg. | Advanced Interactive Understanding Tasks | Real World Cases | |||||||||||
| CR | Proactive Response | Process Tracking | Avg. | |||||||||||
| SER | MER | Avg. | ST | CT | MT | Avg. | Ms | Ht | Ba | Avg. | ||||
| Proprietary Models | ||||||||||||||
| Gemini-3-Pro | 66.4 | 76.0 | 67.8 | 48.4 | 58.6 | 71.8 | 75.0 | 70.2 | 72.2 | 68.2 | 76.4 | 65.8 | 65.0 | 68.0 |
| Gemini-2.5-Pro | 64.6 | 74.0 | 61.2 | 47.8 | 54.8 | 67.8 | 82.2 | 69.4 | 72.0 | 66.4 | 51.0 | 32.8 | 50.0 | 44.8 |
| Doubao-Seed-2.0-lite | 57.3 | 76.0 | 46.6 | 62.6 | 54.9 | 68.6 | 73.8 | 50.3 | 64.9 | 62.1 | 26.0 | 21.4 | 50.0 | 35.5 |
Table 3: Performance comparison on Advanced Interactive Understanding tasks and Real World Cases. Overall Avg. denotes the average score on the overall benchmark. Best scores are marked in bold. Abbreviations of task names are defined in Fig. 2 . All original 0-5 scores have been normalized to 0-100 point.
Figure 8: Major-task level performance comparison of four candidate models under original, visual-only, and audio-only input conditions. While removing visual or audio inputs generally degrades performance, models perform better in the Proactive Response task with visual-only inputs, as this prevents them from mistakenly answering background speech instead of the user query. Additionally, Gemini-3 scores higher in the Context-aware Response task under the audio-only setting, as it avoids formatting penalties triggered by outputting timestamps when video is present.
Figure 9: Heat maps of Pearson correlation coefficients between evaluation scores from different judge LLMs across 4 candidate models. DS-v3.2 stands for DeepSeek-v3.2. All coefficient values are above 0.75, indicating that different judge models are likely to agree with each other on whether a candidate’s answer is good or not. According to the detailed scores listed in Appendix C , all judge models give similar scores for the candidates.
Figure 10: Performance comparison on multi-turn tasks under the common setting and the teacher forcing setting, where historical model responses are replaced by ground truth answers to prevent error accumulation and bridge the difficulty gap between multi- and single-turn tasks. The results indicate that independently evaluating each question in a multi-turn interaction does not always improvements model scores. Abbreviations of task names are defined in Fig. 2 . All original 0–5 scores have been normalized to 0–100.
Figure 11: Performance comparison of Qwen3-Omni-Instruct under different input video frame sizes (1080p and 360p). Reducing the input resolution to fit more video frames within the same context length does not significantly improve performance. All original 0–5 scores have been normalized to 0–100.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Score | Accuracy & Completeness | Redundancy | Response to Silence (GT = “KEEP QUIET”) |
| 5 | Perfect Execution. The model correctly comprehends the intent and semantically matches the Ground Truth (GT). It covers all Key Points. | None. Contains almost no redundant information; only minor, contextually necessary elaborations are allowed. | Exact Match. Correctly judges that silence is needed and outputs the exact required token “[KEEP QUIET]”. |
| 4 | Minor Defects. Comprehends the intent and covers all Key Points, but the response contains minor factual errors or lacks precision in description. | Low. Contains minor redundant information, provided it remains relevant to the core subject. | Descriptive. Describes the state of inaction but fails to follow the specific instruction to output the exact silence token. |
| 3 | Partial Success. Comprehends the intent but misses at least one Key Point. The answer is directionally correct but factually imperfect. | Moderate. Contains information not present in the GT, but it remains relevant to the general context of the video. | Evasive. States that there is insufficient information to answer. |
| 2 | Contradiction. Comprehends the intent but provides a clearly wrong answer (e.g., opposite polarity, wrong object) or directly contradicts the GT. | High. Contains significant irrelevant information, obvious hallucinations, or details unrelated to the query. | Hallucinated Response. Fails to understand that current turn is not a proper time for response. Gives hallucinated answers to the questions instead. |
| 1 | Loose Relation. Fails to comprehend the specific question. The response is only loosely related to the topic. | — | Loose Relation. Provides a response that is tangentially related to the context but completely ignores the silence constraint. |
| 0 | Failure. Complete failure to comprehend the input. The response is entirely unrelated to the question, instruction, or video content. | — | Failure. Output is entirely unrelated to the question or context. |
Table 4: The Scoring Rubric used in OmniAssistBench. The evaluation is based on three dimensions: Accuracy and Completeness, Redundancy, and Adherence to Silence Constraints. These rules are provided to the judgment LLM as prompts during evaluation.
| Major Task | Abbr. | Sub Task | Description | Turns |
| Proactive Response | SER | Single Event- triggered Response | “When the next taxi comes, tell me its license plate number.” Models need to wait until the target object appear, then give the answer. We do not explicitly ask the model whether to respond at the end of each turn. Instead, the user prompt is provided only in the first turn, followed by raw video clips in subsequent turns. The model must autonomously determine the appropriate timing for a response. | 4 turns in avg. |
| MER | Multi Event- triggered Response | “Whenever the professor flips to a new slide, tell me its title.” Models need to wait until the target object appear. The target event may appear several times, but the user prompt appears only once in the first turn. | 4 turns in avg. | |
| Process Tracking | ST | Step Tracking | “I want to make this dish strictly following this example video. What should I do next?” To avoid ambiguity, we provide a reference (manual images or demonstration videos) at the beginning. User might ask for the next step after completing multiple steps or while halfway through one, preventing shortcut responses that merely recite the next sentence | 6 turns in avg. |
| CT | Checklist Tracking | “This is my shopping list. Which items have not been bought yet?” A checklist is provided at the beginning as an image rendered from text. Models need to remember which objects have appeared. Objects may appear in a different order compared with the list. | 4 turns in avg. | |
| MT | Multitask Tracking | “Here is a TODO list and brief steps of how to finish each task. What is the progress of everything now?” The user constantly switch between multiple tasks. There are strict sequential steps within a task, but no dependency between tasks. Models need to track the status of each task, and remind the user whenever required. | 4 turns in avg. | |
| Context- Aware Response | CR | Context- Aware Response | “I’m on the FFmpeg download page. How can I download FFmpeg for this computer?” It can be inferred from the video that the user is using a Windows machine. We prefer the model to tailor its suggestions exclusively to the Windows platform, instead of providing generic suggestions covering all major operating systems. | Single turn |
Table 5: Detailed table of all Advanced Understanding sub-tasks. Italic sentences are explanations of each task.
| Major Task | Abbr. | Sub Task | Description | Turns |
| Social Perception | II | Identity Identification | “Who said that sentence? Who performed that action?” Models need to associate multiple actions or speeches with the same person, instead of simply giving answers like “the man in red” which can be derived from visual information directly without understanding actual identity. | Single turn |
| AI | Addressee Identification | “Who are they talking to?” There are always multiple people in the video apart from the speaker and the addressee. Models may need to check the responses of the people to understand who is the addressee. | Single turn | |
| CE | Complex Emotion Understanding | “What emotion or attitude is the person expressing?” We only keep complex samples where one modality (e.g., expression or dialogue) alone cannot lead to the correct answer. | Single turn | |
| Temporal Perception | ER | Event Retrieval | “When the main person A said xxx, what color clothes were the person passing behind them wearing?” The model needs to establish temporal correlations between the main and non-salient objects or details. | Single turn |
| AO | Appearance Order Perception | “Which colors of clothes did the blogger try on in sequence?” Model may need to distinguish the target object from similar ones (e.g., for example, the blogger may have picked up a lot of clothes but only tried on a part of them). | Single turn | |
| DC | Dynamic Counting | “How many standard pull-ups did the person perform?” Models need to count a large number of instances ( ) or filter for specific objects among similar ones. | Single turn |
Table 6: Detailed table of all Basic Understanding sub-tasks. Italic sentences are explanations of the task.
| Model | Overall Avg. | Basic Interactive Understanding Tasks | ||||||||||||||
| Social Perception | Temporal Perception | Referential Perception | Non-audio Prompt | Avg. | ||||||||||||
| II | AI | CE | Avg. | ER | AO | DC | Avg. | AR | LR | Avg. | GPF | OPF | Avg. | |||
| Proprietary Models | ||||||||||||||||
| Gemini-3-Pro (GPT-5) | 66.4 | 71.8 | 62.8 | 66.2 | 67.0 | 61.2 | 74.6 | 52.6 | 62.6 | 72.6 | 66.4 | 69.8 | 55.0 | 66.6 | 57.4 | 63.6 |
| Gemini-3-Pro (GLM-5) | 69.0 | 72.8 | 63.8 | 67.6 | 68.2 | 61.2 | 78.6 | 55.0 | 64.6 | 74.6 | 67.2 | 71.2 | 57.4 | 75.8 | 60.8 | 65.8 |
| Gemini-3-Pro (DeepSeek-v3.2) | 68.0 | 74.6 | 65.8 | 71.6 | 70.8 | 65.0 | 74.6 | 61.2 | 66.8 | 74.0 | 65.4 | 70.2 | 54.0 | 72.0 | 57.6 | 65.8 |
Table 7: Detailed evaluation scores of candidate models assessed by three different judge LLMs under the same scoring criteria. Overall Avg. denotes the average score across all evaluated tasks, while this table contains results on the Basic Interactive Understanding tasks. All original 0–5 scores have been normalized to a 0–100 scale.
More in this topic
Multimodal Large Language Models
Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.
See all 1,084 papers