Omni-Workspace: Benchmarking Omni-Modal AI Assistants on Multimodal Workspace Tasks
Organizations: Shanghai Jiao Tong University · Tongyi Lab, Alibaba Group · Nanyang Technological University · University of Chinese Academy of Sciences
Abstract
Omni-modal agents are expected to move beyond multimodal understanding toward completing complex tasks in realistic multimodal environments. In these settings, information is distributed across heterogeneous files and modalities, with cross-file and cross-modal dependencies requiring agents to continually coordinate perception, reasoning, and tool use as newly gathered evidence informs subsequent actions. However, existing omni-modal benchmarks primarily focus on multimodal perception and understanding, leaving complex task completion through sustained interaction with multimodal workspaces underexplored. We introduce Omni-Workspace, a benchmark comprising 380 deep-research-style tasks across four multimodal workspace types and two difficulty levels. These tasks require agents to inspect and process workspace files, retrieve external information when needed, integrate evidence across files and modalities, and produce artifacts satisfying explicit requirements. Evaluating 18 representative models reveals limitations in long-horizon multimodal workspace execution, with the best model scoring only 65.93 on average. Further analysis identifies multimodal context inflation and task-state drift as recurring challenges. Motivated by these findings, we further propose OmniFocus, a lightweight approach combining multimodal working-memory management with explicit task-state tracking to improve performance and efficiency. Omni-Workspace advances the evaluation and development of omni-modal agents toward autonomous completion of complex tasks in multimodal workspaces.
Figures & tables
| Model | Workspace Type | Difficulty Levels | Avg. | ||||
|---|---|---|---|---|---|---|---|
| V+A+I+T | V+A+T | V+I+T | A+I+T | L1 | L2 | ||
| Native Omni-Modal Agents | |||||||
| Gemini 3.5 Flash | 68.29 | 61.72 | 65.52 | 68.42 | 72.77 | 57.38 | 65.93 |
| Gemini 3.1 Pro Preview | 65.01 | 52.43 | 58.59 | 63.93 | 65.13 | 53.36 | 59.89 |
| Gemini 3 Flash Preview | 59.40 | 51.65 | 55.34 | 56.33 | 60.87 | 49.06 | 55.62 |
| Gemini 2.5 Pro | 52.36 | 44.53 | 47.06 | 49.65 | 53.50 | 41.93 | 48.35 |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| Benchmark | Modalities | Task Dependencies | Agentic Operations | Output Form | |||||
| Aud. | Vid. | Img. | Cross-file Dependency | Intermediate Artifact Reuse | Web Retrieval | Media Processing | Workspace File I/O | ||
| OmniEval | Short-form Answer | ||||||||
| OmniVideoBench | Short-form Answer | ||||||||
| LVOmniBench | Short-form Answer | ||||||||
| OmniGAIA | Short-form Answer | ||||||||
| Omni-DeepSearch | Short-form Answer | ||||||||
| Check | Acceptance criteria |
|---|---|
| Solvability and necessity | Required information is supported by workspace or Web evidence and obtainable with the assigned tools. Cross-file and cross-modal dependencies contribute to the requested result. For L2 tasks, processing produces information needed for completion. |
| Answer leakage | Instructions, filenames, metadata, and supporting documents do not disclose target conclusions or intermediate identities that the agent is expected to infer. |
| Evaluation correctness | The reference output is supported by evidence and satisfies the requested content and format. Rubric checks correspond to explicit requirements, accommodate equivalent correct answers, and introduce no hidden requirements or unnecessary restrictions on solution paths. |
| Category | Tool Name | Description | Parameters |
| Workspace | list_workspace_files | List visible input files and files generated for the current task. | — |
| get_metadata | Retrieve file format, size, duration, resolution, and aspect ratio, as applicable. | path | |
| read_text_file | Read a text file with an optional character limit. | path , max_chars * | |
| write_text_file | Write text to the current task’s output directory. | path , content | |
| Perception | read_image | Provide an image as visual input to the model. | path |
| read_video | Provide a video as multimodal input to the model. | path |
| Model | Workspace Type | Difficulty Levels | Overall | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| V+A+I+T | V+A+T | V+I+T | A+I+T | L1 | L2 | All 380 tasks | |||||||||||||||
| C | F | S | C | F | S | C | F | S | C | F | S | C | F | S | C | F | S | C | F | S | |
| Native Omni-Modal Agents | |||||||||||||||||||||
| Gemini 3.5 Flash | 65.85 | 90.16 | 68.29 | 59.00 | 86.24 | 61.72 | 63.41 | 84.47 | 65.52 | 66.09 | 89.32 | 68.42 | 70.95 | 89.17 | 72.77 | 54.26 | 85.54 | 57.38 | 63.53 | 87.56 | 65.93 |
| Gemini 3.1 Pro Preview | 61.63 | 95.46 | 65.01 | 48.07 | 91.70 | 52.43 | 55.45 | 86.80 | 58.59 | 60.69 | 93.16 | 63.93 | 62.11 | 92.32 | 65.13 | 49.15 | 91.20 | 53.36 | 56.35 | 91.82 | 59.89 |
| Gemini 3 Flash Preview | 55.54 | 94.11 | 59.40 | 47.84 | 85.94 | 51.65 | 51.36 | 91.17 | 55.34 | 52.34 | 92.28 | 56.33 | 57.49 | 91.34 | 60.87 | 44.50 | 90.13 | 49.06 | 51.71 | 90.80 | 55.62 |
| Model / configuration | NSI | MSU |
|---|---|---|
| Gemini 3.5 Flash (native) | 62.39 | 66.77 |
| Gemini 3.1 Pro (native) | 63.02 | 63.14 |
| Qwen3.7-Plus (ASR + tools) | 52.89 | 64.67 |
| Qwen3.7-Plus (ASR-only) | 51.37 | 53.78 |
| Qwen3.6-Plus (ASR + tools) | 47.75 | 63.09 |
| Category | Definition | Example |
|---|---|---|
| Local execution errors | ||
| Multimodal perception | Incorrectly recognizing or transcribing task-relevant content in images, video, or audio. | Mistaking the person who speaks in a video for another visible person. |
| Search | Failing to find required information, or selecting a result about the wrong entity, source, time, or version. | Selecting the release date of a different film with the same title. |
| Tool invocation | Using incorrect parameters, paths, code, or operations that prevent a tool from performing the intended action correctly. | Processing media using an incorrect file path. |
| Task-state drift errors | ||
| Requirement omission | Omitting or replacing an explicit requirement, or incorrectly treating it as completed. | Analyzing only one recording when the task requires comparing two. |
| Metric | Qwen–Gemini | Qwen–Claude | Gemini–Claude |
|---|---|---|---|
| Atomic agreement | 93.23% | 95.49% | 93.39% |
| Cohen’s | 0.865 | 0.910 | 0.868 |
| Positive agreement | 93.47% | 95.43% | 93.67% |
| Negative agreement | 92.97% | 95.55% | 93.09% |
| Spearman’s | 0.9897 | 0.9897 | 0.9959 |
| Kendall’s | 0.9477 | 0.9477 | 0.9739 |