Omni-Workspace: Benchmarking Omni-Modal AI Assistants on Multimodal Workspace Tasks
Authors: Yuhao Wang, Yucheng Zhao, Yiran Zhong, Feida Zhu, Hongcheng Liu, Qichao Wang, Ming Ma, Yu Wang
Organizations: Shanghai Jiao Tong University · Tongyi Lab, Alibaba Group · Nanyang Technological University · University of Chinese Academy of Sciences
Omni-modal agents are expected to move beyond multimodal understanding toward completing complex tasks in realistic multimodal environments. In these settings, information is distributed across heterogeneous files and modalities, with cross-file and cross-modal dependencies requiring agents to continually coordinate perception, reasoning, and tool use as newly gathered evidence informs subsequent actions. However, existing omni-modal benchmarks primarily focus on multimodal perception and understanding, leaving complex task completion through sustained interaction with multimodal workspaces underexplored. We introduce Omni-Workspace, a benchmark comprising 380 deep-research-style tasks across four multimodal workspace types and two difficulty levels. These tasks require agents to inspect and process workspace files, retrieve external information when needed, integrate evidence across files and modalities, and produce artifacts satisfying explicit requirements. Evaluating 18 representative models reveals limitations in long-horizon multimodal workspace execution, with the best model scoring only 65.93 on average. Further analysis identifies multimodal context inflation and task-state drift as recurring challenges. Motivated by these findings, we further propose OmniFocus, a lightweight approach combining multimodal working-memory management with explicit task-state tracking to improve performance and efficiency. Omni-Workspace advances the evaluation and development of omni-modal agents toward autonomous completion of complex tasks in multimodal workspaces.
Figures & tables
Figure 1: Overview of Omni-Workspace. An omni-modal agent continually coordinates perception, reasoning, and tool use to gather, process, and integrate information across workspace files and modalities, producing artifacts that meet task requirements. The example combines multimodal analysis, web search, and intermediate file creation and reuse to produce a structured research report.
Figure 2: Omni-Workspace construction and OmniFocus. Top: The five-stage dataset construction pipeline. Bottom: OmniFocus combines multimodal working-memory management with explicit task-state maintenance.
Figure 3: Dataset statistics of Omni-Workspace. (a) Task counts across four workspace types and two difficulty levels (L1 and L2). (b) Distribution of initial workspace file counts. (c) Counts and proportions of required output files by format, computed over 402 output files. V, A, I, and T denote video, audio, image, and text, respectively.
Model
Workspace Type
Difficulty Levels
Avg.
V+A+I+T
V+A+T
V+I+T
A+I+T
L1
L2
Native Omni-Modal Agents
Gemini 3.5 Flash
68.29
61.72
65.52
68.42
72.77
57.38
65.93
Gemini 3.1 Pro Preview
65.01
52.43
58.59
63.93
65.13
53.36
59.89
Gemini 3 Flash Preview
59.40
51.65
55.34
56.33
60.87
49.06
55.62
Gemini 2.5 Pro
52.36
44.53
47.06
49.65
53.50
41.93
48.35
Table 1: Main results on Omni-Workspace. Scores are task averages (0–100) computed using Equation 3 with λ=0.9 . V, A, I, and T denote video, audio, image, and text. Bold indicates the highest score in each column.
Figure 4: Performance on audio-focused subsets and error analysis. (a) Scores on NSI and MSU subsets. Squares denote native omni-modal models; filled and open circles denote audio tool-equipped and ASR-only configurations, respectively. (b) Error counts by category.
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
Benchmark
Modalities
Task Dependencies
Agentic Operations
Output Form
Aud.
Vid.
Img.
Cross-file Dependency
Intermediate Artifact Reuse
Web Retrieval
Media Processing
Workspace File I/O
OmniEval
✓
✓
×
×
×
×
×
×
Short-form Answer
OmniVideoBench
✓
✓
×
×
×
×
×
×
Short-form Answer
LVOmniBench
✓
✓
×
×
×
×
×
×
Short-form Answer
OmniGAIA
✓
✓
✓
×
×
✓
×
✓
Short-form Answer
Omni-DeepSearch
✓
✓
✓
×
×
✓
×
×
Short-form Answer
Appendix
Table 3: Comparison of representative benchmarks by modality coverage, task dependencies, agentic operations, and output form.
Check
Acceptance criteria
Solvability and necessity
Required information is supported by workspace or Web evidence and obtainable with the assigned tools. Cross-file and cross-modal dependencies contribute to the requested result. For L2 tasks, processing produces information needed for completion.
Answer leakage
Instructions, filenames, metadata, and supporting documents do not disclose target conclusions or intermediate identities that the agent is expected to infer.
Evaluation correctness
The reference output is supported by evidence and satisfies the requested content and format. Rubric checks correspond to explicit requirements, accommodate equivalent correct answers, and introduce no hidden requirements or unnecessary restrictions on solution paths.
Appendix
Table 4: Quality-control criteria for candidate tasks.
Figure 5: Workspace structure and media distributions. (a–b) File counts and mean modality composition by workspace type. (c–d) Audio and video file durations with shared logarithmic bins; dashed and dotted lines mark the median and 90th percentile. (e–f) Total media duration per task, including only tasks with that modality. Boxplots show medians, interquartile ranges (IQR), whiskers within 1.5 times the IQR of the box boundaries, and outliers.
Figure 6: Topic coverage and query characteristics. (a) Primary topic counts and percentages across 380 tasks, with each task counted once. (b) Query length distribution in words; dashed and dotted lines indicate the median and 90th percentile, respectively. (c) A word cloud highlighting frequently occurring terms in the task instructions.
Category
Tool Name
Description
Parameters
Workspace
list_workspace_files
List visible input files and files generated for the current task.
—
get_metadata
Retrieve file format, size, duration, resolution, and aspect ratio, as applicable.
path
read_text_file
Read a text file with an optional character limit.
path , max_chars *
write_text_file
Write text to the current task’s output directory.
path , content
Perception
read_image
Provide an image as visual input to the model.
path
read_video
Provide a video as multimodal input to the model.
path
Appendix
Table 7: Overview of tool interfaces across model configurations. An asterisk ( * ) marks optional parameters; a dash denotes no parameters.
Model
Workspace Type
Difficulty Levels
Overall
V+A+I+T
V+A+T
V+I+T
A+I+T
L1
L2
All 380 tasks
C
F
S
C
F
S
C
F
S
C
F
S
C
F
S
C
F
S
C
F
S
Native Omni-Modal Agents
Gemini 3.5 Flash
65.85
90.16
68.29
59.00
86.24
61.72
63.41
84.47
65.52
66.09
89.32
68.42
70.95
89.17
72.77
54.26
85.54
57.38
63.53
87.56
65.93
Gemini 3.1 Pro Preview
61.63
95.46
65.01
48.07
91.70
52.43
55.45
86.80
58.59
60.69
93.16
63.93
62.11
92.32
65.13
49.15
91.20
53.36
56.35
91.82
59.89
Gemini 3 Flash Preview
55.54
94.11
59.40
47.84
85.94
51.65
51.36
91.17
55.34
52.34
92.28
56.33
57.49
91.34
60.87
44.50
90.13
49.06
51.71
90.80
55.62
Appendix
Table 8: Detailed results on Omni-Workspace. We report Content (C), Format (F), and composite scores ( S=0.9C+0.1F ) on a 0–100 scale, averaged over tasks in each subset. Overall weights all 380 tasks equally. Bold indicates the highest score in each column.
Model / configuration
NSI ↑
MSU ↑
Gemini 3.5 Flash (native)
62.39
66.77
Gemini 3.1 Pro (native)
63.02
63.14
Qwen3.7-Plus (ASR + tools)
52.89
64.67
Qwen3.7-Plus (ASR-only)
51.37
53.78
(−1.52)
(−10.89)
Qwen3.6-Plus (ASR + tools)
47.75
63.09
Appendix
Table 9: Performance on audio-focused subsets. Bold marks the highest score in each subset. Parenthesized values show score changes relative to the corresponding VLM+ASR configuration.
Category
Definition
Example
Local execution errors
Multimodal perception
Incorrectly recognizing or transcribing task-relevant content in images, video, or audio.
Mistaking the person who speaks in a video for another visible person.
Search
Failing to find required information, or selecting a result about the wrong entity, source, time, or version.
Selecting the release date of a different film with the same title.
Tool invocation
Using incorrect parameters, paths, code, or operations that prevent a tool from performing the intended action correctly.
Processing media using an incorrect file path.
Task-state drift errors
Requirement omission
Omitting or replacing an explicit requirement, or incorrectly treating it as completed.
Analyzing only one recording when the task requires comparing two.
Appendix
Table 10: Error categories, definitions, and illustrative examples.
Figure 9: Error counts before and after OmniFocus. Percentages compare the sums of the three category counts in each group.
Metric
Qwen–Gemini
Qwen–Claude
Gemini–Claude
Atomic agreement
93.23%
95.49%
93.39%
Cohen’s κ
0.865
0.910
0.868
Positive agreement
93.47%
95.43%
93.67%
Negative agreement
92.97%
95.55%
93.09%
Spearman’s ρ
0.9897
0.9897
0.9959
Kendall’s τb
0.9477
0.9477
0.9739
Appendix
Table 11: Cross-judge agreement and model ranking correlations. Qwen, Gemini, and Claude denote Qwen3.7-Max, Gemini 3.5 Flash, and Claude Opus 4.8, respectively.
Figure 10: BookTube publication tracing. Task materials and reference workflow linking PCM decoding and audio reuse, cross-modal source attribution, and verification of publication, film-option, and producer connections, with findings recorded in a structured YAML report.
Figure 11: Cross-program provenance audit. Task request, input materials, recovered intermediate audio, and reference solution linking a Pictionary promotion to two people in a photograph from The Talk . The figure summarizes their cross-program roles, three event dates and the elapsed-day intervals between them, and an excerpt of the JSON report.