Long-horizon agents can execute continuously, but human attention remains intermittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention-coordination layer---Jarvis\footnote{Named after the fictional AI assistant in Iron Man.}---that mediates this interface and allocates human attention across one or more working agents. We introduce JarvisBench to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementation further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve.
Figures & tables
Figure 1: Overview of JarvisBench. Jarvis provides an always-on interface for user interaction while coordinating user attention with frozen agent runtimes through task events and attention requests.
Worker agent
Single-Agent
Multi-Agent
Baseline
w. Jarvis
Req.
Eff.
Baseline
w. Jarvis
Req.
Eff.
Claude Opus 5.0
58.1
77.7 +19.6
0.75
0.26
55.2
78.6 +23.4
0.70
0.33
Claude Opus 4.8
59.0
83.1 +24.1
0.70
0.34
52.6
80.8 +28.2
1.00
0.28
GPT-5.6-Sol
57.2
62.1 +4.9
0.55
0.09
51.6
–
–
–
GPT-5.5
55.1
67.1 +12.0
0.60
0.20
52.9
65.4 +12.5
0.50
0.25
DeepSeek V4-Pro
51.6
76.3 +24.7
0.85
0.29
53.3
69.4 +16.1
0.44
0.37
Table 1: Worker-agent comparison with the Jarvis LLM fixed to GPT-5.6-Sol. Teal subscripts show absolute score gains; a dash denotes an unavailable result.
Jarvis LLM
Single-Agent
Multi-Agent
Baseline
w. Jarvis
Req.
Eff.
Baseline
w. Jarvis
Req.
Eff.
GPT-5.6-Sol
59.0
83.1 +24.1
0.70
0.34
52.6
80.8 +28.2
1.00
0.28
Claude Opus 4.8
70.4 +11.4
0.50
0.23
76.1 +23.5
0.70
0.34
DeepSeek V4-Pro
72.3 +13.3
0.60
0.22
65.0 +12.4
0.30
0.41
GPT-OSS-120B
71.3 +12.3
0.85
0.14
68.6 +16.0
0.80
0.20
Table 2: Jarvis-LLM comparison with Claude Opus 4.8 fixed as the worker agent. Teal subscripts show absolute score gains.
Jarvis LLM
Single-Agent
Multi-Agent
Overall
Latency (s) ↓
General
Follow-up
General
Follow-up
GPT-5.6-Sol
97.5
97.5
95.0
92.5
96.3
1.8
Claude Opus 4.8
87.5
91.3
95.0
92.5
90.8
2.1
DeepSeek V4-Pro
83.8
93.8
87.5
72.5
85.8
2.0
Qwen235B
82.5
83.8
90.0
72.5
82.5
1.6
GPT-OSS-120B
80.0
87.5
85.0
87.5
84.6
1.3
Table 3: User-interaction performance of different Jarvis LLMs. Scores use a 0–100 scale. Overall covers 20 single-agent and 10 multi-agent trajectories. Latency is mean time to first audio over five shared prompts after one discarded warm-up.