JarvisBench: Always-on Intelligence Between Humans and Agents

Chen Chen; Zhehuai Chen

cs.AIAug 14, 2026

JarvisBench: Always-on Intelligence Between Humans and Agents

Authors: Chen Chen, Zhehuai Chen

Organizations: NVIDIA

Abstract

Long-horizon agents can execute continuously, but human attention remains intermittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention-coordination layer---Jarvis\footnote{Named after the fictional AI assistant in Iron Man.}---that mediates this interface and allocates human attention across one or more working agents. We introduce JarvisBench to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementation further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve.

Figures & tables

Figure 1: Overview of JarvisBench. Jarvis provides an always-on interface for user interaction while coordinating user attention with frozen agent runtimes through task events and attention requests.

Worker agent	Single-Agent				Multi-Agent
Worker agent	Baseline	w. Jarvis	Req.	Eff.	Baseline	w. Jarvis	Req.	Eff.
Claude Opus 5.0	58.1	77.7 +19.6	0.75	0.26	55.2	78.6 +23.4	0.70	0.33
Claude Opus 4.8	59.0	83.1 +24.1	0.70	0.34	52.6	80.8 +28.2	1.00	0.28
GPT-5.6-Sol	57.2	62.1 +4.9	0.55	0.09	51.6	–	–	–
GPT-5.5	55.1	67.1 +12.0	0.60	0.20	52.9	65.4 +12.5	0.50	0.25
DeepSeek V4-Pro	51.6	76.3 +24.7	0.85	0.29	53.3	69.4 +16.1	0.44	0.37

Table 1: Worker-agent comparison with the Jarvis LLM fixed to GPT-5.6-Sol. Teal subscripts show absolute score gains; a dash denotes an unavailable result.

Jarvis LLM	Single-Agent				Multi-Agent
Jarvis LLM	Baseline	w. Jarvis	Req.	Eff.	Baseline	w. Jarvis	Req.	Eff.
GPT-5.6-Sol	59.0	83.1 +24.1	0.70	0.34	52.6	80.8 +28.2	1.00	0.28
Claude Opus 4.8		70.4 +11.4	0.50	0.23		76.1 +23.5	0.70	0.34
DeepSeek V4-Pro		72.3 +13.3	0.60	0.22		65.0 +12.4	0.30	0.41
GPT-OSS-120B		71.3 +12.3	0.85	0.14		68.6 +16.0	0.80	0.20

Table 2: Jarvis-LLM comparison with Claude Opus 4.8 fixed as the worker agent. Teal subscripts show absolute score gains.

Jarvis LLM	Single-Agent		Multi-Agent		Overall	Latency (s) $\downarrow$
Jarvis LLM	General	Follow-up	General	Follow-up	Overall	Latency (s) $\downarrow$
GPT-5.6-Sol	97.5	97.5	95.0	92.5	96.3	1.8
Claude Opus 4.8	87.5	91.3	95.0	92.5	90.8	2.1
DeepSeek V4-Pro	83.8	93.8	87.5	72.5	85.8	2.0
Qwen235B	82.5	83.8	90.0	72.5	82.5	1.6
GPT-OSS-120B	80.0	87.5	85.0	87.5	84.6	1.3

Table 3: User-interaction performance of different Jarvis LLMs. Scores use a 0–100 scale. Overall covers 20 single-agent and 10 multi-agent trajectories. Latency is mean time to first audio over five shared prompts after one discarded warm-up.

Explore similar work

Jul 18, 2026

Just A Rather Very Intelligent Spoken Agent

Long-horizon AI agents are becoming increasingly capable, yet their interaction with users remains surprisingly thin. In most workflows, users give an initial instruction, receive only selective textual updates, and lose a clear sense of what the agent is doing or when to step in. This leaves a missing part in the current agent ecosystem: an always-on Jarvis-style mediator that keeps the agent continuously reachable to the user. Such a mediator should support real-time spoken interaction with the user, answer questions without interrupting the worker, proactively report progress or confusion, and inject user guidance back into the agent's execution when useful. In this work, we introduce JarvisBench, a benchmark for measuring the dual value of mediation in long-horizon agent workflows. JarvisBench contains two complementary tracks: an agent-collaboration track that measures whether mediation improves downstream task completion, and a user-interaction track that measures whether mediation makes ongoing execution more understandable, responsive, and accessible to users. We instantiate the benchmark with a modular reference Jarvis prototype and evaluate it on 34 text-only WildClaw tasks executed in OpenClaw. Preliminary results with GPT-5.5, Claude Opus 4.7, Gemini-based, and GPT-based worker agents suggest that Jarvis-style mediation can provide trace-grounded responses to user questions and improve task performance when sparse user guidance is injected at appropriate moments. The results also show that effectiveness depends strongly on the mediator's LLM brain, highlighting both the promise of this missing middle layer and the need for broader community effort. Demo page https://cchen1436.github.io/jarvis

Chen Chen, Zhehuai Chen

NVIDIA

Long-Horizon Agent Evaluation Human-AI Interaction AI Agent Benchmarks Conversational Agents

Ask PDF

May 14, 2026

$π$ -Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

The rise of personal assistant agents, e.g., OpenClaw, highlights the growing potential of large language models to support users across everyday life and work. A core challenge in these settings is proactive assistance, since users often begin with underspecified requests and leave important needs, constraints, or preferences unstated. However, existing benchmarks rarely evaluate whether agents can identify and act on such hidden intents before they are explicitly stated, especially in sustained multi-turn interactions where user needs emerge gradually. To address this gap, we introduce

π

-Bench, a benchmark for proactive assistance comprising 100 multi-turn tasks across 5 domain-specific user personas. By incorporating hidden user intents, inter-task dependencies, and cross-session continuity,

π

-Bench evaluates agents' ability to anticipate and address user needs over extended interactions, jointly measuring proactivity and task completion in long-horizon trajectories that better reflect real-world use. Experiments show (1) proactive assistance remains challenging, (2) a clear distinction between task completion and proactivity, and (3) the value of prior interaction for proactive intent resolution in later tasks.

Haoran Zhang, Luxin Xu, Zhilin Wang +11

Shanghai Jiao Tong University · Shanghai AI Laboratory · Fudan University +7

Long-Horizon Agent Evaluation AI Agent Benchmarks Proactive Assistance Personal AI Agents

Ask PDF

May 8, 2026

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

Agent systems often decompose a task across multiple roles, but these roles are typically specified by prompts rather than enforced by access controls. Without enforcement, a team pass rate can mask whether agents actually coordinated or whether one role effectively did another role's work. We present TeamBench, a benchmark with 851 task templates and 931 seeded instances for evaluating agent coordination under operating system-enforced role separation. TeamBench separates specification access, workspace editing, and final certification across Planner, Executor, and Verifier roles, so that no role can read the full requirements, modify the workspace, and certify the final answer. Prompt-only and sandbox-enforced teams reach statistically indistinguishable pass rates, but prompt-only runs produce 3.6 times more cases where the verifier attempts to edit the executor's code. Verifiers approve 49% of submissions that fail the deterministic grader, and removing the verifier improves mean partial score in the ablation. Team value is also conditional. Teams benefit when single agents struggle, but hurt when single agents already perform well. A 40-session human study under the same role separation shows that our benchmark exposes interaction patterns that pass rate misses. Solo participants work through the task directly, human participants paired with agents often collapse into quick approval, and human teams spend more effort coordinating missing information across roles.

Yubin Kim, Chanwoo Park, Taehan Kim +9

MIT · Google Research · Independent Researcher +1

Multi-Agent Coordination LLM Agent Evaluation AI Agent Security AI Agent Benchmarks Task Decomposition Tool Access Control for LLM Agents+1

Ask PDF

Abstract

Explore similar work

Just A Rather Very Intelligent Spoken Agent

πππ-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows

TeamBench: Evaluating Agent Coordination under Enforced Role Separation

$π$ -Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows