Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models

Yuxiang Wang; Shengbo Cai; Zhixian Zhao; Yingda Shen; Yuancheng Wang; Xueyao Zhang; Liqiang Zhang; Teddy Sun

cs.SDOct 16, 2026

Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models

Authors: Yuxiang Wang, Shengbo Cai, Zhixian Zhao, Yingda Shen, Yuancheng Wang, Xueyao Zhang, Liqiang Zhang, Teddy Sun,

Organizations: The Chinese University of Hong Kong, Shenzhen · Tencent Hunyuan · Amphion Technology Co., Ltd.

Abstract

Interactive models increasingly hand complex reasoning and tool use to a slow brain in the background, leaving a realtime front model free to sustain full-duplex dialogue. Yet the user may refine, cancel or replace a request while its task runs, so a plan or result that made sense a moment ago goes stale. Each new utterance must therefore be read against the work in flight, but the front model follows only the conversation and the back end sees only its own task. Coordination deserves a brain of its own, one that sees both. We introduce Jarvis, a coordination harness where models judge and code enforces. Its typed decision model, JEV, reads each turn against the live tasks and picks among operations built from a versioned task record. Code checks every choice against the current state before applying it, so no decision can act on a task version the user has replaced. Jarvis also rechecks plans before they execute and results before they are spoken, and counts an answer as delivered only once the user has heard it in full. We build two instances of this design. Jarvis-Audio runs unchanged on three full-duplex speech models. Jarvis-Omni adds vision and rechecks the evidence behind an answer before and during playback, since a scene can change while nobody speaks. With the front model and back end held fixed, replacing Qwen Audio Agent with Jarvis raises tau-Voice task success more than sixfold, from 29 to 192 of 278 tasks, and the gains hold on every benchmark and front model. Against published systems, Jarvis-Audio reaches 69% Pass1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime. These results suggest that how the two brains are coordinated matters as much as how strong they are.

Figures & tables

Figure 1: Benchmark results for Jarvis (blue) and other systems (gray). Top: the five highest published entries (Tables 3 and 4 ). Bottom: runtimes on one DeepSeek-V4-Flash back end (Table 2 ) and controllers with the same options (Table 6 ). ∗ Proprietary reasoning back end.

Panel of Figure 2 — Figure 2: A recorded five-minute session with Jarvis-Audio. The user plans a game night while up to four tasks remain in flight. Each numbered utterance becomes one or more typed operations on the Task Ledger (tags below), with task-control operations bound to live task versions. Each bar is one version of a task. The game goes through four versions, the Python question gets a follow-up, a one-night budget of $60 leaves the usual$ 40 in memory untouched, and the cancellation removes the welcome sign and nothing else.

Failure mode	What the user experiences	Mitigated by	Evidence
Stale delivery	The first draft of a game is presented after the user has already asked for a different theme.	Task Ledger versions	Table 5
Misbinding	“Make it sixty dollars” changes the game instead of the snack plan.	Decision Brain over version-bound options	Table 6
Ghost execution	A plan formed before the user corrected the order number is executed anyway.	Freshness check on the plan	Table 5
Lost delivery	The user cuts an answer off halfway, yet the task is recorded as delivered.	Playback receipts and epochs	Table 5
Memory drift	“Only sixty tonight” overwrites the usual budget of forty.	Gated memory writes	Table 7
Expired evidence	Advice to push forward arrives after the teammate on screen has died.	Belief State validity windows	§ 5.6

Table 1: Failure modes of a voice agent whose background work runs while the conversation continues, with the mechanism in Jarvis that addresses each one and the evidence used to evaluate it.

Panel of Figure 3 — Figure 3: Overview of Jarvis. The fast brain keeps the conversation going and decides when to hand a turn over. The slow brain interprets the turn with JEV, records the resulting work in the Task Ledger, runs it asynchronously and returns the answer through the fast brain’s own voice once the result is still valid. The dashed module is used by Jarvis-Omni only.

Panel of Figure 4 — Figure 4: The Native Gate. (a) On each one-second audio unit the gate reads a three-way policy from the native tokens of the front model. It keeps listening through leading silence and through an unfinished request, and fires DELEGATE once the request is complete and the probability passes 0.8. (b) The LoRA adapter is active only in the routing pass. Speech is produced by the frozen weights with a separate key-value cache.

Panel of Figure 5 — Figure 5: A decision request to JEV, simplified from the runtime payload. The runtime writes the options from the live Task Ledger, so the answer is a distribution over operations that the runtime can execute.

Panel of Figure 6 — Figure 6: Life of a task in the Task Ledger. A refinement supersedes the live version and starts a new one, and a result that arrives for a superseded or cancelled version is discarded. COMPLETED becomes DELIVERED only when a playback receipt shows that the whole answer was heard in the current generation epoch. Stopping the voice opens a new epoch but leaves the task itself untouched.

Panel of Figure 7 — Figure 7: Evidence that expires. Claims read from the screen carry a validity window and are invalidated by events such as a death or a camera change. Advice is checked against the Belief State when it is ready, so a recommendation that rests on an expired claim is dropped and planned again from the new state. A question about earlier advice is answered from a historical snapshot.

		$\tau$ -Voice (solved)				FDB-v3		FDB-v2
Front model	Harness	Airline	Retail	Telecom	Total	Exact	Semantic	Task score
Venus-Audio	Native harness	17	6	0	23	45	35	3.09
	QAA	23	6	0	29	41	59	2.57
	Jarvis	35	77	80	192	51	63	3.36
MiniCPM-o 4.5	QAA	23	6	0	29	40	56	2.98
MiniCPM-o 4.5	Jarvis	23	6	9	38	50	63	3.28

Table 2: Harnesses compared on the same front model, admission path and back end.

\tau

-Voice reports solved tasks, FDB-v3 reports passes out of 100 under exact and semantic matching, and FDB-v2 reports the mean task score over 300 task and pace pairs. All rows share one judge. † Released runtime with a DeepSeek-V4-Flash worker in place of its Codex back end.

Figure 8: Jarvis against QAA with the same front model, admission path and back end. (a)

\tau

-Voice tasks solved out of 278. (b) FDB-v2 task score for every combination of front model and category. Jarvis improves on QAA in all nine.

System	Retail $\uparrow$	Airline $\uparrow$	Telecom $\uparrow$	Macro avg. $\uparrow$
Realtime speech models
GPT-Realtime 1.0	35.96	36.00	19.30	30.42
GPT-Realtime 1.5	44.70	40.00	21.10	35.27
GPT-Realtime 2	47.37	58.00	21.93	42.43
Gemini 2.5 Flash Live	29.80	30.00	17.50	25.77
Gemini 3.1 Flash Live (thinking high)	45.61	64.00	21.93	43.85

Table 4: Task success (%) on

\tau

-Voice (regular tasks). Bold marks the best value among systems without a proprietary reasoning back end.

Figure 9: Timely control. (a) Correct decisions that clear each controller’s threshold within a given deadline. (b) Correct decisions when the conservative fallback is counted. (c) Share of cancellations and refinements that stop the old worker before it finishes, as a function of when it would finish. Dotted lines mark the 1 s window and the 3.5 s deadline of the runtime.

Panel of Figure 10 — Figure 10: Jarvis in use. (a) to (c) run Jarvis-Audio and (d) and (e) run Jarvis-Omni.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

	Jarvis-Audio	Jarvis-Omni
Front model	Realtime-Venus-Audio, 9B ( Ant Group, 2026 )	Venus-Omni ( Ant Group, 2026 )
Native Gate	Causal audio prefixes, three actions	Causal audio and screen features with dialogue context, adds a SCENE action for proactive turns
Decision Brain	JEV, text	Jev-Omni, 12B, text and images
Perception state	ASR observer	ASR observer, Evidence Timeline and Belief State
Memory	Profile and task records	Adds in-match events and game knowledge
Shared implementation

Appendix

Table 9: The two instances of Jarvis. Rows in the shared block use the same implementation in both.

Decision	Options written by the runtime	Type	Threshold
Turns and tasks
Turn route	answer fast, memory, worker, clarify, control, ignore, wait	Choice	0.75
Early preparation on partial speech	wait, prefetch memory, prefetch worker	Choice	0.75
Operation on an admitted turn	answer, memory, cancel, refine, status or resume on each live task version	Choice	0.80
Task control	status, refine, replace, cancel or resume on each task version, stop speaking	Choice	0.80
Batch interpretation	new work, refine, cancel, status, overview, memory, save, stop speech, ignore	Choice	0.78

Appendix

Table 10: Decisions that Jarvis hands to JEV. All of them use the typed interface, and none returns free text. The threshold is the confidence below which the runtime falls back to a safe default or asks for a second review.

Control head	Correct $\uparrow$	Missed $\downarrow$	Early
Frozen Venus-Omni	30	66	0
Previous adapter	67	22	1
Contextual GRPO adapter	75	1	13

Appendix

Table 11: The scene-aware gate of Jarvis-Omni on the fixed test set of 96 episodes, with 30 listening, 33 direct and 33 delegated targets. Early means acting before the reference time.

Layer	Size and split	Content
Authored scenarios	264, 144 train, 40 dev, 80 held out	Questions, small talk, tool needs, memory reads, task changes, background speech, game talk and game actions, in Chinese and English
Game round	154, 82 train, 16 dev, 56 held out	Discussing a position, explaining it, authorizing an action and taking control back
Continuity	38 families, 76 episodes	Names, spelled identifiers, addresses, quantities, budgets, edits, cancellations and confirmations, clean and over a telephone channel
Telephony	246 episodes, 166 train, 80 dev	Quiet line and colored noise at 15 dB and 9 dB after 8 kHz resampling and $\mu$ -law coding
Expanded telephony	510 episodes, 382 train, 128 dev	174 source scenarios under three channel conditions
Conversational requests	24 families, 144 episodes	Stories, jokes, explanations, encouragement, side remarks and requests to wait, in three audio variants and two contexts

Appendix

Table 12: Data layers of the Native Gate of Jarvis-Audio. Later layers reuse earlier recordings under new channel conditions or contexts.

Figure 11: Final training stage of the Native Gate. (a) Correct first admissions on the development sets, evaluated every 100 steps. (b) Confusion matrices on the new development set before and after the stage. Rows are targets and columns are predictions.

Front model	Harness	Take-turn $\uparrow$	Tool F1 $\uparrow$	Arg. acc. $\uparrow$	Resp. qual. $\uparrow$	Completion (s)
Venus-Audio	QAA	68	0.794	0.657	0.529	14.3 (14.2)
Venus-Audio	Jarvis	97	0.923	0.740	0.753	15.0 (12.9)
MiniCPM-o 4.5	QAA	92	0.788	0.638	0.511	13.1 (13.1)
MiniCPM-o 4.5	Jarvis	99	0.881	0.720	0.626	14.0 (16.2)
VoiceChat-11B	QAA	96	0.772	0.470	0.396	22.6 (15.7)
VoiceChat-11B	Jarvis	97	0.758	0.583	0.443	17.2 (17.4)

Appendix

Table 14: FDB-v3 quality and completion time with the same front model. Take-turn counts recordings with an audible response. Completion time is the official task-completion latency in seconds, mean with the median in parentheses.

	Domain				Difficulty			Disfluency
System	Ecom.	Fin.	Hous.	Travel	Easy	Med.	Hard	Filler	Pause	Hesit.	False st.	Self-corr.
GPT-Realtime	55.2	96.0	30.8	60.0	75.0	58.8	43.3	62.1	55.6	70.0	66.7	58.8
Gemini Live 2.5	48.3	76.0	23.1	50.0	66.7	50.0	26.7	62.1	44.4	60.0	41.7	47.1
Gemini Live 3.1	44.8	92.0	26.9	55.0	69.4	58.8	30.0	58.6	50.0	60.0	58.3	35.3
Grok	41.4	76.0	11.5	45.0	58.3	47.1	20.0	48.3	33.3	50.0	58.3	29.4
Ultravox	34.5	68.0	19.2	45.0	55.6	38.2	26.7	41.4	33.3	50.0	25.0	35.3

Appendix

Table 15: FDB-v3 Pass@1 (%) by domain, difficulty and type of disfluency. Bold marks the best value in each column.

Figure 12: Where the time goes. (a) The fast path of Jarvis-Omni in recorded CS2 and League of Legends sessions, with stage medians and the end-to-end median from the end of the user’s speech to the first voiced word. (b) Stages of the slow brain of Jarvis-Audio over 42 benchmark sessions.

LLM Agent Harnesses

LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.

See all 357 papers

Aug 14, 2026

JarvisBench: Always-on Intelligence Between Humans and Agents

Long-horizon agents can execute continuously, but human attention remains intermittent and scarce. This creates a bidirectional coordination problem: users may need immediate access to an agent while work continues in the background, whereas agents may encounter consequential decisions that require user judgment after the user has stopped monitoring execution. We posit an always-on attention-coordination layer---Jarvis\footnote{Named after the fictional AI assistant in Iron Man.}---that mediates this interface and allocates human attention across one or more working agents. We introduce JarvisBench to evaluate both directions of this coordination: whether an intermediary can accurately and promptly answer user-initiated questions about ongoing work, and whether it can recognize when an agent requires user judgment, solicit that judgment at the right moment, and route it back to improve task outcomes. JarvisBench contains 45 agentic task instances: 20 single-agent tasks and 25 workstreams organized into 10 multi-agent projects. The tasks span 19 domains and were selected and adapted from more than 2,000 public candidates. Crucially, the need for user attention arises naturally during execution rather than from an obvious omission in the initial prompt. JarvisBench is designed to integrate with arbitrary agent runtimes without modifying their underlying execution loops. Our reference implementation further provides a full-duplex speech interface, allowing users to reach Jarvis naturally while timely attention coordination supports agents working in the background. By separating agent execution from attention coordination, JarvisBench provides a stable evaluation target as agent capabilities continue to improve.

Chen Chen, Zhehuai Chen

NVIDIA

Human-AI Collaboration Human-Computer Interaction AI Agent Benchmarks Long-Horizon Agent Evaluation

Ask PDF

Oct 5, 2026

Jarvis: A Proactive Speech Agent for Multi-Party Conversations

Speech agents are reactive and dyadic: they speak when spoken to, and to one person at a time. We ask what it takes for a speech agent to instead take part in a conversation among several people and speak up only when it can help. We introduce Jarvis, a real-time proactive speech agent that audibly participates in multi-party human conversations. Grounded in a document shared beforehand, Jarvis follows the discussion and intervenes when the group misses or misstates a fact and does not correct itself within a few turns. We make three contributions: a problem setting based on epistemic breakdowns that makes proactive intervention measurable, realized as CHI-180-proactive, a synthetic multi-party dataset seeded with known gaps, errors, and self-corrections; a proactive backbone that harnesses a small, open-weight model with deterministic checks and grounds every claim in a source sentence; and interaction techniques for taking the floor in live speech and showing the cited evidence on screen. On CHI-180-proactive, Jarvis is correct on most events it addresses and stays silent 97% of the time when the group resolves an issue itself. A live study with 23 participants confirms these trends with real-time interventions.

Seunghyun Oh, Hirotaka Hiraki, Shuyue Stella Li +2

Paul G. Allen School of Computer Science and Engineering, University of Washington, WA, USA · Graduate School of Interdisciplinary Information Studies, The University of Tokyo, Bunkyo, Tokyo, Japan

Conversational Agents Proactive Agents Proactive Assistance Spoken Dialogue Systems Automated Fact-Checking

Ask PDF

Sep 21, 2026

Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction

Real-time voice assistants must reason over evolving requests, execute actions, and follow conversational rules. Qwen-Audio-3.1-Realtime brings these requirements together through Think, Act, and Speak and Coordinate. Think combines Core-Cocktail supervised fine-tuning with Multimodality and Multi-Teacher On-Policy Distillation (M

^{2}

-OPD) to transfer language capabilities and develop native audio skills. Act uses self-evolving executable environments and multi-granularity rollouts for Group Relative Policy Optimization (GRPO), teaching the model to use tools, interpret feedback, and complete tasks. Speak and Coordinate aligns whether, when, and how the assistant speaks or acts. We evaluate audio reasoning, multilingual understanding, tool use, conversational behavior, full-duplex interaction, and safety. Compared with Qwen-Audio-3.0-Realtime, 3.1 raises overall task success from 78.4% to 82.0% on our half-duplex speech-to-text adaptation of

τ

-Voice. On speech-to-speech Full-Duplex-Bench v1.5, the response rate to background speech falls from 73.0% to 13.0%. We also present a separate Voice Harness prototype, using Qwen-Audio-3.0-Realtime as its foreground, that extends spoken interaction to persistent tasks through foreground--background coordination and memory.

Lujia Bao, Qian Chen, Luyao Cheng +15

Alibaba Token Foundry, Alibaba Group

Tool-Augmented Language Model Agents Speech Foundation Models Audio Reasoning LLM Agents Conversational Agents Full-Duplex Spoken Dialogue Systems

Ask PDF

Abstract

LLM Agent Harnesses

Explore similar work

JarvisBench: Always-on Intelligence Between Humans and Agents

Jarvis: A Proactive Speech Agent for Multi-Party Conversations

Qwen-Audio-3.1-Realtime: Towards Reliable Agentic Voice Interaction