cs.SDOct 16, 2026
SaveJarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models
Organizations: The Chinese University of Hong Kong, Shenzhen · Tencent Hunyuan · Amphion Technology Co., Ltd.
Abstract
Interactive models increasingly hand complex reasoning and tool use to a slow brain in the background, leaving a realtime front model free to sustain full-duplex dialogue. Yet the user may refine, cancel or replace a request while its task runs, so a plan or result that made sense a moment ago goes stale. Each new utterance must therefore be read against the work in flight, but the front model follows only the conversation and the back end sees only its own task. Coordination deserves a brain of its own, one that sees both. We introduce Jarvis, a coordination harness where models judge and code enforces. Its typed decision model, JEV, reads each turn against the live tasks and picks among operations built from a versioned task record. Code checks every choice against the current state before applying it, so no decision can act on a task version the user has replaced. Jarvis also rechecks plans before they execute and results before they are spoken, and counts an answer as delivered only once the user has heard it in full. We build two instances of this design. Jarvis-Audio runs unchanged on three full-duplex speech models. Jarvis-Omni adds vision and rechecks the evidence behind an answer before and during playback, since a scene can change while nobody speaks. With the front model and back end held fixed, replacing Qwen Audio Agent with Jarvis raises tau-Voice task success more than sixfold, from 29 to 192 of 278 tasks, and the gains hold on every benchmark and front model. Against published systems, Jarvis-Audio reaches 69% Pass1 on Full-Duplex-Bench v3 with an open 9B front model, versus 60% for GPT-Realtime. These results suggest that how the two brains are coordinated matters as much as how strong they are.
Figures & tables
Figure 1: Benchmark results for Jarvis (blue) and other systems (gray). Top: the five highest published entries (Tables 3 and 4 ). Bottom: runtimes on one DeepSeek-V4-Flash back end (Table 2 ) and controllers with the same options (Table 6 ). ∗ Proprietary reasoning back end.
Figure 2: A recorded five-minute session with Jarvis-Audio. The user plans a game night while up to four tasks remain in flight. Each numbered utterance becomes one or more typed operations on the Task Ledger (tags below), with task-control operations bound to live task versions. Each bar is one version of a task. The game goes through four versions, the Python question gets a follow-up, a one-night budget of 40 in memory untouched, and the cancellation removes the welcome sign and nothing else.
| Failure mode | What the user experiences | Mitigated by | Evidence |
| Stale delivery | The first draft of a game is presented after the user has already asked for a different theme. | Task Ledger versions | Table 5 |
| Misbinding | “Make it sixty dollars” changes the game instead of the snack plan. | Decision Brain over version-bound options | Table 6 |
| Ghost execution | A plan formed before the user corrected the order number is executed anyway. | Freshness check on the plan | Table 5 |
| Lost delivery | The user cuts an answer off halfway, yet the task is recorded as delivered. | Playback receipts and epochs | Table 5 |
| Memory drift | “Only sixty tonight” overwrites the usual budget of forty. | Gated memory writes | Table 7 |
| Expired evidence | Advice to push forward arrives after the teammate on screen has died. | Belief State validity windows | § 5.6 |
Table 1: Failure modes of a voice agent whose background work runs while the conversation continues, with the mechanism in Jarvis that addresses each one and the evidence used to evaluate it.
Figure 3: Overview of Jarvis. The fast brain keeps the conversation going and decides when to hand a turn over. The slow brain interprets the turn with JEV, records the resulting work in the Task Ledger, runs it asynchronously and returns the answer through the fast brain’s own voice once the result is still valid. The dashed module is used by Jarvis-Omni only.
Figure 4: The Native Gate. (a) On each one-second audio unit the gate reads a three-way policy from the native tokens of the front model. It keeps listening through leading silence and through an unfinished request, and fires DELEGATE once the request is complete and the probability passes 0.8. (b) The LoRA adapter is active only in the routing pass. Speech is produced by the frozen weights with a separate key-value cache.
Figure 5: A decision request to JEV, simplified from the runtime payload. The runtime writes the options from the live Task Ledger, so the answer is a distribution over operations that the runtime can execute.
Figure 6: Life of a task in the Task Ledger. A refinement supersedes the live version and starts a new one, and a result that arrives for a superseded or cancelled version is discarded. COMPLETED becomes DELIVERED only when a playback receipt shows that the whole answer was heard in the current generation epoch. Stopping the voice opens a new epoch but leaves the task itself untouched.
Figure 7: Evidence that expires. Claims read from the screen carry a validity window and are invalidated by events such as a death or a camera change. Advice is checked against the Belief State when it is ready, so a recommendation that rests on an expired claim is dropped and planned again from the new state. A question about earlier advice is answered from a historical snapshot.
| -Voice (solved) | FDB-v3 | FDB-v2 | ||||||
| Front model | Harness | Airline | Retail | Telecom | Total | Exact | Semantic | Task score |
| Venus-Audio | Native harness | 17 | 6 | 0 | 23 | 45 | 35 | 3.09 |
| QAA | 23 | 6 | 0 | 29 | 41 | 59 | 2.57 | |
| Jarvis | 35 | 77 | 80 | 192 | 51 | 63 | 3.36 | |
| MiniCPM-o 4.5 | QAA | 23 | 6 | 0 | 29 | 40 | 56 | 2.98 |
| Jarvis | 23 | 6 | 9 | 38 | 50 | 63 | 3.28 | |
Table 2: Harnesses compared on the same front model, admission path and back end. -Voice reports solved tasks, FDB-v3 reports passes out of 100 under exact and semantic matching, and FDB-v2 reports the mean task score over 300 task and pace pairs. All rows share one judge. † Released runtime with a DeepSeek-V4-Flash worker in place of its Codex back end.
Figure 8: Jarvis against QAA with the same front model, admission path and back end. (a) -Voice tasks solved out of 278. (b) FDB-v2 task score for every combination of front model and category. Jarvis improves on QAA in all nine.
| System | Retail | Airline | Telecom | Macro avg. |
| Realtime speech models | ||||
| GPT-Realtime 1.0 | 35.96 | 36.00 | 19.30 | 30.42 |
| GPT-Realtime 1.5 | 44.70 | 40.00 | 21.10 | 35.27 |
| GPT-Realtime 2 | 47.37 | 58.00 | 21.93 | 42.43 |
| Gemini 2.5 Flash Live | 29.80 | 30.00 | 17.50 | 25.77 |
| Gemini 3.1 Flash Live (thinking high) | 45.61 | 64.00 | 21.93 | 43.85 |
Table 4: Task success (%) on -Voice (regular tasks). Bold marks the best value among systems without a proprietary reasoning back end.
Figure 9: Timely control. (a) Correct decisions that clear each controller’s threshold within a given deadline. (b) Correct decisions when the conservative fallback is counted. (c) Share of cancellations and refinements that stop the old worker before it finishes, as a function of when it would finish. Dotted lines mark the 1 s window and the 3.5 s deadline of the runtime.
Figure 10: Jarvis in use. (a) to (c) run Jarvis-Audio and (d) and (e) run Jarvis-Omni.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Jarvis-Audio | Jarvis-Omni | |
| Front model | Realtime-Venus-Audio, 9B ( Ant Group, 2026 ) | Venus-Omni ( Ant Group, 2026 ) |
| Native Gate | Causal audio prefixes, three actions | Causal audio and screen features with dialogue context, adds a SCENE action for proactive turns |
| Decision Brain | JEV, text | Jev-Omni, 12B, text and images |
| Perception state | ASR observer | ASR observer, Evidence Timeline and Belief State |
| Memory | Profile and task records | Adds in-match events and game knowledge |
| Shared implementation | ||
Table 9: The two instances of Jarvis. Rows in the shared block use the same implementation in both.
| Decision | Options written by the runtime | Type | Threshold |
| Turns and tasks | |||
| Turn route | answer fast, memory, worker, clarify, control, ignore, wait | Choice | 0.75 |
| Early preparation on partial speech | wait, prefetch memory, prefetch worker | Choice | 0.75 |
| Operation on an admitted turn | answer, memory, cancel, refine, status or resume on each live task version | Choice | 0.80 |
| Task control | status, refine, replace, cancel or resume on each task version, stop speaking | Choice | 0.80 |
| Batch interpretation | new work, refine, cancel, status, overview, memory, save, stop speech, ignore | Choice | 0.78 |
Table 10: Decisions that Jarvis hands to JEV. All of them use the typed interface, and none returns free text. The threshold is the confidence below which the runtime falls back to a safe default or asks for a second review.
| Control head | Correct | Missed | Early |
| Frozen Venus-Omni | 30 | 66 | 0 |
| Previous adapter | 67 | 22 | 1 |
| Contextual GRPO adapter | 75 | 1 | 13 |
Table 11: The scene-aware gate of Jarvis-Omni on the fixed test set of 96 episodes, with 30 listening, 33 direct and 33 delegated targets. Early means acting before the reference time.
| Layer | Size and split | Content |
| Authored scenarios | 264, 144 train, 40 dev, 80 held out | Questions, small talk, tool needs, memory reads, task changes, background speech, game talk and game actions, in Chinese and English |
| Game round | 154, 82 train, 16 dev, 56 held out | Discussing a position, explaining it, authorizing an action and taking control back |
| Continuity | 38 families, 76 episodes | Names, spelled identifiers, addresses, quantities, budgets, edits, cancellations and confirmations, clean and over a telephone channel |
| Telephony | 246 episodes, 166 train, 80 dev | Quiet line and colored noise at 15 dB and 9 dB after 8 kHz resampling and -law coding |
| Expanded telephony | 510 episodes, 382 train, 128 dev | 174 source scenarios under three channel conditions |
| Conversational requests | 24 families, 144 episodes | Stories, jokes, explanations, encouragement, side remarks and requests to wait, in three audio variants and two contexts |
Table 12: Data layers of the Native Gate of Jarvis-Audio. Later layers reuse earlier recordings under new channel conditions or contexts.
Figure 11: Final training stage of the Native Gate. (a) Correct first admissions on the development sets, evaluated every 100 steps. (b) Confusion matrices on the new development set before and after the stage. Rows are targets and columns are predictions.
| Front model | Harness | Take-turn | Tool F1 | Arg. acc. | Resp. qual. | Completion (s) |
| Venus-Audio | QAA | 68 | 0.794 | 0.657 | 0.529 | 14.3 (14.2) |
| Jarvis | 97 | 0.923 | 0.740 | 0.753 | 15.0 (12.9) | |
| MiniCPM-o 4.5 | QAA | 92 | 0.788 | 0.638 | 0.511 | 13.1 (13.1) |
| Jarvis | 99 | 0.881 | 0.720 | 0.626 | 14.0 (16.2) | |
| VoiceChat-11B | QAA | 96 | 0.772 | 0.470 | 0.396 | 22.6 (15.7) |
| Jarvis | 97 | 0.758 | 0.583 | 0.443 | 17.2 (17.4) |
Table 14: FDB-v3 quality and completion time with the same front model. Take-turn counts recordings with an audible response. Completion time is the official task-completion latency in seconds, mean with the median in parentheses.
| Domain | Difficulty | Disfluency | ||||||||||
| System | Ecom. | Fin. | Hous. | Travel | Easy | Med. | Hard | Filler | Pause | Hesit. | False st. | Self-corr. |
| GPT-Realtime | 55.2 | 96.0 | 30.8 | 60.0 | 75.0 | 58.8 | 43.3 | 62.1 | 55.6 | 70.0 | 66.7 | 58.8 |
| Gemini Live 2.5 | 48.3 | 76.0 | 23.1 | 50.0 | 66.7 | 50.0 | 26.7 | 62.1 | 44.4 | 60.0 | 41.7 | 47.1 |
| Gemini Live 3.1 | 44.8 | 92.0 | 26.9 | 55.0 | 69.4 | 58.8 | 30.0 | 58.6 | 50.0 | 60.0 | 58.3 | 35.3 |
| Grok | 41.4 | 76.0 | 11.5 | 45.0 | 58.3 | 47.1 | 20.0 | 48.3 | 33.3 | 50.0 | 58.3 | 29.4 |
| Ultravox | 34.5 | 68.0 | 19.2 | 45.0 | 55.6 | 38.2 | 26.7 | 41.4 | 33.3 | 50.0 | 25.0 | 35.3 |
Table 15: FDB-v3 Pass@1 (%) by domain, difficulty and type of disfluency. Bold marks the best value in each column.
Figure 12: Where the time goes. (a) The fast path of Jarvis-Omni in recorded CS2 and League of Legends sessions, with stage medians and the end-to-end median from the end of the user’s speech to the first voiced word. (b) Stages of the slow brain of Jarvis-Audio over 42 benchmark sessions.
More in this topic
LLM Agent Harnesses
LLM agent harnesses are the software and runtime systems that organize an agent’s tools, context, state, execution, and safeguards.
See all 357 papers