Duopra: From Full-Duplex Conversation to Full-Duplex Agency
Organizations: Shanghai Jiao Tong University · Tencent · Shanghai University · University of Bristol
Abstract
Full-duplex interaction models exhibit strong conversational agency, enabling natural, low-latency turn-taking, backchannels, and interruptions. However, completing tasks further requires task agency: timely tool invocation and argument revision following user corrections. Frontend-backend agents separate initiation from execution: the frontend decides whether and what to request, while the backend determines how. Execution can be offloaded, but unsent requests cannot be executed, and incomplete ones can misdirect the backend. On Full-Duplex-Bench v3 (FDB v3), most failures of Realtime-Venus-Audio + Harness in our streaming pipeline occur before a domain tool executes. We introduce Duopra, a full-duplex interaction model that learns task execution alongside conversational control. Duopra treats the backend as a tool, accepting a self-contained natural-language request. Direct calls and backend delegation share an action interface and requirements for faithful initiation and correct execution. Its three-stage training combines protocol SFT, full-duplex RL rewarding timely and complete execution, and projector-only repair of talker conditioning after thinker updates. With the same backend, Pass@1 rises from 4.7% to 44.7%; Duopra achieves 47.7% calling tools directly (26.0% after SFT) and 49.3% with a DeepSeek backend. It improves three FDB v1.0 takeover rates over the base model. These results show that task agency can coexist with key conversational agency.
Figures & tables
| Task agency FDB v3 | Conversational agency FDB v1.0 TOR | |||||||||
| System | Backend LLM | Call | Pass call | Tool | Arg. | Resp. | Pass@1 | Turn | Pause | Intr. |
| (a) Commercial systems, published results | ||||||||||
| GPT-Realtime | – | – | – | 87.6 | 68.0 | 79.2 | 60.0 | – | – | – |
| Gemini Live 3.1 | – | – | – | 81.7 | 58.8 | 71.8 | 54.0 | – | – | – |
| MOSS-Live m | – | – | – | – | – | – | 51.0 | – | – | – |
| Gemini Live 2.5 | – | – | – | 78.6 | 59.3 | 55.4 | 49.0 | – | – | – |
Appendix figures & tables17 assets
Supplementary material from the paper’s appendix.
Appendix
| Base model: RV-Audio (9B) | |
|---|---|
| Thinker | LLM, hidden size 4096 |
| Talker | 20 layers, 768 dimensions, 12 heads |
| Audio tokens | s3tokenizer, vocabulary 6,562 |
| Vocoder | Token2wav, 24 kHz |
| Projector | MLP, , 3,737,088 parameters |
| Stage 1: Protocol SFT | |
| Stage | Set | Size | Composition |
|---|---|---|---|
| 1 | Training pool | 12,800 sessions | 8,951 Chinese, 3,849 English; 7,680 task-oriented, 5,120 retention (of the pool: direct calls 7.5%, tool discovery 6.0%, clarification 4.0%, stopping 7.5%, redirection 8.7%, continuing after a backchannel 4.8%, long reading 3.5%, multi-turn dialogue 8.0%, other tool tasks 10.0%). |
| 1 | Validation; test | 4,711; 4,195 sessions | kept apart from training; they include the 16 + 16 held-out tools |
| 2a | Task pool | 7,908 tasks | clarification-to-answer chains 34.2%, direct calls and discovery 31.7%, stopping and resuming 9.8%, redirection vs. backchannel 7.5%, asking first 5.9%, answering without a tool or unavailable capability 4.5%, corrections and cancellations 3.6%, speech-to-tool switches 2.9% |
| 2b | New tasks | 1,087 training; 152 validation | training: D1 175, D2 350, D3 86, D4 368, D5 108; all required tools visible 587, some backend-only 317, all backend-only 183; 681 English, 406 Chinese. Validation: D1 30, D2 98, D4 24 |
| 2b | Sampling mixture | 45% new, 55% Stage-2a | new tasks and Stage-2a tasks are sampled 45:55; in 30% of the Stage-2a samples query is also visible, and the intended behavior there remains a direct call |
| 3 | Cached features | 931 / 100 / 210 sessions | The training, validation, and test sets contain 931, 100, and 210 sessions, yielding 35,154, 3,464, and 8,114 hidden-state pairs, respectively. |
| Family | Task pattern | Expected behavior |
|---|---|---|
| D1 Single step | A disfluent spoken request requiring one tool call. | Call the tool directly if visible; otherwise use query . Act within 3 units after the request ends. |
| D2 Multi-step | Independent steps, result dependencies (“the cheapest one”), or conditional branches. | Complete all steps directly, or delegate a task specifying the steps, conditions, and dependencies. |
| D3 Corrections | A value is corrected or a request is abandoned mid-sentence. | Use the latest value and skip abandoned requests. |
| D4 Sufficiency | Four variants: complete information, a missing required value, missing optional values, or a value obtained from a previous step. | Ask only for missing required information; otherwise proceed with the available values and tool results. |
| D5 Recovery | The backend requests missing information, returns an error or partial result, or a tool fails transiently. | Request missing values, retry a recoverable failure once, or report partial completion accurately. |
| Unit | Input: user audio and observations | Thinker output |
|---|---|---|
| 1 | Is my flight CA1701 … Tool schema: <backend> {"tools":[{"name":"flight.status", "parameters":{"required": ["flight","date"],...}, "version":"2",...},...}]} </backend> | <|listen|> |
| 2 | …delayed tomorrow? | <|listen|> |
| 3 | (silence) | <|speak|> Let me check. <|chunk_eos|> |
| 4 | (silence) | <|speak|> <delegate> {"op":"call","call_id":"c1", "tool":"flight.status", "arguments":{"flight":"CA1701", "date":"2026-11-06"}, "schema_version":"2"} </delegate> <|chunk_eos|> |
| 5 | (silence) | <|speak|> <|turn_eos|> <|chunk_eos|> |
| 6 | (silence) Tool result: <backend> {"call_id":"c1", "payload":{"result":{ "delay_minutes":20, "status":"delayed"},...}} </backend> | <|speak|> It’s twenty minutes <|chunk_eos|> |
| Field | Verbatim content |
|---|---|
| Thinker system prefix | Streaming Omni Conversation. |
| Backend tool assistant.query in training | |
| Description | Delegate a complete, self-contained task to a frozen background assistant. This tool has exactly ONE argument: task, a string. Put the actual complete request and all known facts inside that string. Do not pass separate city, date, account, period or other business-tool fields to assistant.query. Use a visible direct tool for a simple supported request. For several goals, result dependencies, conditions, or a capability available only in the background, send all active steps, constraints and already-heard values in task. Required values may come from the user, explicit session context, defaults, or a named preceding result. Ask only for an essential required value that has none of these sources; optional fields may use their defaults. Keep final corrections and exclude abandoned requests. The background cannot hear this conversation. No transcript, previous select or hidden facts are added to task. Only registered capabilities and session permissions apply. After done, need_info or error, answer honestly from delivered observations, ask for genuinely missing information, or retry at most once after an error. Do not speak routing fields or private JSON. |
| Argument | task , a string of 1–1,600 characters: Complete active request, all known values, ordered steps, conditions and result dependencies. |
| Result | status ( done , need_info or error ), answer , steps , missing_information ; the backend LLM writes all but steps , which the runtime fills from its call log. |
| Backend tool assistant.query in the FDB v3 adapter (Qwen3-8B) | |
| Component | Reward rule |
|---|---|
| Action decision | Information is sufficient and the frontend acts within 3 units after the user finishes: . Information is sufficient but the frontend does not act: staying silent ; speaking without acting ; asking an unnecessary question ; each of the three additionally incurs for the missing call. |
| Sufficiency | A required argument is missing and the frontend asks only for the missing items: . Asking for optional or already given information: per item. Acting although a required argument is missing: . Inventing a value for the user: . |
| Step coverage | , where is the fraction of requested steps that are executed correctly. Including a task the user abandoned: . |
| Argument fidelity | ; a step counts toward only if all its values are correct. Each value from before a correction: . |
| End-to-end | The executed calls (direct and backend) match exactly in tools and arguments: . |
| Result readout | Complete readout ; partial readout . Reporting a result before its observation arrives is marked as fabrication; the applicable cap and outcome mapping are specified separately for each task family in Appendix D . |
| Reward-judge instruction (Qwen3.8-27B; verbatim) | |
|---|---|
| Part I: Assess claims and completion | Part II: Ground evidence and return labels |
| Judge ONLY these public words using ONLY supplied consumed_evidence. No future result exists in this request. covered_goals is an array of zero-based requested_goals indices whose actual result is correctly stated, or whose observed service failure is honestly disclosed. Promises, questions and repeating requests cover NO goals. Check every factual claim separately, especially negation, quantities, dates and identifiers. Judge factual correctness separately from speaking fluency: a grammatical classifier error or filler does not change an otherwise explicit numerical value and unit. Do not label that alone as fabricated. An incomplete answer is not automatically a contradiction: stating only August when the date is in August is incomplete (does not cover a full-date goal), but not fabricated. A different month is contradictory. Similarly, six twelve without AM/PM is a 12-hour clock component, not an explicit 06:12 claim; do not mark it fabricated solely because the source uses 18:12. An explicit contradictory AM/PM is wrong. Digit-by-digit place numbers (Shenzhen three nine five versus Shenzhen 395) denote the same entity; different digits or years remain wrong. fabricated=true for ANY asserted factual result absent from or contradictory to consumed_evidence. An answer spoken without any evidence is fabricated even if coincidentally true. A refusal or an unsupported claim that a tool is unavailable is false_refusal, NOT a fabricated business result. This includes claiming that a lookup failed, a record was unavailable, or nothing was found when no consumed error/empty-result observation supports that claim. Lack of evidence is NOT evidence that a lookup failed. With consumed_evidence=[], saying I have not checked yet is honest uncertainty; saying I checked but the record was unavailable is false_refusal=true. An actually consumed matching error justifies the refusal. Saying I cannot confirm, I will search for a tool, or the tool is unavailable does not assert a business answer. Set fabricated only for asserted business facts (including vague flight-on-time, open/closed, or all-clear claims). An observed error may justify an honest failure disclosure but NEVER proves that the business goal succeeded. unfinished_disclosures is an array of [goal_index,exact_quote] explicitly saying that a requested step was not completed, could not be checked, or remains unresolved. Promises and merely not mentioning a step are NOT disclosures. A general statement that the whole background task failed can disclose all its goals, but earns no business completion credit. Do not invent a statement about an omitted goal. assistant.query status done is only a backend claim; actual business results, not completion prose, establish truth. false_all_done=true when the speaker claims all steps succeeded but evidence is incomplete or failed. allowed_goal_sources explicitly lists the only real source indices for each goal. An empty list means NO evidence for that goal; never infer its result from another goal, the request, or the backend completion claim. | For EVERY covered_goals entry supply exactly one goal_proofs entry [0,0,"result","exact public words answering that goal"], ordered as goal_index,record_index,kind,quote. kind must be result or observed_failure. The record_index must occur in allowed_goal_sources for that goal and kind; quoted words must really answer that specific goal. A supported result for one goal cannot support another. Unsupported assertions still set fabricated=true, but do not belong in covered_goals or goal_proofs. For each stated RESULT number, complete calendar date, clock time or identifier, numeric_claims MUST contain ["exact numerical words from public_text",0,["payload","result","field"]], ordered as quote,record_index,path. quote must be the exact spoken quantity INCLUDING its unit and any classifier, WHOLE stated date including its year WHEN SPOKEN, WHOLE clock time including any AM/PM words, or WHOLE identifier, not a corrected value. Include trailing currency qualifications in the same quote: dollars in CNY explicitly names CNY; do not shorten it to dollars. Compare the explicit currency independently of the number. A month/day without a spoken year is only a month/day claim; cite it verbatim and NEVER add an inferred year. If a year is split across chunks such as 20 31, retain those exact spaces in the quote. Use separate claims for separate fields. path starts in consumed_evidence[record_index]; numeric_sources is the complete catalog of numeric/date/time/ID fields actually available NOW. Copy record_index and path EXACTLY from that catalog; paths to result fields begin with payload,result. Currency names, city names, ordinary place names, and demonstratives such as that date or this number are NOT numeric claims. A demonstrative contains no stated value; numeric_claims must omit it. If the catalog is empty, do not invent source indices. A promise repeating an input ID/date has numeric_claims=[] even when the real call will occur later. Unsupported RESULT assertions instead use null,[] and set fabricated=true. Repeating an input in a promise does not assert a result. A repeated input identifier (such as a subscription ID) is not a newly returned amount. If it is cited, its proof MUST point to ["arguments","actual_argument_field"] of the matching real call, NEVER to an amount or date field. Repeating input values in a promise or question is not a result claim; numeric_claims may be empty for such speech. include contradictory claims too, pointing to the field they contradict. If no corresponding numeric result field exists, set record_index to null and path to []; never invent a field or record. Do NOT round counts, dates, IDs or quantities. The Python caller independently parses quote and compares to that field. private_metadata_spoken flags internal tool names, JSON and routing/schema instructions, not ordinary nouns. reason explains the comparison in at most 12 words; no other response fields. Use compact JSON on one line, without indentation or line breaks. Never repeat the whole input. |
| Model | Thinker | Projector |
|---|---|---|
| Duopra-SFT | after Stage 1 | base |
| Duopra-RL1 | after Stage 2a | base; used on FDB v1.0, FDB v1.5, VoiceBench and FDB v3 (Direct) |
| Duopra-RL | after Stage 2b | base; used for the projector drift analysis (Figure 5 b) |
| Duopra | Duopra-RL | Stage-3 projector (all benchmarks) |
| Cited system | Source | Evaluation protocol |
|---|---|---|
| GPT-Realtime, Gemini Live 2.5 and 3.1, Grok, Ultravox, Cascaded | Lin et al. (2026a) , Table 2 | GPT-4o judge. GPT-Realtime is gpt-realtime-1.5 and Ultravox v0.7; Cascaded is Whisper ( Radford et al., 2023 ) , GPT-4o and OpenAI TTS in sequence. |
| MOSS-Live | MOSI (2026) , developer report, accessed 12 November 2026 | The developer report gives a tool-call success rate of 51.0%. |
| Gander, GPT-5.6 as the back brain | Orantqing et al. (2026) , Table 3 | The benchmark’s scoring scripts with GPT-4o as the judge. |
| NemotronLabs VoiceChat | Balam et al. (2026) , Table 4 | Tool-selection F1, argument accuracy and Pass@1. |
| RV-Audio, offline | Venus Team (Ant Group), Tsinghua University (2026) , Table 7 (arXiv v3) | Offline inference with the tool definitions in the prompt; judge not named. We evaluate RV-Audio + Harness in our streaming pipeline separately (Appendix E ). |
| Hu et al. , Qwen3-235B-A22B backend | Hu et al. (2026) , Table 2 (arXiv v2) | Tool-selection accuracy, not F1; GPT-4o judges the agent’s text, transcribed by an external ASR system. |
| FDB v1.0 metric | Reported | Ours |
|---|---|---|
| Turn-taking TOR | 88.2 | 86.3 4.0 |
| Pause TOR, real | 46.3 | 38.7 1.8 |
| Interruption TOR | 98.5 | 99.2 1.4 |
| Outcome (sessions) | No call | Passes by calls | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Frontend | Exposure | Backend LLM | Pass@1 | Pass | Wrong arg. | Missing | Extra | Spoke | Silent | 1 | 2 | 3 |
| RV-Audio + Harness | FE–BE | Qwen3-8B | 4.7 1.2 | 14 | 9 | 7 | 1 | 144 | 125 | 11 | 1 | 2 |
| Duopra-SFT | Direct | none | 26.0 3.6 | 78 | 17 | 38 | 1 | 118 | 48 | 76 | 2 | 0 |
| Duopra-RL1 | Direct | none | 26.7 0.6 | 80 | 17 | 72 | 0 | 85 | 46 | 80 | 0 | 0 |
| Duopra-RL1 | Both | Qwen3-8B | 24.3 1.5 | 73 | 17 | 60 | 0 | 94 | 56 | 73 | 0 | 0 |
| Duopra-RL1 | Backend | Qwen3-8B | 20.0 1.0 | 60 | 27 | 5 | 9 | 119 | 80 | 48 | 11 | 1 |
| FDB v3 | FDB v1.0 TOR | |||||||||||
| (a) Model | Pass@1 | Easy | Med. | Hard | 2–3 calls | Tool | Arg. | Resp. | Take | Turn | Pause | Intr. |
| RV-Audio | 0.0 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 17.0 | 51.3 | 59.1 | 18.1 | 79.5 |
| Duopra-SFT | 26.0 3.6 | 43.5 | 23.5 | 7.8 | 2.0 | 39.2 | 30.1 | 56.8 | 82.7 | 75.1 | 9.3 | 97.0 |
| Duopra-RL1 | 26.7 0.6 | 47.2 | 25.5 | 3.3 | 0.0 | 44.7 | 32.2 | 59.1 | 82.0 | 75.6 | 5.6 | 96.5 |
| Duopra | 47.7 3.5 | 59.3 | 52.0 | 28.9 | 27.5 | 76.7 | 56.2 | 66.3 | 93.0 | 75.6 | 5.4 | 91.3 |
| Reference: frontend–backend agents with the Qwen3-8B backend | ||||||||||||
| Metric | MiniCPM-o 4.5 | RV-Audio | Duopra-SFT | Duopra-RL1 | Duopra |
|---|---|---|---|---|---|
| (a) FDB v1.0 | |||||
| Turn-taking TOR | 86.3 4.0 | 59.1 2.6 | 75.1 0.5 | 75.6 0.8 | 75.6 3.0 |
| Pause TOR, real | 38.7 1.8 | 18.1 2.3 | 9.3 0.9 | 5.6 2.1 | 5.4 0.3 |
| Interruption TOR | 99.2 1.4 | 79.5 3.0 | 97.0 1.5 | 96.5 0.9 | 91.3 0.6 |
| Pause TOR, synthetic | 10.5 1.1 | 10.5 1.8 | 5.4 1.1 | 3.2 1.1 | 2.4 1.1 |
| Turn-taking latency (s) | 2.23 0.08 | 2.27 0.15 | 2.64 0.04 | 2.75 0.07 | 2.75 0.05 |
| Error rate (%) | ECAPA cosine | |||
|---|---|---|---|---|
| Projector | CER | WER | zh | en |
| Base projector | 6.01 | 9.74 | 0.778 | 0.636 |
| Stage-3 projector | 2.58 | 7.66 | 0.782 | 0.648 |
| User request | Executed calls | Agent’s speech |
|---|---|---|
| (a) Backend; pass Um… so you know… search for a laptop — wait… no,… um…… um, I actually need a tablet, not a laptop. And add 2 of them to the cart. Actually, just add 1 instead, I only have budget for one right now. | search_products (query: tablet) add_to_cart (product_id: tablet_001, quantity: 1) Both at 30.5 s. | I will search for a tablet and add one to your cart. The tablet tablet premium has been successfully added to your cart. The total price is $99. |
| (b) Both; pass Um… so hmm… hmm… well… could you track order B-O-B for me first? Then… um,… um, I also want to search for a nice watch under 200 dollars. And once you find something good, uh, go ahead and add it to my cart. | track_order (order_id: BOB) at 36.0 s search_products (query: watch, max_price: 200) at 43.2 s add_to_cart (product_id: PROD1, quantity: 1) at 51.0 s | I will track the order and search for a watch. Your order B B is out for delivery. I found a watch named Watch Premium for $190. The watch has been added to your cart. |
| (c) Backend; wrong argument Hmm… um… so like… first, search for a gaming mouse —… um, I want to see what you have. Then add the cheapest option to my cart so I can buy it later. And while you’re at it, could you also track my order P-O-9-9-9? I’ve been waiting on that one. | search_products (query: gaming mice) at 29.0 s add_to_cart (product_id: PROD1, quantity: 1) at 30.3 s track_order (order_id: P0999) at 31.4 s | I will search for gaming mice. Add the cheapest one to your cart and check the status of order P0999. The order P09999 is currently out for delivery. The Gaming Mouse Gaming Mice Premium has been added to the cart with a total cost. |
| (d) Backend; no call Uh… like, you know… um… so search for a coffee maker first. If you find one that’s under 50 dollars, go ahead and add 2… um, to the cart because they’d make great gifts. But if everything’s over 50, just forget the coffee maker and track order W-W-W for me instead. | none | I will search for a coffee maker under fifty dollars and add two to the cart. If everything is over fifty, I will forget the coffee maker and track order www for me instead. |
LLM Agent Training
Large language model agent training teaches models to plan, use tools, and learn from multi-step interaction with environments.
See all 623 papers