Omni-JEV: Text and Dual-Stream Speech Decisions
Organizations: Zhejiang Merit Interactive Research Institute Co., Ltd.
Abstract
A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history. Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy. We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions. An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs. Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD). We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality. Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text. This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.
Figures & tables
| Field | Content and interpretation |
|---|---|
| text | User/assistant text for text examples; optional completed history preceding the audio window for speech examples. Current-window ASR text is excluded. |
| assistant_state | Source state in value , such as idle , playing , interrupted , or unknown ; optional flags retain pending interruption, resumability, and assistant speech during the collected segment. |
| policy , scenario | Available policy text and explicit scenario conditions, including addressing or resumability assumptions. These may be empty. |
| Role availability | Supplied user and assistant-reference spans; a missing role is stated in the prompt and its acoustic span is omitted. |
| Action | Operational interpretation |
|---|---|
| listen | Await more evidence when no response is to be preserved or resumed. |
| start | Begin a response to the user when no existing response should be preserved. |
| continue | Preserve a response that remains active; the segment does not warrant interruption. |
| stop | Interrupt a response active during the segment because of the user’s intent. |
| recover | Resume a response after a preceding stop or recoverable interruption. |
| Integrated source | Modality | Language | Train | Test |
| BANKING77 [ 18 ] | text | en/zh | 22,726 | 2,509 |
| MASSIVE [ 6 ] | text | en/zh | 27,491 | 3,034 |
| Legacy turn text [ 20 ] | text | en/zh | 45,570 | 5,030 |
| Mini Speech Commands [ 33 ] | audio | en/zh | 14,312 | 1,580 |
| MInDS-14 [ 9 ] | audio | en/zh | 904 | 100 |
| ESC-10 [ 22 ] | audio | none | 360 | 40 |
| Setting | Gaussian RLCD | Categorical RLCD |
|---|---|---|
| Exploration unit | Centered logit vector | Discrete candidate position |
| Target score | Log plus spherical | Negative Brier |
| Samples per input | ||
| Exploration control | ||
| Additional coefficient | Spherical weight | Ordinal weight |
| Model | Scope | Fail/invalid | Audio ACC | Audio Macro-F1 | Text ACC | Text Macro-F1 | |
|---|---|---|---|---|---|---|---|
| Qwen3.1-Realtime API only; 2,000-row budget-limited subset | |||||||
| Qwen3.1-Realtime API | 2k API | 2,000 | 0/0 | 37.06 | 35.18 | 92.32 | 71.97 |
| Full frozen evaluation; AudioJev is audio-only | |||||||
| Qwen3-Omni | full test | 34,860 | 0/0 | 26.43 | 13.17 | 77.53 | 58.18 |
| MiniCPM-o4.5 | full test | 34,860 | 0/0 | 47.40 | 23.28 | 83.58 | 65.51 |
| MOSS-Audio4B | full test | 34,860 | 0/0 | 51.50 | 23.61 | 72.87 | 49.95 |
| Modality | RLCD | ACC | Macro-F1 | Brier | NLL | |
|---|---|---|---|---|---|---|
| Audio | Gaussian | 24,287 | 84.08 | 76.00 | 2806.99 | 0.5863 |
| Audio | Categorical | 24,287 | 84.04 | 75.23 | 2499.43 | 0.4376 |
| Text | Gaussian | 10,573 | 96.55 | 80.56 | 655.79 | 0.1834 |
| Text | Categorical | 10,573 | 96.51 | 81.41 | 616.87 | 0.1331 |
| Pooled | Gaussian | 34,860 | 87.86 | 80.93 | 2154.54 | 0.4641 |
| Pooled | Categorical | 34,860 | 87.83 | 80.32 | 1928.46 | 0.3452 |
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
| Modality | Action | Train | Test |
|---|---|---|---|
| audio | listen | 138,507 | 15,617 |
| audio | start | 52,771 | 5,764 |
| audio | continue | 21,545 | 2,373 |
| audio | stop | 4,720 | 516 |
| audio | recover | 156 | 17 |
| text | listen | 737 | 81 |
| Source | Source unit and state | Action construction | Bilingual handling |
|---|---|---|---|
| BANKING77 | Bank-intent text; idle/no-resumable state added; no assistant history. | Saved DeepSeek text policy described above; the original intent category is excluded. | Translate text fields; retain assigned state and action. |
| MASSIVE | Intent text; the same added idle scenario, without a prior assistant turn. | The same text teacher distinguishes a completed contribution from unfinished or non-addressed input. | Translate text fields; retain assigned state and action. |
| Legacy turn text | Assistant/user text pair; playing state supplied by the source task instruction. | The text teacher rejudges the pair under the five-action policy; native three-class targets are not directly projected. | Translate both text roles; retain state and derived action. |
| Mini Speech Commands | Isolated word recording and its word class; context absent and assistant state unknown. | Fixed convention: stop/no stop ; yes/go continue ; up/down/left/right start . Playback compatibility is unverified. | Chinese TTS of the translated word; copy the action without VAD or ASR. |
| MInDS-14 | User utterance; idle/no-resumable state and completed direct request assumed at clip end. | Scenario rule assigns start ; native bank intent and transcript are not consulted. | English TTS uses the source-provided English transcription; copy the action without VAD. |
| ESC-10 | Environmental or nonlinguistic clip; idle/no-resumable scenario added, with no sound-identification or alarm task. | Under that scenario, assign listen to every included clip. | Nonlinguistic; no translated version is required. |
| Source | G ACC | G F1 | C ACC | C F1 | |
|---|---|---|---|---|---|
| BANKING77 | 2,509 | 99.64 | 49.91 | 99.60 | 49.90 |
| MASSIVE | 3,034 | 97.86 | 69.08 | 97.79 | 70.99 |
| Legacy turn text | 5,030 | 94.21 | 93.36 | 94.19 | 93.34 |
| Mini Speech Commands | 1,580 | 98.61 | 98.54 | 98.42 | 98.31 |
| MInDS-14 | 100 | 100.00 | 100.00 | 100.00 | 100.00 |
| ESC-10 | 40 | 100.00 | 100.00 | 100.00 | 100.00 |
| Slice | G ACC | G F1 | C ACC | C F1 | |
|---|---|---|---|---|---|
| audio / zh | 19,968 | 81.90 | 73.34 | 81.86 | 72.43 |
| audio / en | 4,279 | 94.11 | 93.31 | 94.11 | 93.17 |
| audio / none | 40 | 100.00 | 100.00 | 100.00 | 100.00 |
| text / zh | 5,274 | 96.47 | 80.06 | 96.40 | 81.51 |
| text / en | 5,299 | 96.62 | 81.06 | 96.62 | 81.30 |
| Model | ACC | Macro-F1 | |
|---|---|---|---|
| Text JEV V7-B | 1,606 | 98.38 | 98.40 |
Full-Duplex Spoken Dialogue Systems
Full-duplex spoken dialogue systems let voice agents listen and respond simultaneously, managing timing, overlap, interruptions, and conversational content.
See all 129 papers