cs.LGNov 10, 2026

Omni-JEV: Text and Dual-Stream Speech Decisions

Authors: Shengzhou Kong

Organizations: Zhejiang Merit Interactive Research Institute Co., Ltd.

Abstract

A user utterance does not uniquely determine the next action in full-duplex dialogue: its consequences depend on the assistant's response state and interaction history. Speech activity, utterance completion, and interruption intent provide related evidence, whose implications for a response require an explicit action policy. We formulate this task at collected-segment endpoints and introduce Omni-JEV, which maps text or role-indexed user and assistant-reference speech to probabilities over five interaction actions. An aligned speech-to-speech understanding backbone encodes the acoustic roles separately and shares its candidate readout with text inputs. Bilingual weak supervision supports cross-entropy fine-tuning followed by reinforcement learning for probability-sensitive decisions (RLCD). We compare Gaussian-logit and categorical Choice RLCD from the same supervised initialization under matched training exposure, observing different trade-offs between classification and probability quality. Gaussian Omni-JEV achieves test accuracies of 84.08% for audio and 96.55% for text. This common action interface supports decisions about when a full-duplex voice assistant should respond, preserve ongoing speech, interrupt, or resume an interrupted response.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript

    Oct 2, 2026Jie Jin, Ziyin Ma, Min Yin +4Audio QAAudio Understanding

  2. BayLing-Duplex: Native Full-Duplex Speech Dialogue with a Single Autoregressive LLM

    Jun 12, 2026Qingkai Fang, Shoutao Guo, Yang FengSpoken Dialogue SystemsSpeech Language Models