Full-Duplex Interaction with Speech, Expressions and Gestures

Siyin Wang; Wenyi Yu; Terumi Chiba; Xianzhao Chen; Xiaohai Tian; Jun Zhang; Lu Lu; Chao Zhang

eess.ASOct 15, 2026

Full-Duplex Interaction with Speech, Expressions and Gestures

Authors: Siyin Wang, Wenyi Yu, Terumi Chiba, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Lu Lu, Chao Zhang

Organizations: Tsinghua University · ByteDance

Abstract

Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combines pretrained acoustic features with distinct listening and speaking hidden states from the dialogue backbone and speech synthesizer, enabling continuous non-verbal generation even when the model is verbally silent. To translate dialogue understanding into vivid semantic gestures, we introduce contextual semantic calls and template guidance, separating whether to invoke an available semantic gesture from how to realize it. We also establish a curated data and evaluation pipeline for dialog-aware co-speech gesture generation and investigate reference-based multimodal large language models (MLLMs) as automatic judges. Experiments show that ELLSA 2 remains highly competitive with open-source full-duplex speech interaction models and outperforms cascaded baselines across dialog-aware gesture and expression metrics. Semantic template guidance further significantly improves semantic gesture quality. Strong system-level correlations with human judgments support these reference-based scores as useful preliminary metrics for full-duplex and semantic gesture assessment. Together, these results advance dialogue modeling from spoken responses toward coordinated multimodal interaction.

Figures & tables

Figure 1: ELLSA 2 jointly models full-duplex speech, gesture, and facial expression generation. The gesture-expression synthesizer conditions on hidden states from both user and model speech and autoregressively generates discrete gesture and expression codes every 800 ms. It supports two generation modes: free generation and guided generation, which additionally incorporates optional semantic gesture guidance for more semantically grounded gesture generation.

Figure 2: Data preparation and evaluation overview. Speaker selection and expression extraction supply paired training supervision. The broader source pool supplies semantic templates. Human-verified clips support activity, prosodic, duplex, and semantic assessment. The reference-based duplex and semantic scores assess communicative function beyond activity and rhythmic alignment.

Figure 3: Qualitative comparison for “The house is big and spacious.” Columns show successive poses at 0.25 s intervals. In the guided row, the hands rise in front of the torso and then open outward, providing a visible spatial-extent cue. Aggregate results appear in Table 2 .

Model	Llama Q.		Web Q.		TriviaQA		AlpacaEval
Model	Quality	TT	Quality	TT	Quality	TT	Quality	TT
Moshi	60.8/54.5	85.0	23.4/22.1	76.0	25.6/16.7	37.1	1.84/1.76	83.4
PersonaPlex	62.7/61.7	93.3	22.8/22.5	100.0	24.5/22.8	49.1	2.71/2.62	65.3
Freeze-Omni	74.2/56.2	99.7	40.8/27.9	99.8	45.1/28.5	72.0	3.90/2.46	87.9
SALMONN-omni	79.3 /73.6	99.7	49.7/43.7	99.6	63.6/56.0	92.8	4.01 / 3.22	92.0
ELLSA 2	79.3 / 78.0	99.3	52.6 / 47.3	99.3	68.9 / 63.6	94.4	3.97/3.20	87.9

Table 1: Spoken response quality and simulated turn taking accuracy (TT, %). Each quality cell gives S2T/S2S: QA accuracy (%) or AlpacaEval scores (1–5). Higher is better. Protocol details appear in Appendix C.1 .

Model	$R_{\mathrm{SL}}\uparrow$	$A\uparrow$	Human D $\uparrow$	Human S $\uparrow$	MLLM D/S $\uparrow$
GestureLSM	1.217	0.612	2.164	1.444	0.205/0.309
MIBURI	1.041	0.471	1.858	2.014	0.194/0.313
ConvoFusion $\dagger$	1.326	0.606	1.321	1.049	0.234/0.270
audio2photoreal $\dagger$	1.446	0.572	1.545	1.120	0.161/0.239
Our standalone generator	1.545	0.703	3.007	2.070	0.318 /0.351
ELLSA 2	1.563	0.723	3.201	2.014	0.300/0.347

Table 2: Gesture evaluation results. Higher is better.

A/D/S

denote alignment, duplex, and semantic scores, respectively. Human D/S scores range from 1 to 5, while MLLM D/S scores range from 0 to 1. Human ratings are treated as the gold-standard evaluation.

\dagger

: reproduced.

Metric	MLLM-MLLM	Human-Human	MLLM-Human	MLLM mean-Human mean
Duplex	0.845	0.987	0.806	0.853
Semantic	0.950	0.940	0.904	0.919

Table 3: System-level Pearson correlations between MLLM and human evaluations. For the first three correlation columns, we report the mean pairwise correlation across individual MLLM judges and/or human annotators.

Model	LVE	MHD	FDD (S/N)	JDD (S/N)
ARTalk	9.35	2.07	26.07/n/a	1.74/n/a
MIBURI	13.52	2.90	22.87/n/a	3.45/n/a
UniLS	9.37	2.06	19.14/19.88	1.41/1.79
DualTalk †	12.72	2.66	22.54/20.41	1.08/1.33
Our standalone generator	8.82	1.90	14.90/15.16	1.01/1.26
ELLSA 2	8.77	1.89	14.44/14.82	0.98 /1.25

Table 4: Expression evaluation; lower is better. LVE/MHD use speaking frames (mm); FDD and JDD use units of

10^{-2}

mm and

10^{-2}

rad. S/N: speaking/non-speaking.

\dagger

: partner facial input; n/a: no native listening condition.

Appendix figures & tables10 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	S1 (ASR)	S2 (gesture)	S3 (gesture–expression)
Peak / minimum learning rate	$10^{-4}/10^{-5}$	$4\!\times\!10^{-5}/4\!\times\!10^{-6}$	$10^{-4}/10^{-5}$
Training steps	50000	50000	50000
Warmup updates	500	1,000	1,000
Local batch / accumulation	32 / 1	16 / 2	8 / 2
Global batch	256	256	128
Audio encoder	Frozen	Trainable	Trainable

Appendix

Table 5: Detailed training configuration for independent motion LM.

Setting	Value
Optimizer	Fused AdamW; $\beta=(0.9,0.95)$ , $\epsilon=10^{-6}$ , weight decay 0.1
Base learning rate	$2\times 10^{-5}$ ; cosine decay; floor $10^{-6}$
Warmup	1,000 generator updates; 1,000 speech updates after unfreezing
Learning-rate multipliers	LLM LoRA 0.25; speech synthesizer 0.5; call head 1; gesture synthesizer/connector 1
Precision and clipping	bfloat16, TF32 enabled; global gradient norm 5
Distributed configuration	ZeRO-2, gradient checkpointing

Appendix

Table 6: Detailed ELLSA 2 training configuration of joint optimization stage 4.

Stage and counting unit	Retained
Paired source pool	121,054
Recordings with at least one video-verified semantic event	29,248
Annotated events in those recordings	77,585
Event-centered windows passing pose screening	46,894
Candidates passing text/family/anchor checks	35,436
Candidates also passing video-confidence and visibility checks	34,817

Appendix

Table 7: Semantic exemplar library construction from Seamless Interaction. Participant recordings, event windows, and retained library exemplars are different counting units.

Figure 4: Human test-set verification interface, showing a semantic candidate. Reviewers inspect the reference motion, target/partner audio, transcripts and anchor timing before recording a keep/reject decision and rationale.

Metric / aggregation	Spearman $\rho$	Kendall $\tau$	Pearson $r$	Correct pairs
$A$ , frozen	0.9643	0.9048	0.9729	20/21
BeatAlign, matched mean	$-0.6786$	$-0.5238$	$-0.7452$	5/21
BeatAlign, mean margin	0.5357	0.4286	0.8828	15/21
BeatAlign, sigmoid contrast	0.3929	0.2381	0.8098	13/21

Appendix

Table 8: System-level agreement with GENEA human appropriateness on the same seven generated systems and 616 comparisons (88 segments per system). BeatAlign controls use the same matched/mismatched pairs as

A

. Correlations are descriptive, with system-level

n=7

System	Weights / training provenance	Evaluation conditions	Output adaptation
GestureLSM	Released new_540_shortcut.bin , with released RVQ decoders	Target speech and aligned transcript; fixed released speaker ID	SMPL-X body/hand rotations mapped to common SMPL-H joints at 30 Hz
MIBURI	Released good-speaker last_6500.safetensors , with body/face codecs	Target Mimi speech tokens and aligned text tokens	SMPL-X body/hands plus 100 facial coefficients and jaw; 25 to 30 Hz
ConvoFusion	Reproduced Gaussian VAE epoch 99 and diffusion epoch 199	Target audio/text, partner text, hashed speaker ID	189 pose slots at 25 Hz; merged and resampled to SMPL-H at 30 Hz
audio2photoreal	Reproduced VQ 300k, guide 995k, diffusion 630k	Target/partner audio channels; audio-predicted 1 Hz keyframes	165-dimensional axis-angle output mapped to SMPL-H at 30 Hz
ARTalk	Released ARTalk_wav2vec.pt	Target speech and released style conditioning	100 FLAME expression coefficients plus head/jaw pose; 25 to 30 Hz
UniLS	Released UniLSLoRAGen.pt	Target and partner speech	100 expression coefficients, head pose, jaw opening and eye slots; 25 to 30 Hz

Appendix

Table 9: Baseline configurations used for the non-verbal comparisons.

Figure 5: Gesture human-evaluation interface. The human reference anchors score five, while the anonymous candidate receives one overall 1–5 task rating with observable evidence. The panel includes synchronized playback, event timing, transcripts, silence cues, rating anchors and an invalid-media flag.

Stage	Task	Source	Entries
1	ASR	Following SALMONN-omni	481,241
1, 2, 4	Spoken QA	Following SALMONN-omni	982,398
2, 4	Conversation	Synthetic multi-turn speech dialogues	433,941
3, 4	Free generation	Selected Seamless Interaction with facial supervision, counted as participate recordings	1,318
3, 4	Guided generation	Derived from free generation pool, counted as 20s sample	8,372
4	Semantic call	Contextual gesture-policy annotations	10,145

Appendix

Table 11: Training-data inventory. Counts refer to serialized training entries before runtime length filtering, crop sampling, and task resampling. Ordinary generation entries are paired recording records; inline entries are prepared windows; semantic-call entries are text segments. The Stage-2/4 speech pool is reused rather than counted as two distinct datasets.

Training pool	Recordings	Speakers	$R_{\mathrm{SL}}\uparrow$	$A\uparrow$	$MLLMD\uparrow$	$MLLMS\uparrow$
Full pool	121,054	3,401	1.398	0.525	0.080	0.184
Selected speakers	1,318	81	1.487	0.668	0.332	0.364

Appendix

Table 12: Gesture training-data ablation. D/S are MLLM scores.

Tokenizer	LVE (S)	MHD (S)
Unified six-level RVQ	9.4962	2.0255
Factorized $4+2$ RVQ	8.8214	1.9031

Appendix

Table 13: Facial-tokenization ablation.

Facial Expression Synthesis

See all 29 papers

Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems
Jun 20, 2026Jingjing Jiang, Atsumoto Ohashi, Ryuichiro HigashinakaAudio-Driven Facial Animation Multimodal Generation
Graduate School of Informatics, Nagoya University, Japan
TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction
Sep 8, 2026Yi-Chang Chen, Chun Wei Chen, Dien-Ruei Wu +7TTS Synthesis Streaming TTS Synthesis
MediaTek Research · Internship at MediaTek Research · National Taiwan University +2
Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis
Jul 27, 2026Yifan Hu, Shuwei He, Rui Liu +1TTS Synthesis Audio-Driven Facial Animation
Inner Mongolia University Hohhot, China · SRIBD, School of Artificial Intelligence The Chinese University of Hong Kong Shenzhen, China

Jun 20, 2026J/K move · Enter open · S save

Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems

Jingjing Jiang, Atsumoto Ohashi, Ryuichiro Higashinaka

Graduate School of Informatics, Nagoya University, Japan

Audio-Driven Facial Animation Multimodal Generation Full-Duplex Spoken Dialogue Systems

Open paper AskPDF arXiv

Full-duplex spoken dialogue models, such as Moshi, enable natural, low-latency voice conversations. However, they remain limited to the audio modality, lacking the facial expressions that are integral to human communication. We present Moshi-Face, the first full-duplex dialogue model that jointly processes the user's audio and facial input while simultaneously generating speech and facial motion. We first construct a vector-quantized variational autoencoder (VQ-VAE) as a face codec that encodes 3D head meshes extracted from facial videos into compact discrete tokens, referred to as face tokens, and conversely reconstructs 3D meshes from these tokens. We then extend Moshi with a Face Transformer module that generates face tokens non-autoregressively, enabling Moshi-Face to produce synchronized audio and face tokens in real time. Experiments show that Moshi-Face achieves audiovisual alignment at low latency while preserving the dialogue quality of the original audio-only model.

Abstract

Facial Expression Synthesis

Explore similar work

Integrating Facial Generation into Full-Duplex Spoken Dialogue Systems

TASTE2: Text-Aligned Speech Modeling and Deployment toward Full-Duplex Voice Interaction

Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis