Organizations: College of Computing and Data Science, Nanyang Technological University · Mila, Quebec Artificial Intelligence Institute, Université de Montréal
During conversational interactions, humans subconsciously engage in concurrent thinking while listening to a speaker. Although this internal cognitive processing may not always manifest as explicit linguistic structures, it is instrumental in formulating high-quality responses. Inspired by this cognitive phenomenon, we propose a novel Full-duplex LAtent and Internal Reasoning method named FLAIR that conducts latent thinking simultaneously with speech perception. Unlike conventional "thinking" mechanisms in NLP, which require post-hoc generation, our approach aligns seamlessly with spoken dialogue systems: during the user's speaking phase, it recursively feeds the latent embedding output from the previous step into the next step, enabling continuous reasoning that strictly adheres to causality without introducing additional latency. To enable this latent reasoning, we design an Evidence Lower Bound-based objective that supports efficient supervised finetuning via teacher forcing, circumventing the need for explicit reasoning annotations. Experiments demonstrate the effectiveness of this think-while-listening design, which achieves competitive results on a range of speech benchmarks. Furthermore, FLAIR robustly handles conversational dynamics and attains competitive performance on full-duplex interaction metrics.
Figures & tables
Figure 1: (a) Turn-based interaction. The agent remains idle and starts to respond after the end of the user turn, but cannot be interrupted by user, as shown in the red box. (b) Full-duplex SDLM continuously listens to streaming speech input, supports user barge-in, and automatically switches between thinking and speaking like a human speaker.
Figure 2: The overview of proposed FLAIR. During the user’s speech phase, the LLM performs latent reasoning, using the LLM’s output latent embeddings as the input for the next step. Once the user finishes speaking, the assistant autonomously decides when to respond; the LLM then executes an explicit forward pass, using text tokens as the input for the next step. When the user barges in, the LLM autonomously decides when to stop speaking and reverts to a state of latent reasoning.
Method
FD
LlamaQ
WebQ
TriQA
SDQA
AlpacaE
ComE
OBQA
MMSU
Moshi ( Défossez et al., 2024 )
✓
54.5
22.1
16.7
15.6
2.01
1.60
25.9
24.0
Freeze-Omni ( Wang et al., 2024b )
✓
56.2
27.9
28.5
53.5
4.03
3.46
31.0
28.1
SALMONN-omni ( Yu et al., 2025 )
✓
73.6
43.7
56.0
-
3.22
-
-
30.0
SALM-Duplex ( Hu et al., 2025 )
✓
51.3
25.0
16.9
26.0
2.99
2.50
39.6
26.3
GLM-4-Voice ( Zeng et al., 2024 )
✗
65.7
37.0
47.5
37.0
3.97
3.42
53.4
39.8
Qwen2-Audio ( Chu et al., 2024 )
✗
69.7
45.2
40.3
35.7
3.74
3.43
49.5
35.7
Table 1: Accuracy (%) or GPT-Score of different methods on QA benchmarks. Results of baseline systems are taken from ( Yu et al., 2025 ; Chen et al., 2024 ) . We use “ thk ” to denote the proposed latent reasoning. “FD”, ”LlamaQ”, “WebQ”, “TriQA”, “OBQA”, “AlpacaE” and “ComE” stand for “if Full-duplex”, Llama Questions, WebQuestions, TriviaQA, OpenbookQA and AlpacaEval, CommonEval respectively.
Method
E2E
Turn Taking
Barge-in
MOS
Latency ( ↓ )
Latency ( ↓ )
Success rate ( ↑ )
Freeze-Omni ( Wang et al., 2024b )
✗
1.17
1.20
79.5
4.3
dGSLM ( Nguyen et al., 2023 )
✓
0.57
0.86
85.0
2.2
Moshi ( Défossez et al., 2024 )
✓
-
0.81
55.1
3.9
ORISE ( Chen et al., 2025 )
✓
0.43
0.61
96.8
4.2
FLAIR w/o thk
✓
0.33
0.49
100
4.3
Table 2: Evaluation of conversational behaviors of different methods on the Impatient dataset proposed in ( Chen et al., 2025 ) , in terms of turn-taking latency (s), barge-in latency (s), and barge-in success rate (%). We use “ thk ” to denote the proposed latent reasoning.
Method
Turn-taking
Barge-in
TOR ( ↑ )
Latency ( ↓ )
TOR ( ↑ )
GPT-4o ( ↑ )
Latency ( ↓ )
Freeze-Omni ( Wang et al., 2024b )
33.6
0.95
86.7
3.62
1.41
dGSLM ( Nguyen et al., 2023 )
97.5
0.35
91.7
0.20
2.53
Moshi ( Défossez et al., 2024 )
94.1
0.27
100
0.77
0.26
Gemini Live 1
65.5
1.30
89.1
3.38
1.18
FLAIR w/o thk
94.1
0.37
89.0
4.08
0.35
Table 3: Turn-taking and barge-in performance on Full-Duplex-Bench. Results of baseline systems are taken from ( Lin et al., 2025 ) . We use “thk” to denote the proposed latent reasoning.
Figure 3: The distribution of the input audio, target text, and latent reasoning embeddings. Specifically, the latent reasoning embeddings act as a bridge that connects the input audio with the target text.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Method
LlamaQ
WebQ
OBQA
MMSU
PT Model
63.6
40.4
61.3
49.5
Appendix
Table 4: Accuracy (%) of Pretraining model on QA benchmarks. “LlamaQ”, “WebQ”, and “OBQA” stand for Llama Questions, WebQuestions, and OpenbookQA respectively.
User query
Can you tell me a very easy way to clean a shower head?
Response by PT model
A easy and effective method you can try at home is using common household items.
Response by FLAIR
Sure! A simple way to clean a shower head is to remove it and soak it in white vinegar for about an hour. This helps dissolve any mineral deposits or buildup. After soaking, rinse it thoroughly with water.
Response by LLM
Certainly! Cleaning a shower head is a straightforward process that can help improve water flow and remove mineral buildup. Here’s a simple method:\n\nMaterials Needed:\nWhite vinegar\nPlastic bag\nRubber bands or clips\nTowel\n\nSteps:\nRemove the Shower Head:\n\nTurn off the water supply to the shower.\nUnscrew the shower head from the pipe. You may need a wrench or pliers for this step.\n\nSoak the Shower Head:\n\nPlace the shower head in a plastic bag.\nPour enough white vinegar into the bag to cover the shower head.\nSecure the bag around the shower head with rubber bands or clips to keep the vinegar in contact with the shower head.\n\nLet It Soak:\n\nLeave the shower head in the vinegar for at least 8 hours, or overnight for best results. The longer it soaks, the more effectively it will remove mineral buildup.\n\nRinse and Reattach:\n\nAfter the soaking period, remove the shower head from the bag.\nRinse the shower head thoroughly with water to remove any remaining vinegar.\nReattach the shower head to the pipe and turn the water supply back on.\n\nTest the Flow:\n\nTurn on the shower and check the water flow. You should notice an improvement in the water pressure and clarity.\n\nThis method is effective and easy to do, making it a great way to keep your shower head clean and functioning well.
Appendix
Table 5: Case study for PT model, FLAIR, and LLM (Qwen2.5-7B-Ins)