Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents

Zhenyu Zhang; Kewei Fu; Yongqi Tong; Chenzhi Zheng; Wei Wu; Jiang-Ming Yang; Jianshe Li

cs.CLOct 19, 2026

Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents

Authors: Zhenyu Zhang, Kewei Fu, Yongqi Tong, Chenzhi Zheng, Wei Wu, Jiang-Ming Yang, Jianshe Li

Organizations: Ant International

Abstract

Tool-using voice agents often leave users waiting after speech ends, even when actionable information arrives earlier in the request. Speculative approaches begin work during speech, but chunk-bounded or interrupt-on-update generation can truncate a tool call before its arguments are complete. This couples computation time to speech updates, limiting useful work completed in advance. We introduce Heartbeat Thinking, which handles updated input through new Thinker invocations while allowing ongoing invocations and dispatched tools to finish. Periodic invocations read the latest partial transcript and a shared record of completed reasoning and tool results. These results help resolve missing arguments and initiate dependent calls, allowing preparation to progress across speech updates. Across SpeechFCEval, FDBench v3, and AudioCRAG, Heartbeat improves aggregate speculative tool coverage or retrieval reuse over a SHANKS-style windowed baseline, with benchmark-dependent effects on final task quality. On FDBench v3, it increases tool coverage by 16 percentage points and reduces mean final-response latency by 36.4%. The results show that heartbeat thinking paradigm preserving ongoing computation during speech can turn listening time into reusable preparation for faster and more capable voice-agent responses.

Figures & tables

Figure 1: Reasoning and tool use during speech. (a) Cascaded baselines defer reasoning and tool calls until speech ends. (b) SHANKS may truncate tool-call generation at chunk boundaries ( Chiang et al., 2026 ) . (c) Heartbeat overlaps Thinker invocations and tools across speech updates; the Responder reuses completed preparation without waiting for pending jobs.

Figure 2: Heartbeat Thinking architecture (top) and execution timeline (bottom). Thinker invocations and tools overlap during listening. At the VAD-triggered handoff

c

, unfinished speculative jobs are cancelled; the Responder uses completed preparation available at

c

without waiting for pending work.

Framework	SpeechFCEval		FDBench v3			AudioCRAG
Framework	Acc.	STC	Quality	Pass@1	STC	Correct	STC
Speech-to-Speech	58.66	0.00	84.0	75.0	0.00	56.48	0.00
Pipecat	56.83	0.00	78.0	63.0	0.00	56.94	0.00
LiveKit	52.85	0.00	77.6	67.0	0.00	59.45	0.00
Window Thinking (SHANKS)	67.95	33.13	72.0	57.0	34.67	57.24	20.05
Heartbeat	66.85	43.69	77.0	65.0	50.67	55.49	62.42

Table 1: Main results (%; higher is better). Speculative systems use Qwen3.6-35B-A3B as Thinker; all systems use DeepSeek-V4-Flash as Responder. Further task and model results are in Appendix C .

Framework	Avg. Acc. $\uparrow$	STC $\uparrow$
Responder: DeepSeek-V4-Flash
Speech-to-Speech	58.66	0.00
Pipecat	56.83	0.00
LiveKit	52.85	0.00
Window Thinking (SHANKS)	67.95	33.13
Heartbeat	66.85	43.69

Table 2: SpeechFCEval accuracy and STC across Responders (%), with Qwen3.6-35B-A3B as Thinker.

Framework	Quality $\uparrow$	Pass@1 $\uparrow$	STC $\uparrow$
Responder: DeepSeek-V4-Flash
Speech-to-Speech	84.0	75.0	0.0
Pipecat	78.0	63.0	0.0
LiveKit	77.6	67.0	0.0
Window Thinking (SHANKS)	72.0	57.0	34.67
Heartbeat	77.0	65.0	50.67

Table 3: FDBench v3 results across Responders (%), with Qwen3.6-35B-A3B as Thinker.

Figure 3: Mean Correct Tool Call Latency on FDBench v3 relative to speech end, measured at initiation of the first tool call matching the final request. Negative values indicate events occurring before speech end. Numerical results are reported in Table 5 .

Figure 4: Mean Final Response Latency on FDBench v3 relative to speech end, measured at the onset of the final spoken response. Details are reported in Table 5 .

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value
ASR	Qwen3-ASR-1.7B
TTS	Qwen3-TTS-12Hz-1.7B-Base
ASR update block	500 ms
VAD minimum speech	200 ms
VAD minimum silence	800 ms
VAD padding	500 ms

Appendix

Table 4: Speech pipeline and heartbeat scheduling settings for the FDBench v3 latency evaluation.

System	Correct Tool Call	Final Response
Heartbeat	$-4.37$	4.39
Window Thinking (SHANKS)	$-2.74$	6.90
Speech-to-Speech	$2.72$	8.39
Pipecat	$2.45$	8.08
LiveKit	$3.52$	8.52

Appendix

Table 5: Mean event latencies on FDBench v3 (seconds), measured relative to speech end. Negative values indicate events occurring before speech end.

Table 6: SpeechFCEval results (%) with DeepSeek-V4-Flash as the Responder. (a) Function-calling accuracy and (b) speculative tool coverage (STC).

Framework	Thinker	Quality	Pass@1	STC
Cascaded
Speech-to-Speech	–	84.0	75.0	0.0
Pipecat	–	78.0	63.0	0.0
LiveKit	–	77.6	67.0	0.0
Window Thinking (SHANKS)	Qwen3.6-35B-A3B	72.0	57.0	34.67
	Qwen3.6-35B-A3B	77.0	65.0	50.67

Appendix

Table 7: FDBench v3 task performance and speculative tool coverage (STC; %) with DeepSeek-V4-Flash as the Responder.

Framework	Thinker	Correct $\uparrow$	Missing	Hallucination $\downarrow$	STC $\uparrow$
Cascaded
Speech-to-Speech	–	56.48	11.97	31.55	0.00
Pipecat	–	56.94	11.97	31.10	0.00
LiveKit	–	59.45	12.88	27.67	0.00
Window Thinking (SHANKS)	Qwen3.6-35B-A3B	57.24	14.33	28.43	20.05
	Qwen3.6-35B-A3B	55.49	16.84	27.67	62.42

Appendix

Table 8: AudioCRAG answer outcomes and retrieval reuse (%) with DeepSeek-V4-Flash as the Responder. Here, STC measures the share of samples with completed speculative results included in the Responder’s context and no additional retrieval by the Responder.

Framework	Correct $\uparrow$	Miss.	Hallu. $\downarrow$	STC $\uparrow$
Responder: DeepSeek-V4-Flash
Speech-to-Speech	56.48	11.97	31.55	0.00
Pipecat	56.94	11.97	31.10	0.00
LiveKit	59.45	12.88	27.67	0.00
Window Thinking (SHANKS)	57.24	14.33	28.43	20.05
Heartbeat	55.49	16.84	27.67	62.42

Appendix

Table 9: AudioCRAG answer outcomes and retrieval reuse across Responder models (%), with Qwen3.6-35B-A3B as the Thinker. The highest reuse rate and correct-answer rate occur with different Responders.

Figure 5: Speculative tool calls during listening on FDBench v3. Red markers indicate the last words available to the Thinker when each invocation began; callouts give tool-request initiation times and arguments. Times are relative to audio onset. Word alignment is approximate, and ASR text is normalized for display.

Input group	Success rate
Early information	73.3%
Late information	49.2%
Explicit revision	70.6%

Appendix

Table 10: Correct tool-call initiation before speech end on FDBench v3, grouped by information timing and explicit intent revision. For revision cases, the rate measures corrected calls following an outdated request. Group definitions are given in Appendix D .

Tool-Using Agents

Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.

See all 277 papers

Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution
Oct 6, 2026Kyudan Jung, Hyunsin Park, Yoonhyung Lee +5Conversational Agents
Qualcomm AI Research · KAIST AI
StepAudio 3 Realtime Technical Report
Sep 12, 2026Bin Lin, Bo Zhao, Boyang Zhang +87Audio-Language Models Full-Duplex Spoken Dialogue Systems
Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models
Aug 3, 2026Yuwen Wang, Tian-Hao Zhang, Minghao Cai +7Tool-Augmented Language Model Agents Agent Evaluation
University of Science and Technology Beijing, China · Li Auto Inc., Beijing, China

Oct 6, 2026J/K move · Enter open · S save

Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution

Kyudan Jung, Hyunsin Park, Yoonhyung Lee, Jinhwan Park+4

Qualcomm AI Research · KAIST AI

Conversational Agents

Open paper AskPDF arXiv

Tool-augmented speech assistants typically serialize automatic speech recognition, large language model inference, and external tool execution. As a result, tool latency is incurred only after the user has finished speaking and the LLM has identified the required tool calls. We present speculative tool execution for on-device cascaded voice agents, which predicts tool requests from partial ASR hypotheses and initiates tool execution while speech is still being received, thereby reducing end-to-end response latency. Our approach introduces a Predictor module that anticipates tool calls during speech recognition, executes them speculatively, and caches the results. The cached outputs are then injected into the LLM prompt, enabling faster responses. Additionally, to mitigate errors caused by user self-corrections during speech, we employ a rule-based validation mechanism that selectively injects only valid cached results. As a final safeguard, the LLM retains the ability to issue tool calls directly, ensuring that the latency of our framework is upper-bounded by the baseline serial execution pipeline in the worst case. We evaluate our method using live measurements from a fully implemented Android voice assistant. Our approach reduces the median time-to-first-audio from 5.79,s to 4.60,s and decreases the standard deviation from 3.49,s to 2.81,s, resulting in more predictable response latency.

Abstract

Tool-Using Agents

Explore similar work

Hiding Tool Latency in On-Device Cascaded Voice Agent through Speculative Execution

StepAudio 3 Realtime Technical Report

Hear, Invoke, and Understand: A Skill-Calling Multimodal Agent for Large Audio Language Models