RASSG: Repair-Aware Semantic Survival Gating for Streaming Tool Execution

Streaming Retrieval-Augmented Generation (Streaming RAG) hides tool latency by issuing retrieval queries in parallel with the user's still-arriving input, before the utterance is complete. Speculation can only help, though, when the correct query becomes determinable before the user stops speaking or typing -- a property of the query, not the system. We name and measure this property, tool-intent stabilization: the point in the input stream at which a speculative query's retrieval converges on the answer-bearing result. On the CRAG benchmark (1371 validation questions) we (i) characterize how stabilization is distributed across queries; (ii) derive a model-agnostic bound H on the share of tool latency hideable behind the remaining input, given tool latency L and input cadence delta; (iii) validate it against a working streaming pipeline; and (iv) ask which query properties predict early versus late stabilization. Stabilization is typically early: at a realistic operating point a 73.9% streamable fraction of the benchmark admits latency hiding, and H acts as a conservative aggregate floor that realized savings meet or exceed -- though it does not predict savings query by query. Question type yields a statistically significant but small early/late split. The study needs no model training and runs on commodity CPU hardware; a dense-retriever replication confirms the early-stabilization effect is not a BM25 lexical artifact.

Modern voice-agent systems rely on streaming speech recognition models that operate under stringent latency constraints. This study shows that, due to the limited memory constraints of real-time processing, these systems are adversely impacted by conversational phenomena such as long silences and backchannels. While many agentic pipelines mitigate this by resetting state at each turn, this approach discards vital context and impairs performance at turn onsets. We propose two state-management strategies that preserve cross-utterance context to reduce onset errors. In experiments with two state-of-the-art streaming models on two spoken dialogue benchmarks, our best method yields an average of 15-21% relative WER reduction at utterance onsets.

Tool-augmented speech assistants typically serialize automatic speech recognition, large language model inference, and external tool execution. As a result, tool latency is incurred only after the user has finished speaking and the LLM has identified the required tool calls. We present speculative tool execution for on-device cascaded voice agents, which predicts tool requests from partial ASR hypotheses and initiates tool execution while speech is still being received, thereby reducing end-to-end response latency. Our approach introduces a Predictor module that anticipates tool calls during speech recognition, executes them speculatively, and caches the results. The cached outputs are then injected into the LLM prompt, enabling faster responses. Additionally, to mitigate errors caused by user self-corrections during speech, we employ a rule-based validation mechanism that selectively injects only valid cached results. As a final safeguard, the LLM retains the ability to issue tool calls directly, ensuring that the latency of our framework is upper-bounded by the baseline serial execution pipeline in the worst case. We evaluate our method using live measurements from a fully implemented Android voice assistant. Our approach reduces the median time-to-first-audio from 5.79,s to 4.60,s and decreases the standard deviation from 3.49,s to 2.81,s, resulting in more predictable response latency.