EarlyCommit: Calibrated Word-by-Word Commitment for Robots Acting Before You Finish Speaking

Muhammad Haris Khan

cs.ROOct 14, 2026

EarlyCommit: Calibrated Word-by-Word Commitment for Robots Acting Before You Finish Speaking

Authors: Muhammad Haris Khan

Organizations: University of Copenhagen, Copenhagen, Denmark

Abstract

Human listeners can often identify the intended object several words before an instruction ends. Robots, however, are typically designed to wait until the speaker has finished before deciding how to act. We present EarlyCommit, an open-source tool that lets a robot commit to the intended object while the instruction is being spoken. After every recognized word, a fine-tuned text encoder scores each object in the scene. The robot acts when the calibrated probability of the best object reaches a threshold tau set by the user, so 1 - tau is the accepted risk of choosing the wrong object. Otherwise, it waits or asks. The tool includes a streaming speech front end, a scene and instruction generator with exact word-by-word ground truth, and scripts to train, calibrate, and evaluate. On instructions containing words never seen in training, EarlyCommit commits correctly 1.6 to 2 times as often as keyword matching, is right at least 92 percent of the time when it commits at tau = 0.9, and needs 31 to 35 ms per word on consumer GPUs.

Figures & tables

Figure 1. EarlyCommit architecture. Left: the build pipeline. Generated scenes and instructions are labelled after every word by an incremental oracle. Three word pools with no shared words supply unseen language: the train pool for augmentation, the calibration pool for fitting the temperature and the test pool for evaluation. Right: the loop that runs after every recognised word. The judge scores each scene object, the probabilities are calibrated, and the robot commits, waits for the next word or asks which object is meant. The evaluation feeds generated words to the judge directly, without speech recognition. Two-panel diagram. The left panel shows the offline pipeline from scene and instruction generation, through an incremental oracle and three disjoint word pools, to fine-tuning, temperature fitting and evaluation. The right panel shows the runtime loop: speech is transcribed word by word, a judge scores each scene object, probabilities are calibrated, and a commit rule chooses between committing, waiting for the next word, or asking. Arrows connect the calibrated judge from the left panel into the runtime loop.

Figure 2. Correct against wrong commits as

\tau

varies on unseen language. Up and to the left is better. Keyword matching does not depend on

\tau

and appears as one point. Points for

\tau

between 0.85 and 0.95 coincide. Three line plots of correct commits against wrong commits for keyword matching, Laya and Laya with augmentation.

Condition	Keyword	Laya	Laya + aug.
Clean (in-distribution)	100 / 0	100 / 0	99.8 / 0
Unseen disfluencies	100 / 0	99.7 / 0.3	99.3 / 0
Unseen synonyms	47.1 / 0	60.8 / 3.8	76.9 / 3.9
Unseen ASR slips	30.2 / 0	39.3 / 1.6	56.4 / 3.9
All combined	30.2 / 0	41.6 / 3.3	59.0 / 5.0
Confusable ASR slips	74.2 / 5.3	79.6 / 5.7	80.9 / 6.8

Table 1. Correct / wrong commits (% of instructions with a unique target) at

\tau=0.9

. Unseen conditions use test-pool words only.

Temperature fitted on	Training pool	Calibration pool
Fitted temperature $T$	0.61	5.29
Calibration error, clean	0.001	0.009
Calibration error, unseen	0.070	0.026
Correct commits	65.0%	59.0%
Wrong commits	15.2%	5.0%
Accuracy at commit	0.81	0.92

Table 2. Where the temperature is fitted matters. Laya with augmentation,

\tau=0.9

, all combined unseen language unless stated.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 239 papers

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems
Oct 26, 2026Yizhou Peng, Chen Chen, Donghang Wu +2Spoken Dialogue Systems Spoken Language Understanding
Nanyang Technological University, Singapore · AI Singapore, Singapore
Talk2Agent: Benchmarking Voice Interfaces for Text Agents
Sep 30, 2026Terumi Chiba, Guangzhi Sun, Zheqi Yuan +1ASR Evaluation Voice Agent Evaluation
Department of Electronic Engineering, Tsinghua University · University of Cambridge
Premover: Fast Vision-Language-Action Control via Early Execution During Instruction Delivery
May 12, 2026Joonha Park, Jiseung Jeong, Taesik GongVisuomotor Policy Learning AI Control
UNIST, Ulsan, Republic of Korea · The Catholic University of Korea, Bucheon, Republic of Korea

Oct 26, 2026J/K move · Enter open · S save

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

Yizhou Peng, Chen Chen, Donghang Wu, Hexin Liu+1

Nanyang Technological University, Singapore · AI Singapore, Singapore

Spoken Dialogue Systems Spoken Language Understanding Tool Selection for Language Model Agents

Open paper AskPDF arXiv

Latency-sensitive spoken agents must know not only which tool to call, but also when they have heard enough to act. Waiting for the turn to complete delays execution even when the request is already actionable. However, neither existing spoken agents nor spoken dialogue corpora define the earliest point at which a request becomes actionable. We introduce a synthetic spoken-dialogue pipeline that provides earliest-actionable-point supervision, marking when the observed speech grounds a tool and its required arguments. Through controlled task generation, speech synthesis and alignment, and gated annotation, we construct 26,694 dialogues totaling 786 hours across 48 domains, with disjoint training and test argument values. We additionally create an audio rendering of

τ^2

-bench for external evaluation. We train an omni-modal language model on this corpus, enabling causal, chunk-level decisions and partial-transcript generation for downstream call completion. The model achieves tool-state macro-F1 scores of 0.48 and 0.47 on held-out dialogues and the external benchmark, respectively. For correct calls, the estimated mean lead over a turn-end cascade, using the same formatter, is 1.8-2.0 seconds in-domain and 1.1 seconds on

τ^2

-bench. These results demonstrate a data-driven path toward spoken agents that act while listening. We will open-source the pipeline and the training scripts.

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

Heard Enough to Act: Early Tool Calling in Spoken Dialogue Systems

Talk2Agent: Benchmarking Voice Interfaces for Text Agents

Premover: Fast Vision-Language-Action Control via Early Execution During Instruction Delivery