In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

Xulin Fan; Vishal Sunder; Samuel Thomas; Mark Hasegawa-Johnson; Brian Kingsbury; George Saon

eess.ASApr 14, 2026

In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions

Authors: Xulin Fan, Vishal Sunder, Samuel Thomas, Mark Hasegawa-Johnson, Brian Kingsbury, George Saon

Organizations: University of Illinois Urbana-Champaign · IBM Research

Abstract

Recent advances in speech-aware language models have coupled strong acoustic encoders with large language models, enabling systems that move beyond transcription to produce richer outputs. Among these, word-level timestamp prediction is critical for applications such as captioning, media search, and multimodal synchronization, yet it is often handled by external alignment tools. In this work, we extend an existing speech-aware language model to predict timestamps directly alongside transcripts. We introduce a set of novel lightweight training strategies that improve alignment robustness while preserving recognition quality. Experiments across multiple datasets show that these strategies not only enhance timestamp accuracy, but also yield gains in overall ASR performance. Together, they demonstrate an efficient and unified approach to speech recognition with precise timestamp prediction.

Figures & tables

Figure 1: (a) Overall architecture of the proposed In-Sync framework for joint transcription and timestamp prediction using Granite-speech. (b) Diagram of timestamp embedding space regularization with $N{=}5$ , where the similarity matrix $S$ is encouraged to match a structured target $G$ . (c) Illustration of reduced teacher forcing during autoregressive generation, where a timestamp token is randomly corrupted to encourage model robustness.

External Baselines
Model	Metric	AVG	LS-C ∗	LS-O ∗	CV ∗	AMI ∗	VOXP ∗	MLS ∗†	TIMIT †	BUCK †
CrisperWhisper [ 29 ] 1 1 1 23 CV, 36 VOX, 3 MLS samples excluded from CrisperWhisper evaluation due to decoding error during inference.	WER	5.60	1.71	3.72	7.76	8.43	6.03	5.26	1.29	10.63
	AAS	53.65	30.20	33.84	119.37	64.80	54.93	48.36	34.30	43.41
Qwen-Audio [ 5 ]	WER	10.27	2.19	4.59	10.70 2 2 2 Undergoes a rule-based post-processing for correcting a fixed error pattern before evaluation.	31.82	7.22	7.54	5.96	12.15
	AAS	–	16.67	18.87	–	55.64	–	31.87	22.25	23.17
	MAL	–	0.57	0.54	–	18.64	–	4.51	0.06	0

Table 1: Comparison across datasets for automatic speech recognition (ASR) and speech recognition with timestamps (SRWT). Word error rate (WER

\downarrow

) in percentage is used for evaluating ASR task performance, while accumulated averaging shift (AAS

\downarrow

) [ 22 ] in miliseconds (ms) and percentage of malformed samples (MAL

\downarrow

) measure SRWT task performance in terms of timestamp accuracy and stability to form a correct interleaved sequence. A dash (–) indicates that the model failed to follow the task prompt and instead hallucinated or performed a different task on that dataset. Datasets marked with ∗ use timestamps obtained via the Montreal Forced Aligner, with samples failing alignment excluded. The TIMIT and Buckeye datasets have manual timestamp annotations. Datasets marked with † are evaluated in a zero-shot setting for ASR-only baseline and all our In-Sync variants. The leftmost column with “AVG” denotes the average metrics across all datasets.

Automatic Speech Recognition

Automatic speech recognition converts spoken language into text across speakers, languages, domains, and recording conditions.

See all 480 papers

Relative Time Intervals Representation for Word-level Timestamping with Masked Training
Aug 25, 2026Quanwei Tang, Zhiyu Tang, Xu Li +3Speech Language Models Automatic Speech Recognition
Soochow University · University of Queenland · AISpeech Ltd +1
TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models
Aug 30, 2026Apoorva Kulkarni, Kaousheik Jayakumar, Sreyan Ghosh +3Audio Understanding Audio-Language Models
University of Maryland, College Park, USA · CNH Industrial India
Listen First, Then Answer: Timestamp-Grounded Speech Reasoning
Mar 19, 2026Jihoon Jeong, Pooneh Mousavi, Mirco Ravanelli +1Audio-Language Models Temporal Audio Grounding
Mila-Quebec AI Institute · Universit´e Laval · Concordia University

Aug 25, 2026J/K move · Enter open · S save

Relative Time Intervals Representation for Word-level Timestamping with Masked Training

Quanwei Tang, Zhiyu Tang, Xu Li, Dong Zhang+2

Soochow University · University of Queenland · AISpeech Ltd+1

Speech Language Models Automatic Speech Recognition

Open paper AskPDF arXiv

Although Speech Large Language Models (SpeechLLMs) excel at speech understanding and generation, their capacity for fine-grained, temporally aligned outputs remains underexplored. Our work addresses this gap by enabling SpeechLLMs to jointly model speech content and temporal structure, effectively transforming them from content understanding machines" into temporal-aware content understanding machines". Specifically, we replace traditional absolute timestamps with relative timestamps, achieving a more compact vocabulary and stronger generalization capabilities. To efficiently infuse timestamp prediction ability into pre-trained large language models, we introduce a hybrid fine-tuning strategy: full-parameter fine-tuning of the timestamp-augmented embedding layer and language model head, combined with LoRA fine-tuning of the decoder layers. Moreover, we design a masked timestamp training objective, preventing the model from over-relying on ground-truth timestamps, and thereby enhancing robustness against noisy real-world annotations. Extensive experiments demonstrate that our approach achieves significant improvements in timestamp prediction accuracy while maintaining strong speech transcription performance.

Abstract

Automatic Speech Recognition

Explore similar work

Relative Time Intervals Representation for Word-level Timestamping with Masked Training

TEMPO: Temporally-grounded Multi-task Post-training for Large Audio-Language Models

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning