Audio-Language Models

Audio-language models learn to connect speech, music, and environmental sounds with language so they can answer questions, describe events, retrieve audio, and generate speech or music. Common approaches align audio and text with contrastive dual encoders, or connect audio encoders to language models through projectors, discrete tokens, or continuous representations; training often combines large-scale pretraining with instruction tuning and preference or reinforcement learning. Benchmarks increasingly test temporal grounding, overlapping sounds, multilingual speech, and conversational or clinical settings, alongside standard captioning and question-answering tasks. A central challenge is ensuring answers actually depend on acoustic evidence rather than text priors, while improving robustness, faithful reasoning, and efficiency.