Machine Hearing: Can Audio LLMs Generalize Beyond Human Auditory Perception?
Organizations: Shanghai Artificial Intelligence Laboratory · Harbin Institute of Technology · University of Science and Technology of China · Fudan University ·
Abstract
Multimodal LLMs have made remarkable progress in perceiving the physical world, while their perceptual scope remains largely human-centric, focusing on what humans can directly see or hear. This progress raises the prospect of perception beyond human limits, such as extending hearing to sounds that are inaudible to humans. More broadly, the physical world is full of waves, vibrations, and oscillations, only a small fraction of which are directly audible to humans. This invites the extension of multimodal LLMs toward a broader form of machine hearing, which remains largely underexplored. To investigate this direction, we introduce the Machine Hearing Benchmark (MHB), comprising 6,128 instances across 13 tasks and seven domains. MHB spans highly heterogeneous input regimes, with sampling rates from approximately to Hz, sequence lengths from to samples, and both single- and multi-channel inputs. Evaluations of current open-source and proprietary Audio LLMs (ALLMs) reveal limited zero-shot transfer to machine hearing. Supervised adaptation on machine-hearing data alone does not close this gap, suggesting that the challenge extends beyond data exposure to a systematic mismatch between existing audio front ends and machine-hearing inputs. To address this mismatch, we introduce MachEar, a lightweight encoder extension that complements the pretrained Audio LLM for machine-hearing inputs. The results show that MachEar equips a pretrained Audio LLM with stronger machine-hearing capabilities while largely preserving its existing audio understanding performance.
Figures & tables
| Access | Model | Astronomy | Earth Science | Healthcare | ||||
| GWOSC | LEAVES | Iquique | Llaima | Yaseen | CirCor-loc | |||
| – | Random Guess | 49.95 | 8.94 | 40.66 | 22.64 | 19.79 | 24.92 | |
| Open-source | Qwen2.5-Omni-3B | 34.36 | 1.87 | 18.56 | 5.91 | 7.87 | 22.30 | |
| Qwen2.5-Omni-7B | 38.11 | 0.00 | 24.78 | 3.75 | 12.06 | 18.65 | ||
| Audio-Flamingo-3-7B | 49.83 | 0.22 | 34.77 | 3.75 | 7.74 | 17.83 | ||
| Proprietary | Gemini-3-Flash | 35.62 | 0.00 | 31.88 | 23.16 | 16.05 | 15.68 | |
| Model | Astronomy | Earth Science | Healthcare | ||||
| GWOSC | LEAVES | Iquique | Llaima | Yaseen | CirCor-loc | ||
| Time-MoE | 59.18 | 67.07 | 46.82 | 94.11 | 93.09 | 27.87 | |
| SpectrumFM | 52.86 | 61.70 | 48.85 | 79.48 | 76.05 | 26.52 | |
| SeisLM | 48.75 | 58.42 | 60.42 | 93.33 | 93.08 | 29.12 | |
| FISHER | 68.75 | 0.00 | 0.00 | 0.00 | 94.20 | 38.57 | |
| SPEAR | 78.91 | 44.82 | 52.28 | 95.51 | 93.88 | 51.07 | |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Domain | Name | # Instances | Sample | Sampling | Channel | Data Source |
| Length Range | Rate (Hz) | Range | ||||
| Astronomy | GWOSC | 512 | 6.4e4 | 1.6e4 | 1 | SciTS ( Wu et al., 2026 ) |
| LEAVES | 1024 | 23 1.3e3 | 1.1e-5 | 1 | LEAVES ( Fei et al., 2024 ) | |
| Earth Science | Iquique | 502 | 1.4e4 1.7e4 | 1.0e2 | 3 | Iquique ( Woollam et al., 2019 ) |
| Llaima | 358 | 6.0e3 | 1.0e2 | 1 | Llaima ( Canário et al., 2020 ) | |
| Healthcare | Yaseen | 200 | 1.3e4 2.7e4 | 8.0e3 | 1 | Yaseen ( Yaseen et al., 2018 ) |
| Access | Model | Astronomy | Earth Science | Healthcare | ||||
| GWOSC | LEAVES | Iquique | Llaima | Yaseen | CirCor-loc | |||
| Open-source | Qwen2.5-Omni-3B | 100.00 | 7.13 | 100.00 | 100.00 | 100.00 | 80.92 | |
| Qwen2.5-Omni-7B | 100.00 | 7.13 | 73.71 | 100.00 | 91.50 | 83.79 | ||
| Audio-Flamingo-3-7B | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | 100.00 | ||
| Proprietary | Gemini-3-Flash | 100.00 | 5.47 | 100.00 | 96.37 | 100.00 | 100.00 | |
| GPT-Audio-Mini | 4.88 | 0.00 | 40.24 | 33.52 | 100.00 | 90.82 | ||
| Failure Type | Representative Case |
| Unrecognized Input | Model: Gemini-3-Flash Task: RadioML Output: “The input is missing. Please provide the in-phase and quadrature components.” Why invalid: Treats the provided signal as missing and requests data again. |
| Short Input | Model: Doubao-2.0-Mini Task: LEAVES Error: “The request failed because the duration of the input audio (4ms) is shorter than the limit (50ms).” Why invalid: The input is shorter than the API’s minimum duration of 50 ms. |
| Empty Response | Model: Qwen2.5-Omni-3B Task: CirCor-loc Output: [empty output] Why invalid: No usable textual response is returned. |
| Label/Format Error | Model: Audio-Flamingo-3-7B Task: DeepShip Output: Tank Ground truth: Tanker Why invalid: Tank is not among the predefined labels. |
Audio-Language Models
Audio-language models connect audio and natural language for understanding, retrieval, dialogue, speech generation, and music or sound analysis.
See all 324 papers