Framing Spatial Hearing within the Free Energy Principle: A Hierarchical Model of Perceptual Learning
Authors: Roberto Barumerli, Nils Meyer-Kahlen, Lorenzo Picinali
Organizations: Dyson School of Design Engineering, Imperial College London, UK · Dept. of Information and Communications Engineering, Aalto University, Finland
Spatial hearing relies on dynamic internal representations to decode spatial information from ever-changing acoustic conditions. A fundamental example is sound localisation, where listeners infer the direction of an invisible sound. Bayesian inference provides a computational account of such a localisation process; however, modelling efforts are largely limited to static listening scenarios. Instead, the free energy principle (FEP) describes perception, learning, and action as joint minimisation of variational free energy (VFE), providing a principled framework to account for dynamic conditions. Here, we take a first step toward applying the FEP to spatial hearing. We characterise how listeners' beliefs evolve across trials, in which they estimated sound direction and received subsequent visual feedback on each trial. Listeners are described by a hierarchical generative model where visual feedback drives gradient descent on VFE, updating beliefs over three independent quantities: lateral and polar localisation precisions and hemifield disambiguation. The model captures individual learning trajectories and reveals components with different dynamics: lateral precision improves, as does hemifield disambiguation, consistent with increased sensitivity of the auditory pathway in decoding vertical direction. Notably, polar precision also increases, most for listeners with the lowest initial precision, suggesting that the observed vertical improvement is driven by both hemifield disambiguation and polar precision. This work establishes spatial hearing as a tractable domain for FEP and motivates future closed-loop experiments with full action-perception cycles.
Figures & tables
Figure 1: Interaural-polar coordinate system [ 14 ] . The source direction is defined by φ=(α,β) : lateral angle α identifies the interaural axis and the cone of confusion (green circle); polar angle β defines elevation within that cone. Reference points: front (0∘,0∘) , back (0∘,180∘) , sides (±90∘,0∘) .
Figure 2: Architecture of the hierarchical FEP agent. Level 1 encodes the listener’s generative model of auditory space as a von Mises mixture (polar) and a single von Mises (lateral). Level 2 revises the agent’s beliefs θ=(κL,κP,w) via natural-gradient descent on variational free energy F , using visual feedback φt=(αt,βt) as a high-precision teaching signal. Initial beliefs are estimated from baseline block available from the original experiment (naive localisation without feedback [ 10 ] ) by maximum likelihood. Learning rates (ηL,ηP,ηw) are fitted per subject.
Figure 3: Top: Group-average behavioural trajectories, manual condition ( n=5 ). Solid: empirical (mean ± 1 standard error (SE) ribbon); dashed: FEP model. Left to right: lateral dispersion (standard deviation (SD) of lateral error, ∘ ); polar dispersion (SD of polar error excluding hemifield errors, cos2αt -weighted, ∘ ); hemifield error rate (proportion of responses deviating more than 90∘ from the target, cos2αt -weighted, %). The model captures the direction and shape of all three metrics. Bottom: Individual parameter trajectories, manual group ( n=5 ), truncated to 750 trials (LOESS-smoothed, span = 0.5; thin coloured lines: individual subjects; thick black line: mean ± 1 SE ribbon).
Held-out comparison
cos2 robustness
Subject
κP,0
ΔLLηP=0
ΔLLηw=0
ηP (cos 2 )
ηP (no cos 2 )
ΔκP
NH12
7.5
−5.7
−1.1
0.020 [0.015, 0.033]
0.005 [0.002, 0.010]
+1.9
NH15
2.4
−54.2
−6.5
0.015 [0.013, 0.026]
0.016 [0.013, 0.025]
+2.8
NH16
4.3
−39.0
−39.0
0.008 [0.006, 0.012]
0.006 [0.005, 0.010]
+1.6
NH17
4.3
0.0
−30.2
0.002 [0, 0.012]
0.006 [0, 0.016]
+0.6
NH18
0.9
−68.2
−5.0
0.062 [0.038, 0.084]
0.043 [0.034, 0.079]
+2.4
Table 1: Left: held-out comparison, log-likelihood differences in nats relative to the full model ( ΔLL<0 : reduced model predicts worse). Right: cos2 robustness, polar-precision learning rate ηP with bootstrap 95% interval, with and without the cos2(α) scaling, and ΔκP , the change of κP over the feedback block without scaling.
Auditory models are central tools for studying spatial hearing, yet their validation typically relies on heuristic performance metrics rather than principled statistical methods. We present two contributions building on a Bayesian sound localisation model that jointly infers sound direction from noisy perceptual features and individual head-related transfer functions (HRTFs). First, we derive an explicit likelihood function and validate it through parameter recovery on simulated data and fitting to behavioural responses from 33 participants, demonstrating that the framework reliably identifies individual sensorimotor and spectral parameters. Second, we use this framework to compare four HRTF template interpolation methods, showing that full-sphere spatial coverage and high-frequency spectral fidelity are the primary determinants of template quality, while the specific interpolation algorithm is secondary. Together, these results show that standard model-based statistical methods can address both fundamental questions in spatial hearing and applied problems such as perceptual HRTF evaluation. An open-source Python implementation is released alongside this work.
Dyson School of Design Engineering, Imperial College London, London, United Kingdom · Audio Communication Group, Technische Universität Berlin, Germany · Department of Industrial Systems Technology and Management, University of Padova, Vicenza, Italy
We present BiEAR, a human auditory-inspired adaptive binaural front-end for multi-speaker localisation and distance estimation. Inspired by medial olivocochlear (MOC) feedback in human hearing, BiEAR uses a neural controller to adaptively adjust the frequency selectivity of a binaural auditory filterbank during inference. This yields time-frequency adaptive representations for ears, enabling the model to respond to changing acoustic conditions. We evaluate BiEAR on multi-speaker localisation and distance estimation in anechoic and real-room environments. Results show that the adaptive front-end improves localisation accuracy and robustness to unseen speakers and rooms compared with commonly used fixed binaural front-ends. Visualisation and analysis of learned filter adaptations show that BiEAR emphasises informative frequency bands over time. These findings suggest that adaptive, biologically inspired binaural front-ends can improve machine hearing robustness in complex acoustic scenes.
The University of New South Wales, Sydney, Australia · Tongyi Speech Lab, Alibaba Group, Hangzhou, China · School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen, China
Embodied, ego-centric intelligence fundamentally requires the ability to comprehend spatial audio within complex environments. While large audio-language models excel at mono-channel reasoning, they lack spatial awareness, discarding critical spatial cues that enable sound localization and that can improve the disentanglement of overlapping sound sources. To address this, we present SEA-LM, a Spatial Audio Understanding model. First, we introduce FOACODER, a layout-flexible spatial audio encoder trained on source localization and ego-centric voice activity detection objectives to encode First Order Ambisonics derived from variable-count, variable-position smart-glasses arrays via beamforming. We train a Multimodal Large Language Model (MLLM) to understand these spatial audio embeddings through a two-stage curriculum spanning six tasks, including sound localization and spatially selective transcription in settings with multiple speakers and overlapping sounds. To prevent the transcription outputs from dominating the next token prediction loss and overwhelming the direction predictions, we introduce a Spatio-temporal Weighted Cross-Entropy Loss. On our evaluation set, SEA-LM achieves lower azimuth and elevation MAE, higher temporal IoU, lower external-source hallucination and missing-source rates, and lower WER on most transcription tasks than compared baselines, while remaining robust across 1,211 smart-glasses array configurations with 4 to 9 microphones.
Sonal Kumar, Sinan Hersek, Artem Dementyev +6
University of Maryland, College Park · Google · Google DeepMind