Interpretable Perception and Reasoning for Audiovisual Geolocation

Yiyang Su; Xiaoming Liu

cs.CVMar 5, 2026

Interpretable Perception and Reasoning for Audiovisual Geolocation

Authors: Yiyang Su, Xiaoming Liu

Organizations: Michigan State University, East Lansing, MI 48824, USA

Abstract

While recent advances in Multimodal Large Language Models (MLLMs) have improved image-based localization, precise global geolocation remains a formidable challenge due to the inherent ambiguity of visual landscapes and the largely untapped potential of auditory cues. In this paper, we introduce Audiovisual Geolocation, a framework designed to resolve geographic ambiguity through interpretable perception and reasoning. We present AVG, a high-quality global-scale video benchmark for geolocation, comprising 20,000 curated clips across 1,000 distinct locations. To address the complexity of audiovisual geolocation, we propose a three-stage framework: (1) a Perception stage that utilizes a mixture-autoregressive sparse autoencoder to decompose noisy audio into semantically grounded "acoustic atoms"; (2) a Multimodal Reasoning stage that employs an MLLM finetuned via Group Relative Policy Optimization (GRPO) to synthesize these atoms with visual features; and (3) a Precision Prediction stage using Riemannian Flow Matching on the $S^2$ manifold. Our experiments demonstrate that our framework significantly outperforms unimodal baselines. These results entail that interpretable perception of the soundscape provides a critical, orthogonal signal that, when coupled with multimodal reasoning, enables high-precision global localization.

Figures & tables

Figure 1: In the visual-only approach, ambiguous features like trees and bridges can lead to multiple location candidates. In the audio-only approach, overlapping urban sounds create complex signals that are hard to decipher. Combining modality cues, our proposed framework disambiguates candidates to pinpoint the correct location.

Figure 2: We process audiovisual input in three steps. (1) Perception: visual and audio encoders that extract interpretable elements from foundation models. (2) Reasoning: an MLLM, fine-tuned via GRPO, analyzes these attributes to generate geographically-rich embeddings. (3) Prediction: a Riemann Flow Matching model generate a probability density function on the Earth’s surface conditioned on the reasoning output.

Figure 3: (a) We randomly sample clips from AudioSet to generate synthetic mixtures by a weighted sum, where the weights are randomly assigned but monotonically decreasing gains ( $g_{1}>g_{2}>\dots$ ). (b) We employ an autoregressive pipeline to iteratively decompose the audio. In each iteration, we select the kernel from the dictionary with the highest activation and subtract the reconstructed audio from the mixture. The cross-entropy loss ensures the correct semantic labels and reconstruction loss minimizes the difference between the reconstructed audio and the original audio.

Method	\Block 1-1 $\uparrow$ City 25km	\Block 1-1 $\uparrow$ Region 200km	\Block 1-1 $\uparrow$ Country 750km	\Block 1-1 $\uparrow$ Continent 2500km
\Block [fill=gray!20]1-5Visual-only
GeoCLIP [ 32 ]	6.8	11.0	20.8	32.7
Hybrid [ 1 ]	5.7	9.8	18.7	25.6
RFM $\mathcal{S}_{2}$ [ 8 ]	6.2	10.9	19.6	31.0
GLOBE [ 20 ]	6.8	10.6	20.5	32.1
\Block [fill=gray!20]1-5Audio-only

Table 1: Comparison of geolocation performance on the AVG benchmark.

Method	\Block 1-1 $\downarrow$ Median Error (km)	\Block 1-1 $\uparrow$ City 25km	\Block 1-1 $\uparrow$ Region 200km	\Block 1-1 $\uparrow$ Country 750km	\Block 1-1 $\uparrow$ Continent 2500km
GeoCLAP [ 17 ]	6856	00.2	01.3	07.0	24.9
TaxaBind [ 27 ]	4944	00.4	02.2	11.9	35.3
RFM $\mathcal{S}_{2}$ [ 8 ]	5798	01.4	05.5	16.8	40.4
Ours	1355	04.2	13.8	34.4	64.6

Table 2: Audio-only geolocation performance on the iNatSounds [ 4 ] dataset.

Method	\Block 1-1 $\downarrow$ NLL	\Block 1-1 $\uparrow$ Precision	\Block 1-1 $\uparrow$ Recall	\Block 1-1 $\uparrow$ Density	\Block 1-1 $\uparrow$ Coverage
RFM $\mathcal{S}_{2}$ [ 8 ]	-0.80	0.88	0.93	0.81	0.52
Ours	-0.96	0.88	0.94	0.83	0.57

Table 3: Probabilistic audio geolocation performance on the iNatSounds [ 4 ] dataset.

\Block 2-1ID	\Block 1-3Stages			\Block 1-3Rewards			\Block 2-1 $\uparrow$ City 25km	\Block 2-1 $\uparrow$ Region 200km	\Block 2-1 $\uparrow$ Country 750km	\Block 2-1 $\uparrow$ Continent 2500km
	Pr	P	R	$R_{\text{geo}}$	$R_{\text{align}}$	$R_{\text{calib}}$
0	✓						6.2	10.9	19.6	31.0
1	✓	✓					6.9	11.3	20.3	32.4
2	✓	✓	✓				7.2	11.8	21.4	33.9
4	✓	✓	✓	✓			7.8	11.7	21.8	34.2
5	✓	✓	✓	✓	✓		8.2	12.3	22.5	35.2

Table 4: Ablation study on the AVG test set. P (Perception), R (Reasoning), Pr (Prediction).

Figure 4: Qualitative result of interpretable perception and reasoning.

Audio-Visual Understanding

Audio-visual understanding studies how AI combines sound and visual signals to recognize events, speech, people, and scenes over time.

See all 155 papers

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs
Oct 15, 2026Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung +4Audio-Visual Understanding Multimodal Large Language Models
POSTECH · KAIST · Hankuk University of Foreign Studies +1
Learning to Wander: Improving the Global Image Geolocation Ability of LMMs via Actionable Reasoning
Mar 11, 2026Yushuo Zheng, Huiyu Duan, Zicheng Zhang +2Multimodal Large Language Models VLM Reasoning
Shanghai Jiao Tong University · Shanghai Artificial Intelligence Laboratory
Unlocking Spatial Grounding in Large Audio-Visual Retrieval models
Jun 22, 2026Hugo Malard, Michel Olvera, Sanjeel Parekh +3Multimodal Grounding Audio-Visual Understanding
LTCI, Télécom Paris, Institut Polytechnique de Paris, France · Meta, Reality Labs Research · NVIDIA, France +1

Oct 15, 2026J/K move · Enter open · S save

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

Kim Sung-Bin, Lee Jung-Mok, Jinwoo Jung, Oh Hyun-Bin+3

POSTECH · KAIST · Hankuk University of Foreign Studies+1

Audio-Visual Understanding Multimodal Large Language Models Audio-Visual QA Spatial Audio

Open paper AskPDF arXiv

Audio-visual large language models (AV-LLMs) have made strong progress in multimodal understanding, yet most treat audio as monaural semantic content, limiting their ability to reason about where sounds originate and how they relate to the visual scene. Recent spatial audio-visual studies address parts of this problem, but often focus on specific abilities, such as spatial correspondence or direction and distance reasoning. In this work, we introduce HLR-AVSceneQA, a comprehensive benchmark with a hear-localize-reason hierarchy that evaluates whether models can recognize what is heard, localize sound sources in egocentric and allocentric views, and use these localized sources to reason about how they relate to the visual scene. We further introduce HLR-LLM, which augments a strong audio-visual foundation model with a binaural spatial audio branch alongside its monaural semantic audio pathway. We train HLR-LLM with a curriculum that incorporates final-stage chain-of-thought supervision for multi-step spatial reasoning. Experiments show that existing AV-LLMs remain limited in spatial grounding and relational reasoning, while HLR-LLM substantially improves spatial audio-visual scene understanding without compromising semantic audio-visual perception.

Abstract

Audio-Visual Understanding

Explore similar work

Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-Visual LLMs

Learning to Wander: Improving the Global Image Geolocation Ability of LMMs via Actionable Reasoning

Unlocking Spatial Grounding in Large Audio-Visual Retrieval models