SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

Paweł Pozorski; Jakub Muszyński; Maria Ganzha

cs.SDFeb 25, 2026

SGPA: Spectrogram-Guided Phonetic Alignment for Feasible Shapley Value Explanations in Multimodal Large Language Models

Authors: Paweł Pozorski, Jakub Muszyński, Maria Ganzha

Organizations: Warsaw University of Technology, Warsaw, Poland

Abstract

Explaining the behavior of end-to-end audio language models via Shapley value attribution is intractable under native tokenization: a typical utterance yields over $150$ encoder frames, inflating the coalition space by roughly $10^{42}$ relative to text; individual audio frames lack standalone meaning; and token boundaries that bisect phonetic transitions introduce masking artifacts. We introduce Spectrogram-Guided Phonetic Alignment (SGPA), a four-stage pipeline that combines Connectionist Temporal Classification forced alignment with spectral boundary refinement to produce acoustically stable, word-aligned audio segments. Controlled diagnostics on LFM2-Audio-1.5B with VoiceBench show that SGPA yields a 43 $\times$ reduction in model evaluations. Statistical testing confirms that SGPA significantly alters attribution concentration while preserving the global cumulative profile, establishing it as a feasibility-enabling layer for audio explainability.

Figures & tables

Figure 1: SGPA pipeline: transcript decomposition $\to$ CTC alignment $\to$ spectral boundary refinement $\to$ word-level aggregation. The CTC boundary (dashed) is shifted to a spectrally stable region (solid).

SGPA	Mode	Calls (mean)	Time (s)
✓	SM2S	59.42	67.53
✓	SF2S	59.32	66.08
–	SM2S	2365.76	1696.49
–	SF2S	2552.14	1819.88

Table 1: Effect of SGPA on Neyman SV estimation cost. Mean model calls and wall-clock time per sample over

100

VoiceBench single-sentence prompts.

Figure 2: Distribution of explainable token counts per sample with and without SGPA across modes. SGPA concentrates counts in a narrow regime; native tokenization yields substantially longer and more dispersed sequences.

Mode	Metric	$p$	Cohen’s $d$	Sig.
SM2S	Gini	$<$ 0.01	0.50	✓
	Entropy	$<$ 0.01	$-$ 1.37	✓
	Top-20%	$<$ 0.01	0.86	✓
SF2S	Gini	0.01	0.21	—
	Entropy	$<$ 0.01	$-$ 0.97	✓
	Top-20%	$<$ 0.01	0.72	✓

Table 2: Paired significance tests of per-observation SV summary metrics (SGPA on vs. off ), Bonferroni-corrected

\alpha=0.05

. Entropy is normalized by

\sqrt{n}

Figure 3: Normalized attribution entropy ( $H/\sqrt{n}$ ) by mode with and without SGPA. SGPA widens the interquartile range, reflecting greater variability in concentration across samples.

Figure 4: Position-normalized cumulative SV profile (with derivative) for speech-to-speech mode, illustrating the preserved ”early mass” macro-trend under SGPA.

Shapley Value Attribution

Shapley value attribution fairly assigns a prediction or outcome among contributing features, data points, agents, or modalities.

See all 103 papers

mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models
Apr 21, 2026Jakub Muszyński, Paweł Pozorski, Maria GanzhaMultimodal Large Language Models Shapley Value Attribution
Warsaw University of Technology, Warsaw, Poland
Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis
Apr 21, 2026Paweł Pozorski, Jakub Muszyński, Maria GanzhaAudio-Text Alignment Multimodal Large Language Models
student record book number 327304 · student record book number 327246 · thesis supervisor
What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability
Sep 11, 2026Lucia Cascone, Valeria Fraenza, Michele Nappi +2Multimodal Grounding Audio-Language Models
Department of Computer Science, University of Salerno, Italy

Apr 21, 2026J/K move · Enter open · S save

mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models

Jakub Muszyński, Paweł Pozorski, Maria Ganzha

Warsaw University of Technology, Warsaw, Poland

Multimodal Large Language Models Shapley Value Attribution Multimodal Model Interpretability

Open paper AskPDF arXiv

We introduce mllm-shap, an open-source Python framework designed to extend Shapley Value (SV) explainability from text-only Large Language Models to Multimodal LLMs (MLLMs) processing joint text and audio inputs. While text-based attribution is well-studied, mllm-shap addresses three critical challenges unique to the multimodal regime: (1) Modality-aware coalition masking, which manages the interleaved processing of discrete text tokens and dense audio encoder frames. (2) Multi-turn conversation tracking, utilizing per-token metadata to maintain role and modality context. (3) Phonetic alignment-based token grouping, a novel technique that reduces the coalition space by 10x to 50x, rendering SV estimation computationally feasible for long-form audio. The platform implements five SV estimation strategies, including a Complementary Contributions (CC) estimator with Neyman-optimal allocation that demonstrates superior convergence over standard Monte Carlo baselines. mllm-shap is provided as a pip-installable package featuring an interactive web-based GUI for granular attribution visualization. To our knowledge, this is the first publicly available framework providing a complete, reproducible pipeline for SV-based explainability in text-audio MLLMs.

Abstract

Shapley Value Attribution

Explore similar work

mllm-shap: A Shapley Value Explainability Platform for Text-Audio Multimodal Large Language Models

Bridging Traditional Explainability Methods and Multimodal Multilingual Models: An XAI-Based Analysis

What Did the MLLM Hear? Token-Level Spectro-Temporal Grounding for Audio MLLM Explainability