Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detection

Yonghyun Kim; Chaeyeon Han; Sancho Gatungay; Subhrajit Guhathakurta; Alexander Lerch

eess.ASSep 24, 2026

Cross-Modal Knowledge Distillation for Acoustic Pedestrian Detection

Authors: Yonghyun Kim, Chaeyeon Han, Sancho Gatungay, Subhrajit Guhathakurta, Alexander Lerch

Organizations: Music Informatics Group, Georgia Institute of Technology, USA · Center for Urban Resilience and Analytics, Georgia Institute of Technology, USA · College of Computing, Georgia Institute of Technology, USA

Abstract

Audio-only pedestrian detection is attractive for urban sensing but limited by weak acoustic cues. An appealing strategy is cross-modal knowledge distillation, in which a video teacher supervises the audio student during training so that the deployed model runs on audio alone. Under this task's severe class imbalance and wide video-audio modality gap, however, what such distillation contributes is unclear. We introduce Trust-Filtered Distillation (TFD), which selectively suppresses teacher supervision on pedestrian samples, and interpret its logit formulation as conditional label smoothing under a shared temperature. Across ten distillation configurations under five-fold cross-session validation on ASPED, most methods yield modest gains in macro accuracy accompanied by small changes in PR-AUC. The main effect is higher no-pedestrian accuracy at the cost of lower pedestrian recall. Adding TFD to logit distillation strengthens this trade-off without improving mean PR-AUC. These findings clarify the benefits and limitations of selective cross-modal supervision by distinguishing operating-point shifts from discrimination gains in imbalanced acoustic detection.

Figures & tables

Panel of Figure 1 — Figure 1: System pipeline ( trainable, frozen). Teacher supervision is used only during training, and inference is audio-only. The TFD mask gates the logit and hybrid losses via Eq. 3 ( $\tau{=}0.4$ ), and the masked KD term is aggregated by Eq. 2 .

Logit-based KD
Method	Macro $\uparrow$	No-Ped $\uparrow$	Ped $\uparrow$	$\mathbf{F1_{Ped}}\uparrow$	PR-AUC $\uparrow$	$\bm{d}\uparrow$
Baseline (CE)	$72.9$	$63.5$	$82.3$	$28.6$	$0.328$	–
LogitKD	$73.6$	$70.8$	$76.4$	$31.2$	$0.332$	$1.17$
LogitKD 0.3	$73.8$	$70.3$	$77.2$	$30.9$	$0.326$	$1.36$
Imbalance-aware KD
LogitKD EMA	$73.6$	$68.5$	$78.7$	$30.4$	$0.324$	$1.15$

Table 1: LOSO 5-fold means (accuracies and

\mathrm{F1}_{\mathrm{Ped}}

in percent, PR-AUC in

[0,1]

, per-fold macro std

1.6

–

2.1

d

is Cohen’s

d

(population std) on per-fold macro differences vs. the baseline. Paired

t

-test

p

by row ( ∗ :

p{<}0.05

0.079

0.053

0.083

0.104

0.324

0.106

0.579

0.096

0.033^{*}

0.026^{*}

. None survives Bonferroni (

0.005

, ten tests).

Cross-Modal Knowledge Distillation

Cross-modal knowledge distillation transfers useful representations or predictions between modalities, often enabling cheaper single-modality inference.

See all 122 papers

Detecting Phone-Induced Pedestrian Distraction via a Multimodal Fusion Transformer
Sep 20, 2026Yuanzhe Li, Hounian Liu, Xiaotong Chang +1Pedestrian Behavior Modeling Autonomous Driving Perception
Chair of Automotive Engineering Technische Universität Berlin Berlin, Germany
Learning from imperfect teachers for low-resource acoustic generalization
Oct 4, 2026Shuanglin Li, Ruxiao Qian, Jian Liu +3Uncertainty-Aware Knowledge Distillation Audio Classification
Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation
Sep 15, 2026Vijay John, Amar DabajaMultimodal Robustness Multimodal Classification
Department of Mathematics and Computer Science Lawrence Technological University, Southfield, MI, USA

Sep 20, 2026J/K move · Enter open · S save

Detecting Phone-Induced Pedestrian Distraction via a Multimodal Fusion Transformer

Yuanzhe Li, Hounian Liu, Xiaotong Chang, Yidi Huang

Chair of Automotive Engineering Technische Universität Berlin Berlin, Germany

Pedestrian Behavior Modeling Autonomous Driving Perception Multimodal Fusion

Open paper AskPDF arXiv

The increasing reliance on mobile phones has made phone-induced pedestrian distraction increasingly prevalent. Activities such as texting, watching videos, and making phone calls have become significant contributors to traffic accidents. Reliable detection of pedestrian distraction is essential for autonomous vehicles, as it improves situational awareness and enables timely risk assessment, thereby supporting safe motion planning and vehicle control. We propose a multimodal fusion Transformer (MFT) for detecting phone-induced pedestrian distraction. MFT jointly extracts skeletal dynamics from body pose keypoints and visual appearance features from pedestrian images, effectively leveraging the complementary information provided by the two modalities. A cross-modal attention module is proposed to capture inter-modal dependencies through multi-head cross-attention, facilitating effective fusion of complementary information across the two modalities. Then, a temporal attention fusion module, implemented with a Transformer encoder, is employed to capture temporal dependencies. MFT is trained and evaluated on a manually annotated dataset comprising 287 pedestrian instances with 20,741 images. Extensive experiments demonstrate that MFT attains an overall accuracy of 95%, exceeding the performance of six baseline approaches by 6%.

Abstract

Cross-Modal Knowledge Distillation

Explore similar work

Detecting Phone-Induced Pedestrian Distraction via a Multimodal Fusion Transformer

Learning from imperfect teachers for low-resource acoustic generalization

Multimodal Emergency Vehicle Classification via Audio-Visual Transformers and Knowledge Distillation