How Much Physics Can Vision Learn from Audio? Probing and Distilling Damping Structure

Zainab Afolabi; Vadim Selyutin; Zhao Gao; Babs Khalidson; Mohamed Ghoneim

cs.CVOct 15, 2026

How Much Physics Can Vision Learn from Audio? Probing and Distilling Damping Structure

Authors: Zainab Afolabi, Vadim Selyutin, Zhao Gao, Babs Khalidson, Mohamed Ghoneim

Organizations: Amazon Web Services

Abstract

Physical understanding is crucial for decision making, yet vision models struggle to infer properties such as damping that govern how materials behave. Audio encodes such properties directly, because an object's acoustic response to impact reveals its damping through measurable decay. We probe frozen encoders for the quality factor $Q$ , a dimensionless measure of how many oscillations an object completes before its ring dies away, and report how well each embedding orders impacts by $Q$ using the Spearman rank correlation $ρ$ , for which $ρ=1$ is a perfect ordering and $ρ=0$ an unrelated one. Audio embeddings reach $ρ=0.73$ and retain most of that advantage on materials withheld from training, while a single video frame taken at the moment of impact, encoded by CLIP, supports only $ρ=0.28$ . Having established where the signal lives, we ask how much of that gap vision can recover. We distill a frozen audio teacher into a frozen CLIP backbone through a trained projection head and find that this recovers a useful fraction of the audio advantage. Because pooled correlations on this task are dominated in practice by category identity, we also evaluate under a leave-one-material-out protocol in which the held-out material is withheld from the projection head as well as the probe. Distillation with a four-frame input improves the correlation there from 0.234 to 0.338 (+0.053 over the matched four-frame baseline, 14 of 17 materials), a gain that no category-level strategy can produce. Widening the visual input to four frames spanning the impact supplies half of that improvement at no acoustic cost, which indicates that part of what a single frame misses is visible dynamics.

Figures & tables

Figure 1: What $Q$ measures, and how we transfer it. (a) Example impacts from the Greatest Hits dataset: a drumstick strike shown at the frame of contact, with example $Q$ values ranging from 223 (glass) to 15 (cloth); per-material medians are given in Section 3.3 . (b) Hilbert envelopes plotted against time show that absolute duration is governed by the damping rate $\beta$ : the metal bowl ( $\beta=14$ s -1 ) rings longest even though glass has the higher $Q$ . (c) Plotted against elapsed oscillation cycles $tf_{0}$ , the ordering follows $Q$ , with cloth and wood falling silent within tens of cycles while glass and metal persist past two hundred. $Q$ therefore measures the quality of resonance rather than its duration. (d) The distillation setup: both encoders stay frozen and only the projection head is trained, with a symmetric InfoNCE objective. No $Q$ label enters training, and the ridge probe used for evaluation is fit afterward on held-out videos.

Model	Modality	$\boldsymbol{\rho}$	95% CI
PE-AV	Audio	0.727	[0.707, 0.746]
CAV-MAE	Audio	0.701	[0.681, 0.721]
PE-AV	Joint	0.696	[0.676, 0.717]
CAV-MAE	Joint	0.608	[0.583, 0.633]
PE-AV	Video	0.362	[0.333, 0.393]
CLIP	Video	0.281	[0.247, 0.313]

Table 1: Audio embeddings capture the quality factor

\boldsymbol{Q}

, while single-frame vision largely does not. Spearman

\rho

between ridge-probe predictions from frozen embeddings and ground-truth

Q

, on the held-out video split.

Model	Modality	LOMO $\boldsymbol{\rho}$	95% CI
PE-AV	Audio	0.600	[0.528, 0.658]
CAV-MAE	Audio	0.595	[0.538, 0.645]
PE-AV	Joint	0.572	[0.505, 0.630]
CAV-MAE	Joint	0.492	[0.441, 0.541]
DINOv2	Video	0.239	[0.177, 0.299]
CLIP	Video	0.227	[0.167, 0.289]

Table 2: Physical structure generalizes to unseen materials. Mean over the 17 held-out materials, with intervals bootstrapped over materials.

Figure 2: Why pooled correlation is a weak measure of physical understanding. Predicted against true $Q$ on the held-out split ( $n=3{,}510$ ), with predictions standardized to the target scale and the diagonal shown for reference. Left: a single frame at impact supports only a weak trend. Middle: after distillation, the cloud tightens appreciably. This panel shows the first, and highest, of the three seeds ( $\rho=0.466$ , against the three-seed mean of 0.440 in Table 3 ). Right: mapping the true material label to its per-category median $Q$ produces just 17 distinct predictions, visible as horizontal bands, and still reaches $\rho=0.430$ . A model that cannot separate two metal objects at all therefore scores almost as well on this measure as our distilled encoder, which is why the argument rests on Section 6.4 instead.

No audio supervision
Visual input / model	$\boldsymbol{\rho}$	Interval	Type
Single frame at impact	0.281	[0.247, 0.313]	95% CI
Four frames, concatenated	0.299	[0.269, 0.332]	95% CI
Four frames + deltas	0.298	[0.267, 0.329]	95% CI
Distilled from the audio teacher
Four frames + deltas	0.440	[0.426, 0.466]	three-seed range

Table 3:

\boldsymbol{Q}

prediction from vision on the held-out video split, with and without distillation from the audio teacher.

Model	LOMO $\boldsymbol{\rho}$	95% CI	vs. single frame
Baseline, single frame †	0.234	[0.175, 0.289]	—
Baseline, four frames + deltas †	0.285	[0.200, 0.372]	+0.051
Distilled, four frames + deltas	0.338	[0.249, 0.430]	+0.104

Table 4: True LOMO: the held-out material is excluded from the projection head as well as the probe. Mean over 17 materials and three seeds, with intervals bootstrapped over materials.

Control	Condition	$\boldsymbol{\rho}$
How much is category recognition alone worth?
	CLIP material classifier (45.6% acc.) $\to$ median $Q$	0.279
	true material label $\to$ median $Q$	0.430
Is the trainable head doing the work? (cached single-frame features † )
	linear probe on $\log(Q+1)$	0.300
	three-layer MLP on $\log(Q+1)$ , no teacher	0.275

Table 5: Controls, as pooled

\boldsymbol{\rho}

on the held-out video split. Correspondence arms hold the seed fixed, so the matched-teacher row is the first, and highest, of the three seeds (0.466) rather than their mean (0.440).

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

	Pooled $\boldsymbol{\rho}$		LOMO $\boldsymbol{\rho}$
Model	$f_{0}$	$\beta$	$f_{0}$	$\beta$
PE-AV (audio)	0.700	0.729	0.485	0.686
CAV-MAE (audio)	0.667	0.732	0.423	0.719
PE-AV (joint)	0.671	0.682	0.439	0.628
CLIP	0.347	0.221	0.137	0.257
DINOv2	0.353	0.210	0.152	0.244

Appendix

Table 6: Probing

\boldsymbol{f_{0}}

and

\boldsymbol{\beta}

separately. Audio embeddings predict the damping term

\beta

better than the spectral term

f_{0}

, and the margin widens on unseen materials. Bold marks the better of

f_{0}

and

\beta

within each row and setting. The distilled row’s LOMO columns use true LOMO, with the held-out material excluded from both the projection head and the probe, and comes from a single-frame student (the frame at impact), trained separately from the four-frame model of Tables 3 – 4 , so it is compared like-for-like with the single-frame CLIP row.

Model	Metal	Wood	Ceramic	Plastic
PE-AV (audio)	0.67	0.57	0.31	0.45
CAV-MAE (audio)	0.65	0.45	0.12	0.32
PE-AV (joint)	0.64	0.41	0.02	0.34
CAV-MAE (joint)	0.43	0.25	$-$ 0.10	0.18
PE-AV (video)	0.36	0.07	$-$ 0.08	0.00
DINOv2	0.30	0.09	0.09	$-$ 0.03

Appendix

Table 7: Within-material

\boldsymbol{Q}

prediction (

\boldsymbol{\rho}

), shown for four representative rigid materials.

Audio-Visual Representation Learning

Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.

See all 31 papers

When Vision Speaks for Sound
May 13, 2026Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6Audio-Language Model Evaluation Audio-Visual Understanding
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1
Objects as Audio-Visual Modal Sound Fields
Aug 5, 2026Zisen Shao, Zihao Wei, Derong Jin +1
University of Maryland, College Park
Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings
Jul 4, 2026Héctor Martel, Joe Hennessy-Priest, Taemin ChoRepresentation Geometry Audio Representation Learning
BandLab Technologies, Singapore

May 13, 2026J/K move · Enter open · S save

When Vision Speaks for Sound

Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai+5

dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison+1

Audio-Language Model Evaluation Audio-Visual Understanding Video-Language Model Evaluation Audio Reasoning Audio Understanding Video-Language Models+5

Open paper AskPDF arXiv

Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.

Abstract

Audio-Visual Representation Learning

Explore similar work

When Vision Speaks for Sound

Objects as Audio-Visual Modal Sound Fields

Probing Low-Level Acoustic Attribute Encoding in CLAP Audio Embeddings