Physical understanding is crucial for decision making, yet vision models struggle to infer properties such as damping that govern how materials behave. Audio encodes such properties directly, because an object's acoustic response to impact reveals its damping through measurable decay. We probe frozen encoders for the quality factor Q, a dimensionless measure of how many oscillations an object completes before its ring dies away, and report how well each embedding orders impacts by Q using the Spearman rank correlation ρ, for which ρ=1 is a perfect ordering and ρ=0 an unrelated one. Audio embeddings reach ρ=0.73 and retain most of that advantage on materials withheld from training, while a single video frame taken at the moment of impact, encoded by CLIP, supports only ρ=0.28. Having established where the signal lives, we ask how much of that gap vision can recover. We distill a frozen audio teacher into a frozen CLIP backbone through a trained projection head and find that this recovers a useful fraction of the audio advantage. Because pooled correlations on this task are dominated in practice by category identity, we also evaluate under a leave-one-material-out protocol in which the held-out material is withheld from the projection head as well as the probe. Distillation with a four-frame input improves the correlation there from 0.234 to 0.338 (+0.053 over the matched four-frame baseline, 14 of 17 materials), a gain that no category-level strategy can produce. Widening the visual input to four frames spanning the impact supplies half of that improvement at no acoustic cost, which indicates that part of what a single frame misses is visible dynamics.
Figures & tables
Figure 1: What Q measures, and how we transfer it. (a) Example impacts from the Greatest Hits dataset: a drumstick strike shown at the frame of contact, with example Q values ranging from 223 (glass) to 15 (cloth); per-material medians are given in Section 3.3 . (b) Hilbert envelopes plotted against time show that absolute duration is governed by the damping rate β : the metal bowl ( β=14 s -1 ) rings longest even though glass has the higher Q . (c) Plotted against elapsed oscillation cycles tf0 , the ordering follows Q , with cloth and wood falling silent within tens of cycles while glass and metal persist past two hundred. Q therefore measures the quality of resonance rather than its duration. (d) The distillation setup: both encoders stay frozen and only the projection head is trained, with a symmetric InfoNCE objective. No Q label enters training, and the ridge probe used for evaluation is fit afterward on held-out videos.
Model
Modality
ρ
95% CI
PE-AV
Audio
0.727
[0.707, 0.746]
CAV-MAE
Audio
0.701
[0.681, 0.721]
PE-AV
Joint
0.696
[0.676, 0.717]
CAV-MAE
Joint
0.608
[0.583, 0.633]
PE-AV
Video
0.362
[0.333, 0.393]
CLIP
Video
0.281
[0.247, 0.313]
Table 1: Audio embeddings capture the quality factor Q , while single-frame vision largely does not. Spearman ρ between ridge-probe predictions from frozen embeddings and ground-truth Q , on the held-out video split.
Model
Modality
LOMO ρ
95% CI
PE-AV
Audio
0.600
[0.528, 0.658]
CAV-MAE
Audio
0.595
[0.538, 0.645]
PE-AV
Joint
0.572
[0.505, 0.630]
CAV-MAE
Joint
0.492
[0.441, 0.541]
DINOv2
Video
0.239
[0.177, 0.299]
CLIP
Video
0.227
[0.167, 0.289]
Table 2: Physical structure generalizes to unseen materials. Mean over the 17 held-out materials, with intervals bootstrapped over materials.
Figure 2: Why pooled correlation is a weak measure of physical understanding. Predicted against true Q on the held-out split ( n=3,510 ), with predictions standardized to the target scale and the diagonal shown for reference. Left: a single frame at impact supports only a weak trend. Middle: after distillation, the cloud tightens appreciably. This panel shows the first, and highest, of the three seeds ( ρ=0.466 , against the three-seed mean of 0.440 in Table 3 ). Right: mapping the true material label to its per-category median Q produces just 17 distinct predictions, visible as horizontal bands, and still reaches ρ=0.430 . A model that cannot separate two metal objects at all therefore scores almost as well on this measure as our distilled encoder, which is why the argument rests on Section 6.4 instead.
Visual input / model
ρ
Interval
Type
No audio supervision
Single frame at impact
0.281
[0.247, 0.313]
95% CI
Four frames, concatenated
0.299
[0.269, 0.332]
95% CI
Four frames + deltas
0.298
[0.267, 0.329]
95% CI
Distilled from the audio teacher
Four frames + deltas
0.440
[0.426, 0.466]
three-seed range
Table 3: Q prediction from vision on the held-out video split, with and without distillation from the audio teacher.
Model
LOMO ρ
95% CI
vs. single frame
Baseline, single frame †
0.234
[0.175, 0.289]
—
Baseline, four frames + deltas †
0.285
[0.200, 0.372]
+0.051
Distilled, four frames + deltas
0.338
[0.249, 0.430]
+0.104
Table 4: True LOMO: the held-out material is excluded from the projection head as well as the probe. Mean over 17 materials and three seeds, with intervals bootstrapped over materials.
Control
Condition
ρ
How much is category recognition alone worth?
CLIP material classifier (45.6% acc.) → median Q
0.279
true material label → median Q
0.430
Is the trainable head doing the work? (cached single-frame features † )
linear probe on log(Q+1)
0.300
three-layer MLP on log(Q+1) , no teacher
0.275
Table 5: Controls, as pooled ρ on the held-out video split. Correspondence arms hold the seed fixed, so the matched-teacher row is the first, and highest, of the three seeds (0.466) rather than their mean (0.440).
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Pooled ρ
LOMO ρ
Model
f0
β
f0
β
PE-AV (audio)
0.700
0.729
0.485
0.686
CAV-MAE (audio)
0.667
0.732
0.423
0.719
PE-AV (joint)
0.671
0.682
0.439
0.628
CLIP
0.347
0.221
0.137
0.257
DINOv2
0.353
0.210
0.152
0.244
Appendix
Table 6: Probing f0 and β separately. Audio embeddings predict the damping term β better than the spectral term f0 , and the margin widens on unseen materials. Bold marks the better of f0 and β within each row and setting. The distilled row’s LOMO columns use true LOMO, with the held-out material excluded from both the projection head and the probe, and comes from a single-frame student (the frame at impact), trained separately from the four-frame model of Tables 3 – 4 , so it is compared like-for-like with the single-frame CLIP row.
Model
Metal
Wood
Ceramic
Plastic
PE-AV (audio)
0.67
0.57
0.31
0.45
CAV-MAE (audio)
0.65
0.45
0.12
0.32
PE-AV (joint)
0.64
0.41
0.02
0.34
CAV-MAE (joint)
0.43
0.25
− 0.10
0.18
PE-AV (video)
0.36
0.07
− 0.08
0.00
DINOv2
0.30
0.09
0.09
− 0.03
Appendix
Table 7: Within-material Q prediction ( ρ ), shown for four representative rigid materials.
Audio-visual representation learning develops models that connect sound and video for recognition, retrieval, localization, and robust multimodal understanding.
Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.
Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu +6
dUniversity of California, Davis · pPrinceton University · wUniversity of Wisconsin–Madison +1
While modern 3D reconstruction excels at modeling object geometry and appearance, it largely ignores the rich acoustic cues revealed through physical interaction. Object impact sounds convey material, stiffness, and structural properties that complement vision, yet existing impact sound modeling approaches either rely on expensive physics-based simulation or require large datasets to generalize in a purely data-driven manner. We introduce Audio-Visual Modal Sound Field (AV-MSF), a novel object-level acoustic representation reconstructed from multi-view images and only a few impact sound recordings. AV-MSF builds on 3D Gaussian Splatting integrated with dense 3D visual feature to provide a strong geometry-aware prior, and represents the impact sound field using compact, physically meaningful modal parameters, enabling robust few-shot reconstruction. Experiments on two real-world datasets show that AV-MSF achieves state-of-the-art impact sound rendering, outperforming both physics-based and data-driven baselines. Furthermore, we demonstrate downstream applications enabled by our representation, including contact localization and object sound editing.
Audio foundation models are widely adopted as general-purpose feature extractors, yet the internal structure of their learned representations remains insufficiently understood. In this work, we analyze CLAP audio embeddings through a probing framework, studying the encoding of three fundamental perceptual dimensions: reverberation (RT60), loudness (LUFS), and spectral content, measured via spectral centroid (SC) and relative pitch (RP). Probes of increasing complexity are trained to predict each attribute from frozen embeddings across five datasets spanning noise, speech, monophonic musical notes, and music mixtures. Our primary finding is that all of these attributes are reliably recoverable from the CLAP embedding space across the examined datasets. Within this global picture, two encoding regimes emerge: RT60, LUFS, and RP are approximately linearly encoded, while SC requires non-linear probes. Both regimes generalize across eight additional audio foundation models, with the notable exception that amplitude-invariant architectures discard loudness entirely by construction. The identified linear feature directions are geometrically consistent across datasets for RT60 and LUFS, while highly domain-specific for RP. Finally, we provide a qualitative demonstration of cross-modal consistency, showing that text embeddings of acoustic descriptors align geometrically with the identified RT60 feature direction.