PhysicsSliders: Learning Physical Controls for Audio Foley Generation
Authors: Peiwei Chang, Yongqiang Chen, Yuelan Cheng, Liting Gao, Yishun Lu, Li Zhang, Mark D. Plumbley, Philip J. B. Jackson, Wenwu Wang
Organizations: Centre for Vision, Speech and Signal Processing, University of Surrey, UK · Department of Engineering Science, University of Oxford, UK · Department of Computer Science, Royal Holloway, University of London, UK · Department of Informatics, King’s College London, UK
Physical properties provide intuitive controls for sounds made by interacting objects. The challenge is to make a generative model respond predictably to these controls, both individually and in combination. We present \method\ (PXS), trained on simulated contact sounds with 15 material, geometry and contact parameters. Two separately trained variants accept direct parameter inputs or numerical captions. PXS controls hardness, size, speed and damping, supports room descriptions and produces spectrally distinct outputs at fixed targets. When controlling damping and size together, specifying thickness improves frequency accuracy and joint target delivery. Measurements from real recordings define effective-damping targets, which simulator inversion maps to generation inputs without real-audio fine-tuning. Across 18 impact classes from five real-audio datasets, PXS-Physical follows these targets with mean grouped Spearman correlation ρ=0.976 and class-median error 0.094 dex (a factor of 1.24).
Figures & tables
Figure 1: PXS variants train separately with a shared architecture and frozen decoder. Paired labels show PXS-Physical first. Objects and latents are schematic. Upper-right brass impacts vary one input at fixed sampling noise. Spectra (attack 30 ms, resonance 500 ms) and RMS envelopes are peak-normalised. Low/mid/high use E={0.05,2.24,100} GPa, L={0.126,0.200,0.316} m and η={1.71,4.30,10.8}×10−3 . Lower curves show five-level impact sweeps with paired noise. PXS curves geometrically average valid-reading medians (32 samples) across three held-out materials and three runs. Simulator curves average over materials. q is measured effective damping.
Impulsive ( n=18 )
Sustained ( n=12 )
Generator
ρ↑
g→1
Error ↓
ρ↑
g→1
Error ↓
PXS-Physical
.976
.77
.094
.829
.19
.293
PXS-Text
.954
.78
.058
.844
.27
.360
Table 1: Effective-damping control. (a) Tracking ( n : classes; error: dex). (b) Impact requests with two or more of four outputs on target (%).
Table 4: Metrics: VGGish Fréchet audio distance (FAD) [ 23 ] ; Kullback–Leibler (KL) divergence of pretrained audio neural networks (PANNs) [ 24 ] softmax outputs (reference to generated, nats); contrastive language–audio pretraining (CLAP) [ 25 ] audio–text cosine for material-and-strike descriptions. References are 16 independent renders per θ∗ or real target recordings. SAO captions specify material, plate and strike.
Figure 2: Damping–size grid (log axes). Open points geometrically average PXS-Physical four-output medians over three runs. Grey points are eight-output simulator medians. Arrows increase damping (red) or size (blue). Bold lines fix the other control at its midpoint; faint lines show full grids.
Impact sound rendering synthesizes the sound produced when a 3D object is struck, but practical renderers often rely on fixed material presets such as wood, plastic, or steel. These presets limit the range of impact sounds a renderer can express, while manually adjusting the underlying material parameters remains difficult without expertise in material acoustics. We therefore study inverse impact sound rendering: predicting material parameters from a reference impact sound so that a simulator can recreate a similar material response. To support this task, we introduce ImpactMat, a dataset and benchmark of single and blended material impact sounds paired with ground-truth material parameters. We further propose a feed-forward model that predicts these parameters from one or more recordings, using blended materials to learn smooth transitions between material types. Experiments show that our method outperforms competitive baselines and enables re-rendering from real recordings without manual parameter tuning. The project page is available https://material-from-impact.github.io/material-from-impact/.
Hyebin Cho, Bumsoo Kim, Joon Son Chung
Korea Advanced Institute of Science and Technology, South Korea · Chung-Ang University, South Korea
Modern video diffusion models excel at appearance synthesis but still struggle with physical consistency: objects drift, collisions lack realistic rebound, and material responses seldom match their underlying properties. We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation. Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback. This combination enables a generative model to produce physically consistent and controllable outputs through variations in physical attributes-without any simulator or geometry reconstruction at inference. On the Physics-IQ benchmark, PhyCo significantly improves physical realism over strong baselines, and human studies confirm clearer and more faithful control over physical attributes. Our results demonstrate a scalable path toward physically consistent, controllable generative video models that generalize beyond synthetic training environments.
Video generation models have advanced rapidly and can now synthesize plausible videos of robot manipulation from image and text prompts. Recent work extracts robot actions directly from such generated videos, but the result is purely kinematic and lacks force information, causing failures in contact-rich tasks where appropriate contact forces are essential for success. We present a pipeline that jointly leverages generated video and audio to derive motion trajectories and desired-force profiles. The force profile is shaped by the loudness of the generated contact sound, and we execute the resulting force-aware trajectories on a Franka robot using a closed-loop force regulator. We evaluate our pipeline on multiple tasks that require making contact and demonstrate successful zero-shot manipulation where a kinematic-only baseline fails. We also show that the pipeline can be used as a data generation engine to train policies that achieve the tasks in a closed-loop manner. Project website, videos, and dataset: https://dreamingcontactsound.github.io/
Guanhua Ji, Tianyu Li, Dayoon Suh +3
GRASP Lab, University of Pennsylvania, PA 19104, USA