VeV: Video-Enhanced Vibration Data Augmentation For Structural Difference
Organizations: Yonsei University, Seoul, South Korea · University of Michigan, Ann Arbor, MI, United States · University of Illinois Urbana-Champaign, IL, United States
Abstract
Surface-mounted vibration sensing provides non-invasive structural monitoring with high temporal fidelity across diverse ubiquitous computing scenarios such as smart farms, yet structure-borne wave models remain fragile to physical relocation and structural shifts. However, addressing this spatial data discontinuity remains limited by costly manual recalibration, insufficient physical grounding in data-driven synthesis, and substantial sim-to-real gaps in numerical digital twins. To resolve this spatial bottleneck without labor-intensive data recollection, we propose VeV, a physics-consistent spatial augmentation framework that synthesizes vibration responses at unmeasured target coordinates from sparse reference observations by coupling a monocular Vision-Derived Propagation Possibility Map for structural wave-guiding paths with a Visual Feature Map for transient source kinetics and implicit material dynamics. Evaluated on a 1,237 sensor-hour sow pig and piglet farm across disparate structures, VeV reduces waveform reconstruction MSE by over 57.5% compared to data-driven baselines and achieves up to 81.69% zero-shot classification accuracy, while reaching peak performance within just 30-50 calibration chunks to effectively eliminate cold-start retraining overhead and establish a scalable foundation extending to human-centric floor vibration sensing and ambient intelligence.
Figures & tables
| Vision Accuracy | Vibration Accuracy | ||||||
| Seen Crate | Unseen Crate | ||||||
| Arch | Size | Seen Crate | Unseen Crate | Seen Sensor Loc. | Unseen Sensor Loc. | Seen Sensor Loc. | Unseen Sensor Loc. |
| CNN | small | 0.8465 | 0.8381 | 0.8027 | 0.3421 | 0.1802 | 0.1145 |
| medium | 0.8548 | 0.8471 | 0.7598 | 0.2854 | 0.0957 | 0.0712 | |
| large | 0.7810 | 0.7911 | 0.7170 | 0.2519 | 0.0801 | 0.0583 | |
| RNN | small | 0.7323 | 0.7350 | 0.8661 | 0.3812 | 0.1378 | 0.0924 |
| Augmentation Paradigm | Seen Crate | Unseen Crate | ||
| Seen Loc | Unseen Loc | Seen Loc | Unseen Loc | |
| Classifier Upper Bound (Classifier w/ real data) | 90.40% | 91.42% | 90.96% | 89.87% |
| (1) Naive Data-Driven (Classifier w/ LSTM based synthetic data) | 38.05% | 28.35% | 42.20% | 21.94% |
| (2-a) Audio Encoder (Classifier w/ WavLM based synthetic data) | 52.74% | 53.34% | 63.03% | 54.33% |
| (2-b) Audio Encoder (Classifier w/ BEATs based synthetic data) | 59.23% | 61.73% | 69.91% | 63.07% |
| (3-a) Video-to-Audio (Classifier w/ MMAudio based synthetic data) | 5.12% | 4.97% | 4.12% | 4.96% |
| Crate | Crate Group | Seen Loc (2,3,5) [hours] | Unseen Loc (1,4) [hours] | Total [hours] |
| Crate_02 | Seen | 139.692 | 0.000 | 139.692 |
| Crate_03 | Seen | 139.827 | 0.000 | 139.827 |
| Crate_08 | Seen | 277.185 | 185.233 | 462.419 |
| Crate_04 | Unseen | 68.455 | 0.000 | 68.455 |
| Crate_06 | Unseen | 69.896 | 0.000 | 69.896 |
| Crate_07 | Unseen | 214.073 | 142.661 | 356.734 |
| Method | Seen Crate + Seen Loc. | Seen Crate + Unseen Loc. | Unseen Crate + Seen Loc. | Unseen Crate + Unseen Loc. | ||||
| Acc. (%) | Micro-F1 | Acc. (%) | Micro-F1 | Acc. (%) | Micro-F1 | Acc. (%) | Micro-F1 | |
| Naive | 38.05% | 0.3470 | 28.35% | 0.1909 | 42.20% | 0.2554 | 21.94% | 0.1916 |
| Audio | 59.23% | 0.4243 | 61.73% | 0.4288 | 69.91% | 0.2075 | 63.07% | 0.1923 |
| FEM | 0% | 0.0 | 0% | 0.0 | 0% | 0.0 | 0% | 0.0 |
| Ours | 81.69% | 0.8180 | 78.54% | 0.7437 | 72.16% | 0.6852 | 71.69% | 0.6749 |
| Method | Seen Crate + Seen Loc. | Seen Crate + Unseen Loc. | Unseen Crate + Seen Loc. | Unseen Crate + Unseen Loc. | ||||
| Acc. (%) | Micro-F1 | Acc. (%) | Micro-F1 | Acc. (%) | Micro-F1 | Acc. (%) | Micro-F1 | |
| Proposed method ( VeV ) | 81.69% | 0.8180 | 78.54% | 0.7437 | 72.16% | 0.6852 | 71.69% | 0.6149 |
| Visual Feature Map Only | 64.80% | 0.6321 | 62.54% | 0.7107 | 58.71% | 0.5752 | 59.20% | 0.5471 |
| Propagation Map Only | 29.58% | 0.2717 | 29.56% | 0.1492 | 32.21% | 0.2763 | 28.57% | 0.2682 |
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Architecture | Size | Vibration Params | Vision Params |
| CNN | small | 141,445 | 99,589 |
| medium | 561,413 | 391,685 | |
| large | 2,236,933 | 1,553,413 | |
| RNN | small | 125,445 | 183,237 |
| medium | 793,093 | 1,019,269 | |
| large | 4,341,765 | 5,236,485 |
| Configuration Category | Design Specification | Value / Choice |
| Material Density ( ) | Structural Steel Baseline | |
| Young’s Modulus ( ) | Baseline Elasticity | |
| Poisson’s Ratio ( ) | Transverse Contraction | |
| Rayleigh Damping ( ) | Mass/Stiffness Damping | |
| Spatial Element Type | 3D Tetrahedron | 10-node quadratic tetrahedral |
| Polynomial Order ( ) | Quadratic Basis | (Mitigate shear locking) |
Data Augmentation
Data augmentation expands or transforms training examples to improve learning, data efficiency, and robustness to distribution shifts.
See all 273 papers