Predictive audio representations for early detection and tracking of hidden dynamic objects
Authors: Katerina Vinciguerra, Moritz Brandes, Danilo Hollosi, Letizia Marchegiani
Organizations: Department of Engineering and Architecture, University of Parma, Parma, Italy · Fraunhofer Institute for Digital Media Technology IDMT, Oldenburg, Germany
Predicting potential dangers is core to safety. Forecasting the presence of other traffic agents is core to danger prediction. Occluded traffic agents challenge detection systems as they might become visible too late, leaving the autonomous vehicle too little time to identify, plan and act accordingly in a robust and safe way. Previous works proved that auditory perception, being omnidirectional and not constrained by a field-of-view, provides fundamental cues for early spotting of different road users, even when hidden by other vehicles or infrastructures. Yet, those contributions deal with scenarios with only one vehicle present, and they either identify the type of vehicle or estimate its direction of arrival. In this work, we move forward, and propose a multi-task system which, simultaneously, estimates the number of vehicles present, their type, and their direction of arrival. Our methodological contribution is a two-stage pipeline: a self-supervised pre-training stage inspired by the Joint- Embedding Predictive Architecture (JEPA) applied directly to multichannel raw waveforms, followed by supervised multi-task fine-tuning with a bidirectional LSTM and three classification heads. The pre-training stage trains the encoder without labels, pushing it to predict the latent representation of a future audio segment from its past context. To train and test our framework, since no suitable dataset was publicly available, we collected an ad-hoc one covering Non-Line-Of-Sight scenarios with multiple traffic agents simultaneously operating. Experimental evaluation shows that our method outperforms the state of the art; it also proves that our design choice allows the model to learn robust representations, which can be transferred to an unseen driving scenario, maintaining reasonable and stable performance.
Figures & tables
Fig. 1 : Example of a typical use-case: a car is approaching from the left and a van is approaching from the right at T-junction with buildings occluding them. The 8-microphone on-board array captures their raw audio signatures, which a self-supervised pretrained encoder maps into latent embeddings. A supervised head then aggregates a 3-second window of these embeddings to predict, at every second, the direction, number, and type of the non-line-of-sight vehicles.
Fig. 2 : Overview of the proposed two-stage pipeline. Stage 1 - JEPA-inspired pre-training (self-supervised): the online encoder fθ maps the past frame xn−2∈RM×L to a latent vector zn−2∈RD ; the predictor gϕ forecasts the future frame xn , supervised by the target encoder fθˉ via LJEPA (Eq. 5 ); gradients flow through fθ and gϕ only, after which gϕ is discarded and weights θ are transferred to Stage 2. Stage 2 - Multi-task fine-tuning (supervised): the three frames of Wn are encoded independently, yielding (zn−2,zn−1,zn) , processed by a bidirectional LSTM whose final hidden state hn passes through a shared dropout-MLP block and three linear head predicting direction (5 classes), count (3 classes), and vehicle type (4 classes) via L (Eq. 6 , λ=5 ).
Fig. 3 : Sensor configuration used in data collection. In (a): schematic top-view representation; In (b), GPS (1) and camera–microphone (2) system mounted on the vehicle.
Fig. 4 : Reference scenarios ; ego-vehicle in blue. (a-d) static (T-junction): one (a) or two vehicles (b,c) approaching from the same direction (one vehicle occluding the other); two vehicles approaching from different directions (d). (e-f) driving (take-over): one (e) or two vehicles (f) approaching from the back and overtaking the ego-vehicle.
Method / Config.
Direction
Count
Type
Baselines
LSTM only
33.7
47.0
40.0
Related Works
DCASE baseline [ 5 ]
67.5
89.5
66.9
SRP-PHAT + SVM [ 1 ]
53.4
-
-
DOA + TF + CNN [ 3 ]
61.9
-
-
TABLE I : Results obtained in Experiment 1 and 2 , expressed as accuracy (%) in the three tasks: Direction , Count , and Type . The symbol ‘ − ’ stands for ‘not applicable’.
TO Dataset
0-shot
5-shot
10-shot
Task
Class
TJ Dataset
all
A
B
C
A
B
C
Direction
left
81.8
1.7
32.4±22.0
31.4±20.9
28.4±18.8
15.6±17.3
17.7±17.6
20.9±15.5
front
60.0
5.6
60.4±18.0
60.3±17.2
57.9±20.3
70.8±8.2
70.7±8.0
71.7±6.0
right
71.3
13.0
6.0±2.4
4.3±3.2
0.0±0.0
8.0±7.6
9.0±7.0
4.7±8.2
back
N/A
0.0
57.7±29.0
60.4±27.6
62.1±27.6
83.7±23.7
82.9±23.4
80.3±17.3
TABLE II : Per-class recall (%) of the proposed model on the TJ (static) validation set and on the TO (driving) dataset. Static results correspond to the best checkpoint trained on T-junction data only. Driving results are reported for the three adaptation strategies: A (head-only), B (full-model), C (SSL re-pretraining + head). For K>0 , each cell reports mean, ± std over N=10 random support-set draws. At zero-shot ( K=0 ) no fine-tuning is performed and the three options are identical. N/A stands for ‘Not Available’, as the class is not present in this scenario.
Fig. 5 : Predictions on the front class in the Direction task , broken down by frame position within the front visibility cone (cf., Section IV-E2 ); each row reports the share of predictions in each subclass. k the number of frames.
Fig. 6 : UMAP projection of the frozen-encoder validation latents (same projection coordinates across all panels), coloured by the three downstream classification labels. (a) Direction (b) Count (c) Type.
Anticipating traffic accidents from dashcam videos is a critical challenge in intelligent transportation systems. Existing methods typically map visual context directly to a collision probability without explicitly modeling the future evolution of the driving scene. In this paper we propose FLaRA (Predicting Future Latent Representations for Accident Anticipation), a novel predictive architecture that shifts this paradigm by forecasting future latent representations for accident anticipation. Building upon the Video Joint-Embedding Predictive Architecture (V-JEPA2), our model conditions a predictor network on observed context frames to predict the forthcoming latent features of the scene. A classifier then operates on these predicted future representations rather than only on past observations. To ensure these forecasts remain grounded in realistic future dynamics, we introduce a joint training objective that simultaneously optimizes an auxiliary feature-level reconstruction loss and a cross-entropy classification loss. Extensive evaluations on the Nexar dataset, alongside cross-domain validations on the DAD, DADA-2000, and DoTA benchmarks, demonstrate that our approach achieves state-of-the-art performance while maintaining realistic early warning capabilities.
Lorenzo Caselli, Tomaso Trinci, Tommaso Bianconcini +4
Department of Information Engineering, University of Florence, Italy. · 2Verizon Connect, Florence, Italy.
Emergency vehicle detection in autonomous driving is a safety-critical perception task that demands robustness under diverse and adverse real-world conditions. Existing approaches rely on a single modality, either audio or video, which leads to systematic failure when that modality is degraded: microphone-based systems fail in noisy urban environments, and camera-based systems fail at night or under occlusion. This report presents AVNet, a multimodal audio-visual transformer that classifies emergency vehicles (ambulance, fire engine, police car) and road background using both audio and video, while gracefully handling the absence of either modality at inference time. AVNet introduces three key contributions: (1) a temporally aligned cross-modal fusion module that performs second-level cross-attention between audio spectrogram tokens and video frame tokens, exploiting their exact temporal correspondence without any learned alignment mechanism; (2) learned null embeddings that substitute for missing modality tokens, enabling a single unified model to operate in audio-only, video-only, or joint audio-visual mode without retraining; and (3) a knowledge distillation training strategy in which specialist unimodal teacher models transfer inter-class dark knowledge into the multimodal student fusion branch via soft probability targets. Evaluated on 281 clips from the Google AudioSet dataset, AVNet achieves 66.6% overall accuracy in audio-visual mode, outperforming the audio-only branch by +10.4% and the video-only branch by +15.0%. The largest per-class gain is observed for the hardest class, Ambulance, where fusion achieves +29.5% over either unimodal branch alone, demonstrating that the two modalities provide complementary information that the aligned cross attention mechanism successfully exploits.
Vijay John, Amar Dabaja
Department of Mathematics and Computer Science Lawrence Technological University, Southfield, MI, USA
We present the 1st-place solution to the ACCIDENT challenge at the CVPR 2026 AUTOPILOT Workshop, which asks for zero-shot prediction of accident timing, impact centroid, and collision type from CCTV footage. On a frozen Qwen3-VL-32B-Instruct checkpoint we build a three-stage pipeline (full-video joint prediction, time refinement, and single-frame grounding of the impact centroid), run the same pipeline a second time on a 235B Mixture-of-Experts sibling, blend the two outputs 9:1, and finally snap each predicted point onto the nearest vehicle detection. The final system reaches Public LB 0.55469 / Private LB 0.57080, roughly +0.21 over the strongest host baseline (Molmo-7B, 0.358) and wins the challenge. We ablate each component, report the negative results that shaped the final design, and release the code at https://github.com/fuumin621/cvpr2026-accident-1st-place-solution.