Multimodal Masked Diffusion Autoencoder for Modeling Impaired Driving
Organizations: Toyota Research Institute †
Abstract
Human cognitive state (e.g., distraction, stress), gaze, and behavior interact with the driving environment in subtle ways that AI could use to assist drivers, but ground-truth labels for human cognitive state levels are costly to obtain at scale. We propose a multimodal masked-diffusion autoencoder, ID-MAE, that adds a self-supervised objective to impairment prediction: alongside predicting the impairment label as a masked token, it reconstructs masked gaze and trajectory. To probe the effects of different modalities and types of impairments on model inference, we introduce DIVERT, a driving-simulator dataset containing 103 hours of driving data (n = 109) with ten impairment inductions spanning cognitive, visual, and emotional states, recorded with gaze, vehicle state, and scene video. We demonstrate how ID-MAE affords significant improvements in impairment inference results on both the DIVERT dataset and existing datasets, and enables strong results in additional tasks, such as trajectory and gaze prediction. For location-matched pairs, our model achieves a paired AUC of 0.706 for impairment detection on DIVERT, compared to 0.590 without reconstruction loss, improving over existing baselines. We probe the design space in terms of input modalities, masking scheme, positional encoding, attention mechanism, and decoding. We also investigate the interaction between various impairments in training and inference, demonstrating that visually demanding distractions are separable from cognitive distractions. Code and data will be available upon acceptance.
Figures & tables
| Modality | Hz | Clean token content | Attn. | Decoded | |
|---|---|---|---|---|---|
| Impair. | 10 | 1 | one-hot ( ) | logits ( ) | |
| Gaze | 10 | 1 | validity (4), events (12) | (4) | |
| Traj. | 2 | 8 | , flags (9) | (4) | |
| Video | 2 | 21 | ViT-B/32 patch (768) | – |
| Dataset | # Impairment Inductions (Categories) | #Hazard Categ. | Released Modalities | Subjects | Hours |
|---|---|---|---|---|---|
| CoCAtt [ 80 ] | 1 (cog. distraction) a | 0 | G, S, V, L | 11 | 11.9 |
| Keshtkaran et al. [ 46 ] | 1 (alcohol) | 1 b | S, C | 60 | 30 |
| DATAD [ 39 ] | 1 (phone distraction before takeover) a | 4 | G, S c , V, T | 60 | n/r |
| IDD [ 30 ] | 3 (alcohol, cog. distraction) | 4 | G, S, V, Q, T | 52 | 23.7 |
| manD 1.0 [ 20 ] | 19 (cog. load, emot., visual/manual distraction) a, d | 6 | G, C, V, P, Q, T | 50 e | n/r |
| C42CN [ 84 ] | 5 (cog., emot., and sensorimotor distraction) | 1 | G, S, C, V, P, Q | 68 | 80 f |
| Model | ADE | ADE best | FDE | FDE best |
|---|---|---|---|---|
| Constant velocity | 10.37 | - | 26.80 | - |
| Train mean rate | 10.51 | - | 27.20 | - |
| Diffusion, past vid. | 7.61 | 5.80 | 20.99 | 15.17 |
| Single step, past vid. | 7.05 | 6.68 | 19.09 | 17.87 |
| Diffusion, full vid. | 5.11 | 4.73 | 12.79 | 11.59 |
| Single step, full vid. | 5.28 | 5.12 | 13.27 | 12.76 |
| Angular error @ 1 s | Angular error @ 2 s | |||
|---|---|---|---|---|
| Model | Single | Best | Single | Best |
| Visible context mean | 11.17 | - | 11.02 | - |
| Train mean | 11.36 | - | 11.14 | - |
| Diffusion, past vid. | 10.60 | 8.16 | 10.54 | 8.45 |
| Single step, past vid. | 9.31 | 9.15 | 9.23 | 9.09 |
| Diffusion, full vid. | 9.84 | 8.07 | 9.54 | 8.11 |
Multimodal Diffusion Models
Multimodal diffusion models generate or interpret text, images, audio, video, motion, and other data by jointly denoising representations conditioned across modalities.
See all 186 papers