DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset
Authors: Yiming Li, Dan Lin, Wenfei Liang, Tianyi Liu, Chen Cai, Wenqian Wang, Kim-Hui Yap, Liang Zhao, Bingbing Li
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · College of Computer Science and Technology, Harbin Engineering University · Singapore University of Technology and Design (SUTD), Singapore · Continental Automotive Singapore Pte. Ltd.
Multimodal action understanding requires complementary sensory observations to be connected with structured semantic knowledge. This is particularly challenging in vehicle cabins, where fine-grained actions exhibit similar upper-body motion, relevant objects may be small or occluded, and appearance varies substantially across illumination conditions. Existing in-cabin datasets, however, rarely provide synchronized visual modalities together with dense object annotations and explicit action--object alignment, limiting the study of object-grounded multimodal recognition. To address this gap, we introduce the Driver Action with Object Synergy (DAOS) dataset, containing 9,787 video clips, 36 fine-grained actions, 15 object categories, and more than 2.5 million annotated object instances. DAOS provides synchronized RGB, infrared, and depth observations from four camera views, thereby combining complementary appearance, low-light, and geometric information. Building on DAOS, we propose the Action-Object-Relation Network (AOR-Net), a vision-language framework that organizes multimodal understanding at three semantic levels: action, object, and human-object relation. Its Chain-of-action Prompting mechanism progressively grounds global motion with localized object and relation cues, while a textual prototype bank supplies corresponding semantic descriptions. A Mixture of Thoughts module then dynamically selects and integrates the most informative semantic levels and visual modalities under object-rich and object-scarce conditions. Extensive experiments show that AOR-Net consistently outperforms strong baselines on DAOS, demonstrating the effectiveness of combining heterogeneous visual sensing with structured action-object-relation semantics for fine-grained multimodal video understanding.
Figures & tables
Fig. 1: Comparison of existing driver-monitoring datasets and DAOS. DAOS provides precise object annotations, multimodal calibration, varied lighting conditions, and multi-view videos for all three visual modalities.
Fig. 2: Comparison of model architectures for driver action recognition. (Top left) Traditional action recognition models classify actions from global video features without explicit object awareness. (Top right) Object-augmented models incorporate bounding boxes or region features but typically fuse them without explicitly determining their semantic relevance. (Bottom) The proposed Action–Object–Relation Network (AOR-Net) progressively models action, object, and human–object relation cues and dynamically combines them through the Mixture-of-Thoughts (MoT) module.
Attributes
Year
Size a
Subjects
F/M b
Action Annotation
Data Modality
Lighting Conditions
Camera Views
Object annotation
Scope
Dataset
Description c
Classes
RGB
IR
Depth
Day
Night
Category
Format
Alignment
Instances
Driving-related Datasets
DrivFace [ 2 ]
2016
0.6k
4
2/2
✗
✗
640 × 480
✗
✗
N.A.
N.A.
1
✗
✗
✗
✗
DROZY [ 13 ]
2016
7h
14
11/3
✗
✗
✗
512 × 424
✗
N.A.
N.A.
1
✗
✗
✗
✗
Pandora [ 3 ]
2017
250k
22
10/12
✗
✗
1920 × 1080
✗
512 × 424
✓
N.A.
1
✗
✗
✗
✗
DriveAHead [ 14 ]
2017
10.5h
20
4/16
✗
✗
✗
512 × 424
512 × 424
✓
N.A.
1
✗
✗
✗
✗
DD-Pose [ 15 ]
2019
6h
27
6/21
✗
13
2048 × 2048
2048 × 2048
✗
✓
N.A.
2
✗
✗
✗
✗
TABLE I: Comparison of vision-based driver monitoring datasets.
Fig. 3: Statistics of our proposed DAOS dataset. (a) Action label data distribution on a logarithmic scale: at the coarse level, consecutive actions involving the same object are merged into 12 major categories; at the fine‑grained level, those are split into 36 more detailed action classes. (b) Data distribution comparison of lighting conditions for existing and our proposed dataset. Image samples for different lighting conditions, covering sunny, rainy, cloudy and night conditions.
Fig. 4: Visualization and setup of the proposed DAOS dataset. (a) Car cabin setup with four Azure Kinect devices (front, facial, left, right) capturing RGB, Depth/IR under LED and NIR illumination. (b) Multimodal multiview image samples from the four camera views (Left, Facial, Front, Right) and three sensing modalities (RGB, IR, Depth).
Fig. 5: An overview of the proposed AOR-Net framework. The model employs a Chain-of-action Prompting structure to sequentially extract and refine action-level, object-level, and relation-level features using attention in Object Prompting and Relation Prompting. These features are then aligned with their corresponding textual embeddings using a differentiable one-hot alignment. The Mixture of Thoughts (MoT) module dynamically adjusts the weights of these aligned features to produce the final action classification.
Fig. 6: Details of the Mixture of Thoughts module. The MoT module aligns these multi-level visual and textual features via a differentiable one-hot trick and dynamically fuses them to produce the final action prediction.
Methods
Validation
Test
Fine-grained
Coarse
Fine-grained
Coarse
Top-1
Top-5
Mean-1
Top-1
Top-5
Mean-1
Top-1
Top-5
Mean-1
Top-1
Top-5
Mean-1
ViViT-B [ 29 ]
42.36
74.12
28.90
48.46
87.96
39.22
39.47
69.43
30.35
43.24
85.14
36.22
TimeSFormer-B [ 28 ]
43.11
75.53
29.15
54.55
88.03
48.93
40.21
70.28
28.42
49.64
88.38
44.37
ORViT [ 44 ]
51.05
79.35
39.33
57.74
88.80
52.66
50.15
78.38
37.24
53.05
89.19
51.98
VideoMAE [ 32 ]
52.55
80.86
42.16
60.00
89.70
52.50
50.98
79.35
39.26
57.19
88.25
50.38
TABLE II: Comparison of state-of-the-art methods on fine-grained and coarse levels of DAOS Dataset.
# Max Object
( O )
Metrics
Top-1
Top-5
Mean-1
4
53.94
81.62
43.11
6
55.51
82.50
45.39
8
54.99
81.62
42.98
10
54.76
80.21
41.26
TABLE III: Ablation study of different numbers of maximum objects per frame.
Modules
Metrics
Top-1
Top-5
Mean-1
Base
53.76
81.03
42.96
+ CoA
54.69
81.62
43.76
+ CoA + MoT
55.51
82.50
45.39
TABLE IV: Ablation study of different settings of modules compared with baseline. CoA and MoT stand for CoA Prompting and Mixture of Thoughts, respectively.
Fig. 7: Ablation studies on key hyperparameters. (a) Gumbel Softmax temperature T : dashed lines indicate the Normal Softmax baseline; performance peaks at T=5 (Top-1: 55.51%, Top-5: 82.50%, Mean-1: 45.39%). (b) Relation encoder depth with d=512 : dashed lines indicate the Simple Addition baseline; the best accuracy is achieved at L=5 . (c) Relation encoder embedding dimension with L=5 : d=512 yields the optimal balance across all three metrics.
Fig. 8: Comparative visualization of attention maps from different models in the last ViT layer. Check mark stands for correct prediction and cross mark means wrong predictions.
Methods
Metrics
Top-1
Top-5
Mean-1
ORViT [ 44 ]
73.75
93.47
62.15
Open-VCLIP [ 12 ]
76.49
94.06
65.26
AOR-Net
77.67
96.22
66.90
TABLE V: Comparison results on Drive&Act [ 4 ] subset.
Modalities
Metrics
Top-1
Top-5
Mean-1
RGB
55.51
82.50
45.39
IR
53.23
81.35
43.51
Depth
54.58
83.54
48.62
RGB+IR
61.15
85.73
50.89
RGB+Depth
60.31
84.90
51.22
TABLE VI: Ablation study of single- and multi-modality model on fine-grained level data.
Fine-grained understanding of operating room (OR) activity could enable workflow-aware assistance, yet remains difficult due to clutter, occlusions, and limited sensing. The prevailing approach to model this environment is scene graphs as an interpretable representation of OR interactions. Converting their frame-wise relational predictions into temporally extended, fine-grained actions however, is challenging without explicit temporal modeling. To enable a principled temporal evaluation of current OR understanding methods, we introduce the first action-centric benchmark built on a publicly available ego-exocentric OR dataset by defining a fine-grained, multi-role action taxonomy and generating dense action segments via distillation from ground-truth scene graph state changes. Experiments on this benchmark show that current scene graph prediction methods struggle to model temporal structure, even when adding explicit modeling through Graph Neural Networks. We therefore introduce a vision-only temporal model that outperforms graph-based methods significantly when using all available egocentric video as input. Building on this model we also introduce a novel multi- to single-view feature alignment strategy that improves single-view performance on multi-role action recognition, mitigating the need for extensive egocentric video capture. Benchmark and code will be released upon acceptance.
Felix Tristram, Ege Özsoy, Christian Benz +3
Technical University of Munich, Germany · Munich Center for Machine Learning (MCML) · Carl Zeiss AG, Germany
Robust multimodal visual analytics remains challenging when heterogeneous modalities provide complementary but input-dependent evidence for decision-making.Existing multimodal learning methods mainly rely on fixed fusion modules or predefined cross-modal interactions, which are often insufficient to adapt to changing modality reliability and to capture fine-grained action cues. To address this issue, we propose a Mixture-of-Modality-Experts (MoME) framework with a Holistic Token Learning (HTL) strategy. MoME enables adaptive collaboration among modality-specific experts, while HTL improves both intra-expert refinement and inter-expert knowledge transfer through class tokens and spatio-temporal tokens. In this way, our method forms a knowledge-centric multimodal learning framework that improves expert specialization while reducing ambiguity in multimodal fusion.We validate the proposed framework on driver action recognition as a representative multimodal understanding taskThe experimental results on the public benchmark show that the proposed MoME framework and the HTL strategy jointly outperform representative single-modal and multimodal baselines. Additional ablation, validation, and visualization results further verify that the proposed HTL strategy improves subtle multimodal understanding and offers better interpretability.
Tianyi Liu, Yiming Li, Wenqian Wang +4
Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore · Singapore University of Technology and Design, 8 Somapah Rd, 487372, Singapore · The Hong Kong Polytechnic University, 6/F, Core D, Lui Che Woo Building, Hung Hom, Hong Kong SAR, China
Understanding subtle driver actions is essential for building reliable driver monitoring systems. Existing visionlanguage models (VLMs) are trained on general datasets and struggle to recognize fine distinctions in driver behaviors. This paper addresses this limitation by creating a detailed natural language version of the Drive&Act dataset. We evaluate three VLMs on our new benchmark using LLM-based scoring methods. Their performance on the new benchmark shows that they cannot reliably generate accurate fine-grained driver activity descriptions. Based on the labeled Drive&Act dataset we create a new Drive&Act description dataset containing finegrained descriptions to train VLMs on driver activity understanding. Cross dataset evaluation on the Driver Monitoring Dataset (DMD) shows that the VLM fine-tuned on our new Drive&Act description dataset generalizes well to actions in the DMD dataset. The VLM fine-tuned on our Drive&Act description dataset achieves an ACCR score of 76 outperforming the zero-shot VLM baseline with an ACCR score of 66. These findings demonstrate that adapting VLMs with richly described driver actions can significantly improve their ability to interpret driver behavior while also highlighting the need for more diverse datasets to support broader generalization in future applications. Our Drive&Act description dataset and code will be publicly available on GitHub.
David J. Lerch, Sarath Mulugurthi, Manuel Martin +2
Fraunhofer IOSB, Karlsruhe, Germany · Technische Hochschule Ingolstadt, Ingolstadt, Germany · Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany