DAOS: A Multimodal In-cabin Behavior Monitoring with Driver Action-Object Synergy Dataset
Authors: Yiming Li, Dan Lin, Wenfei Liang, Tianyi Liu, Chen Cai, Wenqian Wang, Kim-Hui Yap, Liang Zhao, Bingbing Li
Organizations: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore · College of Computer Science and Technology, Harbin Engineering University · Singapore University of Technology and Design (SUTD), Singapore · Continental Automotive Singapore Pte. Ltd.
Multimodal action understanding requires complementary sensory observations to be connected with structured semantic knowledge. This is particularly challenging in vehicle cabins, where fine-grained actions exhibit similar upper-body motion, relevant objects may be small or occluded, and appearance varies substantially across illumination conditions. Existing in-cabin datasets, however, rarely provide synchronized visual modalities together with dense object annotations and explicit action--object alignment, limiting the study of object-grounded multimodal recognition. To address this gap, we introduce the Driver Action with Object Synergy (DAOS) dataset, containing 9,787 video clips, 36 fine-grained actions, 15 object categories, and more than 2.5 million annotated object instances. DAOS provides synchronized RGB, infrared, and depth observations from four camera views, thereby combining complementary appearance, low-light, and geometric information. Building on DAOS, we propose the Action-Object-Relation Network (AOR-Net), a vision-language framework that organizes multimodal understanding at three semantic levels: action, object, and human-object relation. Its Chain-of-action Prompting mechanism progressively grounds global motion with localized object and relation cues, while a textual prototype bank supplies corresponding semantic descriptions. A Mixture of Thoughts module then dynamically selects and integrates the most informative semantic levels and visual modalities under object-rich and object-scarce conditions. Extensive experiments show that AOR-Net consistently outperforms strong baselines on DAOS, demonstrating the effectiveness of combining heterogeneous visual sensing with structured action-object-relation semantics for fine-grained multimodal video understanding.
Figures & tables
Fig. 1: Comparison of existing driver-monitoring datasets and DAOS. DAOS provides precise object annotations, multimodal calibration, varied lighting conditions, and multi-view videos for all three visual modalities.
Fig. 2: Comparison of model architectures for driver action recognition. (Top left) Traditional action recognition models classify actions from global video features without explicit object awareness. (Top right) Object-augmented models incorporate bounding boxes or region features but typically fuse them without explicitly determining their semantic relevance. (Bottom) The proposed Action–Object–Relation Network (AOR-Net) progressively models action, object, and human–object relation cues and dynamically combines them through the Mixture-of-Thoughts (MoT) module.
Attributes
Year
Size a
Subjects
F/M b
Action Annotation
Data Modality
Lighting Conditions
Camera Views
Object annotation
Scope
Dataset
Description c
Classes
RGB
IR
Depth
Day
Night
Category
Format
Alignment
Instances
Driving-related Datasets
DrivFace [ 2 ]
2016
0.6k
4
2/2
✗
✗
640 × 480
✗
✗
N.A.
N.A.
1
✗
✗
✗
✗
DROZY [ 13 ]
2016
7h
14
11/3
✗
✗
✗
512 × 424
✗
N.A.
N.A.
1
✗
✗
✗
✗
Pandora [ 3 ]
2017
250k
22
10/12
✗
✗
1920 × 1080
✗
512 × 424
✓
N.A.
1
✗
✗
✗
✗
DriveAHead [ 14 ]
2017
10.5h
20
4/16
✗
✗
✗
512 × 424
512 × 424
✓
N.A.
1
✗
✗
✗
✗
DD-Pose [ 15 ]
2019
6h
27
6/21
✗
13
2048 × 2048
2048 × 2048
✗
✓
N.A.
2
✗
✗
✗
✗
TABLE I: Comparison of vision-based driver monitoring datasets.
Fig. 3: Statistics of our proposed DAOS dataset. (a) Action label data distribution on a logarithmic scale: at the coarse level, consecutive actions involving the same object are merged into 12 major categories; at the fine‑grained level, those are split into 36 more detailed action classes. (b) Data distribution comparison of lighting conditions for existing and our proposed dataset. Image samples for different lighting conditions, covering sunny, rainy, cloudy and night conditions.
Fig. 4: Visualization and setup of the proposed DAOS dataset. (a) Car cabin setup with four Azure Kinect devices (front, facial, left, right) capturing RGB, Depth/IR under LED and NIR illumination. (b) Multimodal multiview image samples from the four camera views (Left, Facial, Front, Right) and three sensing modalities (RGB, IR, Depth).
Fig. 5: An overview of the proposed AOR-Net framework. The model employs a Chain-of-action Prompting structure to sequentially extract and refine action-level, object-level, and relation-level features using attention in Object Prompting and Relation Prompting. These features are then aligned with their corresponding textual embeddings using a differentiable one-hot alignment. The Mixture of Thoughts (MoT) module dynamically adjusts the weights of these aligned features to produce the final action classification.
Fig. 6: Details of the Mixture of Thoughts module. The MoT module aligns these multi-level visual and textual features via a differentiable one-hot trick and dynamically fuses them to produce the final action prediction.
Methods
Validation
Test
Fine-grained
Coarse
Fine-grained
Coarse
Top-1
Top-5
Mean-1
Top-1
Top-5
Mean-1
Top-1
Top-5
Mean-1
Top-1
Top-5
Mean-1
ViViT-B [ 29 ]
42.36
74.12
28.90
48.46
87.96
39.22
39.47
69.43
30.35
43.24
85.14
36.22
TimeSFormer-B [ 28 ]
43.11
75.53
29.15
54.55
88.03
48.93
40.21
70.28
28.42
49.64
88.38
44.37
ORViT [ 44 ]
51.05
79.35
39.33
57.74
88.80
52.66
50.15
78.38
37.24
53.05
89.19
51.98
VideoMAE [ 32 ]
52.55
80.86
42.16
60.00
89.70
52.50
50.98
79.35
39.26
57.19
88.25
50.38
TABLE II: Comparison of state-of-the-art methods on fine-grained and coarse levels of DAOS Dataset.
# Max Object
( O )
Metrics
Top-1
Top-5
Mean-1
4
53.94
81.62
43.11
6
55.51
82.50
45.39
8
54.99
81.62
42.98
10
54.76
80.21
41.26
TABLE III: Ablation study of different numbers of maximum objects per frame.
Modules
Metrics
Top-1
Top-5
Mean-1
Base
53.76
81.03
42.96
+ CoA
54.69
81.62
43.76
+ CoA + MoT
55.51
82.50
45.39
TABLE IV: Ablation study of different settings of modules compared with baseline. CoA and MoT stand for CoA Prompting and Mixture of Thoughts, respectively.
Fig. 7: Ablation studies on key hyperparameters. (a) Gumbel Softmax temperature T : dashed lines indicate the Normal Softmax baseline; performance peaks at T=5 (Top-1: 55.51%, Top-5: 82.50%, Mean-1: 45.39%). (b) Relation encoder depth with d=512 : dashed lines indicate the Simple Addition baseline; the best accuracy is achieved at L=5 . (c) Relation encoder embedding dimension with L=5 : d=512 yields the optimal balance across all three metrics.
Fig. 8: Comparative visualization of attention maps from different models in the last ViT layer. Check mark stands for correct prediction and cross mark means wrong predictions.
Methods
Metrics
Top-1
Top-5
Mean-1
ORViT [ 44 ]
73.75
93.47
62.15
Open-VCLIP [ 12 ]
76.49
94.06
65.26
AOR-Net
77.67
96.22
66.90
TABLE V: Comparison results on Drive&Act [ 4 ] subset.
Modalities
Metrics
Top-1
Top-5
Mean-1
RGB
55.51
82.50
45.39
IR
53.23
81.35
43.51
Depth
54.58
83.54
48.62
RGB+IR
61.15
85.73
50.89
RGB+Depth
60.31
84.90
51.22
TABLE VI: Ablation study of single- and multi-modality model on fine-grained level data.
Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore · Singapore University of Technology and Design, 8 Somapah Rd, 487372, Singapore · The Hong Kong Polytechnic University, 6/F, Core D, Lui Che Woo Building, Hung Hom, Hong Kong SAR, China