Organizations: Nanyang Technological University, 50 Nanyang Avenue, 639798, Singapore · Singapore University of Technology and Design, 8 Somapah Rd, 487372, Singapore · The Hong Kong Polytechnic University, 6/F, Core D, Lui Che Woo Building, Hung Hom, Hong Kong SAR, China
Robust multimodal visual analytics remains challenging when heterogeneous modalities provide complementary but input-dependent evidence for decision-making.Existing multimodal learning methods mainly rely on fixed fusion modules or predefined cross-modal interactions, which are often insufficient to adapt to changing modality reliability and to capture fine-grained action cues. To address this issue, we propose a Mixture-of-Modality-Experts (MoME) framework with a Holistic Token Learning (HTL) strategy. MoME enables adaptive collaboration among modality-specific experts, while HTL improves both intra-expert refinement and inter-expert knowledge transfer through class tokens and spatio-temporal tokens. In this way, our method forms a knowledge-centric multimodal learning framework that improves expert specialization while reducing ambiguity in multimodal fusion.We validate the proposed framework on driver action recognition as a representative multimodal understanding taskThe experimental results on the public benchmark show that the proposed MoME framework and the HTL strategy jointly outperform representative single-modal and multimodal baselines. Additional ablation, validation, and visualization results further verify that the proposed HTL strategy improves subtle multimodal understanding and offers better interpretability.
Figures & tables
Figure 1: In-cabin action recognition presents subtle motion pattern and complicated illumination variation, making fine-grained multimodal evidence sparse and unevenly reliable. These challenges therefore requires robust multimodal visual analytics to adaptively integrate knowledge from different sources.
Figure 2: Overview of our proposed Mixture-of-Modality-Experts (MoME) framework enhanced by the Holistic Token Learning (HTL) strategy.
Methods
Modality
Mean-1 Accuracy
Top-1 Accuracy
Pose ( Martin et al., 2019 )
3DS
-
55.17
C3D ( Tran et al., 2015 )
NIR
-
43.41
P3D ( Qiu et al., 2017 )
NIR
-
45.32
CTA-NET ( Wharton et al., 2021 )
RGB
-
65.25
I3D ( Carreira and Zisserman, 2017 )
NIR
-
60.80
Depth
-
60.52
Table 1: The benchmark results of MoME compared with single- and multimodal DAR baselines on Drive&Act ( Martin et al., 2019 ) dataset. “LF” denotes basic late fusion-based multimodal DAR variants implemented with different backbones
Method
Split 0
Split 1
Split 2
Average
Mean-1 Acc.
Top-1 Acc.
Mean-1 Acc.
Top-1 Acc.
Mean-1 Acc.
Top-1 Acc.
Mean-1 Acc.
Top-1 Acc.
Early-Fusion
54.19
72.86
52.24
65.70
43.21
63.34
49.88
67.30
Late-Fusion
65.34
84.29
60.16
77.36
59.25
75.19
61.58
78.53
Multifuser ( Wang et al., 2024b )
71.95
85.42
72.07
82.85
68.00
78.89
70.67
82.39
Ours
74.32
88.35
7 3.54
82.91
73.60
80.56
73.82
83.94
Table 2: Quantitative comparison across different fusion methods based on the UniformerV2 ( Li et al., 2023a ) backbone.
Structure
Strategy
Mean-1 Acc.
Top-1 Acc.
MoE
None
72.14
87.94
MoME
None
73.48
88.03
MoME
MoDE ( Xie et al., 2024 )
72.98
88.17
MoME
HTL
74.32
88.35
Table 3: Ablation study results of MoME design and comparison between HTL and MoDE ( Xie et al., 2024 )
Methods
Mean-1 Acc.
Top-1 Acc.
Basic Fine-tuning
63.02
83.08
Deep supervision ( Lee et al., 2015 )
66.10
83.71
Self-distillation ( Sultana et al., 2022 )
63.96
83.44
Holistic Token Learning -
69.18
84.16
Table 4: HTL can also be applied to signal-modality models. The single modality variant HTL - can contribute more performance improvement than other self-distillation-like methods.
Figure 3: HTL can guide the model to concentrate on subtle spatio-temporal cues when handling challenging samples.