Multimodal Large Language Models (MLLMs) have advanced open-world action understanding and can be adapted as generative classifiers for closed-set settings by autoregressively generating action labels as text. However, this approach is inefficient, and shared subwords across action labels introduce semantic overlap, leading to ambiguity in generation. In contrast, discriminative classifiers learn task-specific representations with clear decision boundaries, enabling efficient one-step classification without autoregressive decoding. We first compare generative and discriminative classifiers with MLLMs for closed-set action understanding, revealing the superior accuracy and efficiency of the latter. To bridge the performance gap, we design strategies that elevate generative classifiers toward performance comparable with discriminative ones. Furthermore, we show that generative modeling can complement discriminative classifiers, leading to better performance while preserving efficiency. To this end, we propose Generation-Assisted Discriminative~(GAD) classifier for closed-set action understanding. GAD operates only during fine-tuning, preserving full compatibility with MLLM pretraining. Extensive experiments on temporal action understanding benchmarks demonstrate that GAD improves both accuracy and efficiency over generative methods, achieving state-of-the-art results on four tasks across five datasets, including an average 2.5% accuracy gain and 3x faster inference on our largest COIN benchmark.
Figures & tables
Figure 1: T-SNE plot comparing feature spaces of generative and discriminative classifiers on CrossTask dataset for actions sharing the verb ‘add’. Generative feature - mean of output token features; Discriminative feature - the learnable token feature.
Figure 2: Comparison between different architectures for downstream video-related recognition tasks: (1) Generative classifier: treating action labels as free text. (2) Discriminative classifier: learning an extra representation for downstream tasks. (3) Generation-assisted discriminative (GAD) classifier: learning an extra representation that is regularized through task-related generation.
LLM
Model
THUMOS’14
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
S-F1 / P-F1
FPS
S-F1 / P-F1
FPS
S-F1 / P-F1
FPS
S-F1 / P-F1
FPS
Llama3.2- 1B-Instruct
Gen
56.9 / 38.8
38.3
46.8 / 31.7
44.0
16.7 / 13.9
28.8
8.9 / 3.4
17.8
Disc
57.8 / 40.1
58.0
48.8 / 34.0
59.4
23.2 / 19.3
51.1
10.6 / 4.1
53.6
Qwen2.5- 0.5B-Instruct
Gen
55.8 / 38.9
30.2
44.0 / 28.6
36.3
16.2 / 13.4
26.6
8.8 / 3.2
12.3
Disc
57.3 / 39.6
48.8
45.3 / 29.6
52.8
22.0 / 18.3
48.6
9.9 / 3.4
52.1
Table 1: Generative (Gen) vs. discriminative (Disc) classifier on OAD tasks. The runtime analysis is performed on a single NVIDIA RTX A5000 GPU.
Figure 4Figure 5
Model
OAD Benchmark (segment-F1 / point-F1)
COIN (Top-1 acc.)
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Step
Next
Task
Disc
48.8 / 34.0
23.2 / 19.3
10.6 / 4.1
64.1
50.1
92.8
GAD (label_2stage)
41.8 / 26.7
9.3 / 6.7
5.9 / 1.7
35.9
28.0
85.5
GAD (label_joint)
49.1 / 33.9
23.6 / 19.6
10.8 / 4.1
64.4
50.4
93.2
GAD (context)
50.3 / 34.5
24.1 / 20.1
11.0 / 4.3
65.3
51.4
93.5
Table 4: Comparison with GAD variants with Llama3.2-1B-Instruct.
Model
OAD Benchmark. (frame acc./segment F1/point F1)
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Disc
81.7/48.8/34.0
34.8/23.2/19.3
33.9/10.6/4.1
GAD
81.8/50.3/34.5
35.1/24.1/20.1
34.4/11.0/4.3
w/o [CLS]
81.5/49.2/33.8
34.6/20.7/17.7
33.7/8.6/3.6
GAD _prev_disc
80.8/48.2/31.6
34.8/23.6/19.3
33.2/10.1/3.8
GAD _prev_disc+
80.9/49.5/32.7
34.7/23.0/19.6
33.3/10.6/4.0
Table 5: Ablation studies of our GAD classifier on OAD tasks.
Model
OAD Benchmark (segment-F1 / point-F1)
THUMOS’14
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Testra ( Zhao & Krähenbühl, 2022 )
43.0 / 32.0
48.4 / 33.8
16.5 / 14.7
8.7 / 3.5
MAT ( Wang et al., 2023 )
49.4 / 34.2
49.7 / 34.2
17.5 / 15.5
9.5 / 3.8
CMeRT ( Pang et al., 2025 )
48.9 / 34.6
50.1 / 34.3
17.7 / 15.8
9.7 / 3.9
Disc (Llama3.2-1B-Instruct)
57.8 / 40.1
48.8 / 34.0
23.2 / 19.3
10.6 / 4.1
GAD (Llama3.2-1B-Instruct)
58.1 / 40.2
50.3 / 34.5
24.1 / 20.1
11.0 / 4.3
Table 6: Comparison with SOTA on OAD tasks.
Model
COIN Benchmark (Top-1 acc.)
Step
Next
Task
VideoTaskGraph ( Ashutosh et al., 2023 )
57.2
40.2
90.5
Videollm-online-8B ( Chen et al., 2024a )
63.1
49.1
92.7
Videollm-MoD-8B ( Wu et al., 2024 )
63.4
49.7
92.8
StreamMind-8B ( Ding et al., 2025 )
63.7
49.9
93.2
Disc (Llama3.2-1B-Instruct)
64.1
50.1
92.8
Table 7: Comparison with SOTA on COIN Benchmark.
Figure 5: Qualitative results on the step recognition for Generative (Gen) vs. discriminative (Disc) classifier. Red text denotes false predictions and green correct predictions. Callouts display GAD generation outputs.
Appendix figures & tables16 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: T-SNE plot comparing feature spaces of the generative and discriminative classifier on CrossTask dataset for actions sharing the verb ‘add’. Generative feature is derived from token representations using four strategies: mean, max, first, or last token. Discriminative feature is the representation of the learnable token.
Task
Query
Online action detection
What is the action in the last frame?
Step recognition
What is the action in the video?
Step forecasting
What is the next action in the video?
Task recognition
What is the overall activity in the video?
Appendix
Table 8: Task-specific queries.
Datasets
long-term (s)
short-term (s)
sample rate
visual feature@FPS
THUMOS’14
128
32
1
ResNet50@4
CrossTask
128
20
1
DINOV2@1
EPIC-Kitchens-100
128
20
1
TSN@4
Ego4DGoalStep
128
16
1
DINOV2@1
Appendix
Table 9: Hyperparameters for OAD setting.
Figure 14Figure 15Figure 16
Model
OAD Benchmark. (frame acc./segment F1/point F1)
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Gen
81.3/46.8/31.7
32.9/16.7/13.9
32.9/8.9/3.4
Disc
81.7/48.8/34.0
34.8/23.2/19.3
33.9/10.6/4.1
GAD _gen
81.5/46.8/31.8
32.1/16.0/13.2
31.4/7.8/3.0
GAD _emb
81.6/49.1/33.9
34.8/23.6/19.6
34.0/10.8/4.1
GAD _seq-g_gen
81.2/46.3/31.3
33.3/17.1/14.2
32.5/8.9/3.3
Appendix
Table 10: Discriminative and generative performance in the proposed GAD classifier, where generative modeling is for labeling task same as the discriminative learning.
Model
OAD Benchmak. DScore
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Gen
0.76
1.3
1.8
Disc
0.66
0.79
1.5
GAD
0.67
0.80
1.49
Appendix
Table 11: Comparison of misclassification diversity for generative (Gen), discriminative (Disc), and generation-assisted discriminative (GAD) classifiers.
Model
OAD Benchmark. (frame acc./segment F1/point F1)
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Disc
81.7/48.8/34.0
34.8/23.2/19.3
33.9/10.6/4.1
GAD
81.8/50.3/34.5
35.1/24.1/20.1
34.4/11.0/4.3
GAD _prev_disc
80.8/48.2/31.6
34.8/23.6/19.3
33.2/10.1/3.8
GAD _prev_disc+
80.9/49.5/32.7
34.7/23.0/19.6
33.3/10.6/4.0
GAD _next
81.6/49.8/34.3
34.6/23.7/19.8
34.1/10.8/4.2
Appendix
Table 12: Ablation studies of our GAD classifier on OAD tasks.
Figure 7: Quantitative analysis of discriminative ouputs (Disc) and generative outputs (Gen & Gen_ex). Larger areas indicate higher occurrence. Checks and crosses denote correct and incorrect outputs, respectively.
Figure 8: Qualitative results on the OAD and step recognition tasks. Gen - generative classifier, Disc- discriminative classifier. Text in red denotes false predictions, while in green represents correct predictions. Callouts display the generation outputs produced by our GAD. Bar charts in the bottom show predictions for streaming videos.
Model
OAD Benchmark. (frame acc./segment F1/point F1)
THUMOS’14
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Testra ( Zhao & Krähenbühl, 2022 )
76.9/43.0/32.0
81.2/48.4/33.8
34.7/16.5/14.7
31.7/8.7/3.5
MAT ( Wang et al., 2023 )
78.4/49.4/34.2
81.4/49.7/34.2
35.0/17.5/15.5
32.9/9.5/3.8
CMeRT ( Pang et al., 2025 )
78.2/48.9/34.6
81.3/50.1/34.3
35.2 /17.7/15.8
32.8/9.7/3.9
Gen (Llama3.2-1B-Instruct)
78.6/56.9/38.8
81.3/46.8/31.7
32.9/16.7/13.9
32.8/8.9/3.4
Disc (Llama3.2-1B-Instruct)
78.5/57.8/40.1
81.7/48.8/34.0
34.8/23.2/19.3
33.9/10.6/4.1
Appendix
Table 13: Comparison with SOTA on OAD tasks.
Model
OAD Benchmark (segment-F1 / point-F1)
CrossTask
EPIC-Kitchens-100
Ego4DGoalStep
Gen
46.8 / 31.7
16.7 / 13.9
8.9 / 3.4
Testra [1]
48.4 / 33.8
16.5 / 14.7
8.7 / 3.5
MAT [2]
49.7 / 34.2
17.5 / 15.5
9.5 / 3.8
CMeRT [3]
50.1 / 34.3
17.7 / 15.8
9.7 / 3.9
Disc
48.8 / 34.0
23.2 / 19.3
10.6 / 4.1
Appendix
Table 14: Comparison with SOTA on OAD tasks.
Model
Step
Next
GAD (fine-tuned)
67.3
51.6
Videollm-online-8B (open-ended)
4.8
3.5
Qwen2.5-VL-7B (open-ended)
16.1
8.9
Qwen2.5-VL-7B (categories in prompt)
11.9
6.5
Appendix
Table 15: Zero-shot performance of existing MLLMs on COIN dataset.
Epoch 0
Epoch 1
Epoch 2
Epoch 3
Epoch 4
Epoch 5
Memorization_ratio (on Breakfast)
0.0
98.1
99.6
99.8
99.9
99.9
Accuracy on COIN (test set)
8.3
77.0
85.3
91.3
92.8
92.7
Accuracy on Breakfast (unseen)
6.3
4.5
3.5
3.6
3.6
3.6
Appendix
Table 16: Performance-generalization trade-off on our generative baseline.
Figure 9: Confusion matrix comparison on CrossTask for actions shared verb ’add’. Generative classifier incurs more diverse misclassifications.
Micro-actions are subtle, short, low-amplitude body movements, such as a fidgeting hand or a slight head tilt, that humans perform with little conscious intent yet that reliably leak emotional and psychological state. Understanding them goes beyond assigning a label: a model must also describe which body parts move and reason, faithfully, about why a clip warrants a particular fine-grained category. We present the training-free, prompt-only system that won first place in the fine-grained understanding track (MA-Bench) of the MAC~2026 Micro-Action Challenge, where both fine-tuning and ground-truth supervision are disallowed. Built entirely upon frozen multimodal large language models (MLLMs), the system dynamically routes each of the eight sub-tasks to the MLLM empirically best suited for that task: a discriminative MLLM for closed-ended recognition tasks and a generative MLLM for open-ended description and reasoning tasks. This architecture achieves a statistically significant performance advantage on open-ended tasks, attaining an average score of 2.68 (on a five-point scale) compared to 1.44 for the second-best approach.
With the rapid development of Multimodal Large Language Models (MLLMs), their potential in Micro-Action understanding, a vital role in human emotion analysis, remains unexplored due to the absence of specialized benchmarks. To tackle this issue, we present MA-Bench, a benchmark comprising 1,000 videos and a three-tier evaluation architecture that progressively examines micro-action perception, relational comprehension, and interpretive reasoning. MA-Bench contains 12,000 structured question-answer pairs, enabling systematic assessment of both recognition accuracy and action interpretation. The results of 23 representative MLLMs reveal that there are significant challenges in capturing motion granularity and fine-grained body-part dynamics. To address these challenges, we further construct MA-Bench-Train, a large-scale training corpus with 20.5K videos annotated with structured micro-action captions for fine-tuning MLLMs. The results of Qwen3-VL-8B fine-tuned on MA-Bench-Train show clear performance improvements across micro-action reasoning and explanation tasks. Our work aims to establish a foundation benchmark for advancing MLLMs in understanding subtle micro-action and human-related behaviors. Project Page: https://MA-Bench.github.io
Kun Li, Jihao Gu, Fei Wang +3
CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology +2
Which multimodal model should we use for classification? Previous studies suggest that the answer lies in CLIP-like contrastive Vision-Language Models (VLMs), due to their remarkable performance in zero-shot classification. In contrast, Large Multimodal Models (LMM) are more suitable for complex tasks. In this work, we argue that this answer overlooks an important capability of LMMs: in-context learning. We benchmark state-of-the-art LMMs on diverse datasets for closed-world classification and find that, although their zero-shot performance is lower than CLIP's, LMMs with a few in-context examples can match or even surpass contrastive VLMs with cache-based adapters, their "in-context" equivalent. We extend this analysis to the open-world setting, where the generative nature of LMMs makes them more suitable for the task. In this challenging scenario, LMMs struggle whenever provided with imperfect context information. To address this issue, we propose CIRCLE, a simple training-free method that assigns pseudo-labels to in-context examples, iteratively refining them with the available context itself. Through extensive experiments, we show that CIRCLE establishes a robust baseline for open-world classification, surpassing VLM counterparts and highlighting the potential of LMMs to serve as unified classifiers, and a flexible alternative to specialized models.