Zero-MELO: Test-Time Evidence Calibration with Multimodal LLMs for Zero-Shot Micro-Gesture Recognition

Chengyan Wang; Hanliang Xie; Yueyi Yang; Haoyu Chen

cs.CVAug 14, 2026

Zero-MELO: Test-Time Evidence Calibration with Multimodal LLMs for Zero-Shot Micro-Gesture Recognition

Authors: Chengyan Wang, Hanliang Xie, Yueyi Yang, Haoyu Chen

Organizations: CMVS, University of Oulu Oulu, Finland · Peking University Beijing, China

Abstract

While Multimodal Large Language Models (MLLMs) excel in general video understanding, their capability in fine-grained and motion-centric tasks remains limited. This limitation is particularly critical in micro-gesture recognition (MGR), where micro-gestures (MGs) - subtle, short-duration, and spatially localized human movements - serve as key discriminative signals for implicit affective analysis, yet are easily neglected following common prompting practices. Although MGR has been intensively studied by many discriminative approaches, the use of MLLMs for MGR is underexplored, with notably poor performance. We hypothesize that the motion-sensitive representation ability of MLLMs is constrained by their inherent single-pass forward inference, which can be substantially enhanced through carefully designed test-time guidance. Motivated by this, building on our prior findings regarding temporal insensitivity in Video LLMs, we diagnose zero-shot MGR errors in the Negative Log-Likelihood (NLL) space. We observe that MLLMs suffer from two bottlenecks: 1) insufficient localized evidence and 2) severe score biases driven by language and motion-agnostic appearances. Thus, we propose a novel test-time evidence calibration framework that improves both reasoning details and prediction reliability. Specifically, we introduce a tree search mechanism to progressively acquire localized, fine-grained visual evidence, coupled with a test-time calibration module to mitigate score biases. The multi-cue fusion module then integrates evidence from multiple cues without relying on a single cue for final prediction. Our framework achieves mean-class accuracies of 26.84% on iMiGUE and 22.10% on MA-52, significantly outperforming the Qwen2.5-VL baseline, which produces 16.15% and 10.20%, respectively. The code will be available at https://zero-melo.github.io/Zero-MELO.

Figures & tables

Figure 1. Comparison between existing methods and Zero-MELO. Given the input video clips and question, the target is to predict the correct label in MGR. Compared to the single-pass forward inference of other MLLMs, Zero-MELO can utilize the relevant local evidence and predict the correct label via calibration at inference time.

Figure 2. The overall algorithm of Zero-MELO. It mainly consists of three key modules: the cue-guided search (Sec. 3.3 ), the test-time calibration (Sec. 3.4 ), and the multi-cue fusion (Sec. 3.5 ). The global video input will first be fed to the MLLM for the global inference. Then, the model will acquire local evidence via the cue-guided tree search. Next, the test-time calibration will be conducted, and the calibrated scores will be yielded. Based on the global and local calibrated scores, the multi-cue fusion contributes to the final prediction.

Panel of Figure 3 — Figure 3. The TAM visualization of a failure case for the gesture Pressing lips . The wrong prediction is Bulging face, deep breath . We show the token-wise activation maps for three successive generated tokens during free-form generation in (a), (b), and (c), respectively. Although the model suggests the GT in Top-5, the highlighted regions are not consistently grounded on the lip area or the most discriminative motion phase, indicating that the naive single-pass MLLM fails to capture compact motion-centric local evidence.

Panel of Figure 4 — Figure 4. Failure analysis on iMiGUE of the prompt-based free-form generation. (a) The comparison of distributions of Qwen2.5-VL predictions and the ground-truths (GTs) of iMiGUE . Due to the space limitations, only the appearing frequency of labels is larger than 1% in either GT or Top-1 prediction is listed. (b) The comparison of the severity of the prediction-collapse across different vanilla MLLMs .

Prompt-based Zero-shot
Method	iMiGUE			MA-52
Method	mCA@1	mCA@5	MF1	mCA@1	mCA@5	MF1
GPT-4o mini ( OpenAI, 2024 )	0.1113	0.3392	0.0699	0.0576	0.2362	0.0363
Gemini2.5-Flash ( Comanici et al., 2025 )	0.1917	0.3689	0.1367	0.1666	0.3072	0.1217
MiMo-VL-7B ( Team et al., 2025 )	0.2445	0.5643	0.1506	0.1869	0.3814	0.1042
InternVL3.5-8B ( Wang et al., 2025c )	0.2034	0.4358	0.0927	0.1796	0.3677	0.0914

Table 1. Zero-shot performance comparison on iMiGUE testset and MA-52 valid set.

Method	mCA@1	mCA@5	MF1
VQA	0.177	0.4201	0.0842
Label Scoring	0.0899	0.2545	0.0623
Ours	0.2210	0.4359	0.1221

Table 2. Comparison between our method and the SFT method on MA-52 dataset.

Figure 5. Case study of Zero-MELO. Each panel shows the global video clip, the zoomed-in view of the most supportive cue, and the trajectory of the prediction Top-1 across global inference, calibration, and final multi-cue fusion. The (a) , (b) , (c) correspond to the successful samples, while (d) is the failure case.

Method	mCA@1	mCA@5	MF1
Vanilla (ablation format)	0.0357	0.2084	0.0094
+ TS	0.0358	0.2629	0.0100
+ Cali (Blank)	0.2040	0.4943	0.0855
+ Cali (Aggregate)	0.2363	0.5436	0.1097
+ TS + Cali (Aggregate, Single-Cue)	0.2244	0.5342	0.1204
+ TS + Cali (Blank)	0.2131	0.5078	0.0924

Table 3. Ablation study of different components of Zero-MELO on iMiGUE dataset. Vanilla (ablation format) uses the unified paired-input format adopted by all component ablations, differing from Table 1 . TS, Cali, and F denote tree search, calibration, and fusion; Max and Average denote lowest-NLL cue selection and uniform cue averaging.

Appendix figures & tables21 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Model	Total clips	Clips with mismatched labels
iMiGUE	GPT-4o mini	4534	36 (0.79%)
iMiGUE	Gemini2.5-Flash	4534	280 (6.18%)
MA-52	GPT-4o mini	5586	441 (7.89%)
MA-52	Gemini2.5-Flash	5586	195 (3.49%)

Appendix

Table 4. Simple statistics of exact-name mismatches in free-form generation. A clip is counted if at least one returned top-

k

label is not an exact-string-matched label.

Dataset	Model	Representative misaligned labels
iMiGUE	GPT-4o mini	Touching or scratching jaw; Touching forehead; Touching or adjusting hair
iMiGUE	Gemini2.5-Flash	Holding back arms; Adjusting tie; Touch jaw; Touchng jaw
MA-52	GPT-4o mini	Bowling head; Scratching or touching head; Raising up; Touching face
MA-52	Gemini2.5-Flash	Scratching face; Touching face; Touching chest; Tipping head

Appendix

Table 5. Representative misaligned labels observed in prompt-based free-form generation.

Parameter	Qwen2.5-VL	InternVL3.5	MiMo-VL
$\tau_{d}$	0.9	0.7	1.1
$\pi_{g}$	0.5	0.5	0.5
$\pi_{c}$	0.5	0.5	0.5
$\alpha_{y}$	0	0.85	0.97
$w_{head}$	0.2	1.0	1.5
$w_{hand}$	1.0	0.8	1.12

Appendix

Table 6. Parameters of multi-cue fusion in iMiGUE.

Parameter	Qwen2.5-VL
$\tau_{d}$	0.5
$\pi_{g}$	0.5
$\pi_{c}$	0.5
$\alpha_{y}$	0.25
$w_{head}$	0.7
$w_{upper-limbs}$	0.1

Appendix

Table 7. Parameters of multi-cue fusion in MA-52.

Label $y$	Label-Conditioned Cue Set $\mathcal{C}(y)$	Label $y$	Label-Conditioned Cue Set $\mathcal{C}(y)$
Turtle neck	$\{\texttt{neck}\}$	Folding arms	$\{\texttt{body-hand},\texttt{arms}\}$
Bulging face, deep breath	$\{\texttt{face}\}$	Dusting off clothes	$\{\texttt{body-hand},\texttt{arms}\}$
Touching hat	$\{\texttt{head-hand}\}$	Putting arms behind body	$\{\texttt{arms}\}$
Touching or scratching head	$\{\texttt{head-hand}\}$	Moving torso	$\{\texttt{torso}\}$
Touching or scratching forehead	$\{\texttt{head-hand},\texttt{face}\}$	Sitting straightly	$\{\texttt{torso}\}$
Covering face	$\{\texttt{head-hand},\texttt{face}\}$	Scratching or touching arms	$\{\texttt{arms},\texttt{hands}\}$

Appendix

Table 8. Label-conditioned cue sets

\mathcal{C}(y)

for iMiGUE.

Panel of Figure 6 — Figure 6. A failure case of free-form generation, analogous to Fig. 3. The GT of this sample is Pressing lips . (a) : When only the original video clip is provided, the vanilla MLLM predicts Touching jaw . (b) : When additional local evidence of the mouth is provided, the vanilla MLLM will predict Pressing lips .

Figure 7. Class-wise effect of blank and lowrank calibration in NLL-based label scoring. Left: the ground-truth class share of samples that are misclassified by the vanilla model ( vanilla wrong ). Right: per-class calibration effect, where fix means that a sample is originally misclassified by the vanilla model but becomes correct after the corresponding calibration method, and harm means that a sample is originally correct under the vanilla model but becomes wrong after calibration. The rates are normalized by class size.

Figure 8. Sample-level overlap of the two calibrations on vanilla-wrong samples. Both means that the sample is corrected by both blank and lowrank calibration; Blank only and Lowrank only mean that only one calibration corrects it; Neither means that neither method corrects it. Each cell reports the corresponding percentage within the vanilla-wrong subset.

Figure 9. Per-class overlap composition on vanilla-wrong samples. For each class, the bar is decomposed into four groups: Both , Blank only , Lowrank only , and Neither , using the same definitions as in Fig. 8 . Thus, each bar shows how the two calibrations overlap, or fail to overlap, when correcting errors in that class.

Panel of Figure 10 — Figure 10. Bias diagnosis by overlap group. The four groups ( Both , Blank only , Lowrank only , and Neither ) are defined in the same way as in Fig. 8 . $A_{\mathrm{blank}}$ measures language-prior bias revealed by the blank reference, while $A_{\mathrm{lowrank}}$ measures appearance bias revealed by the lowrank reference relative to the blank reference.

Method	mCA@1	mCA@5	MF1
Qwen2.5-VL	0.1667	0.3561	0.0563
Qwen2.5-VL w/ label list	0.0523	0.2524	0.0072

Appendix

Table 9. Comparison of NLL-based Qwen2.5-VL with and without the label list in the prompt.

Dataset	Model	Mode	mCA@1	#Top-1 labels	Dominant ratio	$H_{\mathrm{norm}}$	JSD	Dominant Top-1 label
iMiGUE	Qwen2.5-VL-7B	Prompt-based	0.1615	19	37.6%	0.503	0.478	Touching or scratching forehead
iMiGUE	Qwen2.5-VL-7B	Vanilla NLL	0.0523	6	89.9%	0.116	0.643	Buckle button, pulling shirt collar, adjusting tie
iMiGUE	Qwen2.5-VL-7B	Blank-cal. NLL	0.1363	23	38.6%	0.614	0.430	Crossing fingers
iMiGUE	MiMo-VL-7B	Prompt-based	0.2445	27	54.1%	0.491	0.339	Rubbing or holding hands
iMiGUE	MiMo-VL-7B	Vanilla NLL	0.0325	2	99.7%	0.006	0.673	Buckle button, pulling shirt collar, adjusting tie
iMiGUE	MiMo-VL-7B	Blank-cal. NLL	0.1573	26	28.9%	0.623	0.310	Touching or scratching facial parts

Appendix

Table 10. iMiGUE prediction-distribution comparison among prompt-based free-form generation, raw NLL-based scoring, and blank-calibrated NLL-based scoring. For prompt-based and raw NLL-based, mCA@1 is aligned with Table 1 in the main paper; the remaining statistics are computed from the corresponding Top-1 histograms. Lower dominant ratio and JSD are better, while higher #Top-1 labels and

H_{\mathrm{norm}}

indicate less concentrated predictions.

Method	mCA@1	mCA@5	MF1
BlockerGCN(J)	0.2324	0.6080	0.2375
BlockerGCN(B)	0.2362	0.6040	0.2407
BlockerGCN(V)	0.2375	0.5820	0.2467
BlockerGCN(3s)	0.2658	0.6469	0.2809
PoseC3D	0.3402	0.7159	0.3583
MMN(J)	0.2991	0.6626	0.3062

Appendix

Table 11. Full comparison between Zero-MELO and supervised discriminative models on the iMiGUE dataset.

InternVL3.5	iMiGUE
InternVL3.5	mCA@1	mCA@5	MF1
Prompt-based zero-shot	0.2034	0.4358	0.0927
NLL-based	0.0329	0.2408	0.0007
Zero-MELO (Ours)	0.2425	0.4375	0.0974

Appendix

Table 12. Comparison between Zero-MELO and the corresponding vanilla MLLM under the InternVL3.5 backbone on iMiGUE.

MiMo-VL	iMiGUE
MiMo-VL	mCA@1	mCA@5	MF1
Prompt-based zero-shot	0.2445	0.5643	0.1506
NLL-based	0.0325	0.2480	0.0008
Zero-MELO (Ours)	0.2977	0.5834	0.1534

Appendix

Table 13. Comparison between Zero-MELO and the corresponding vanilla MLLM under the MiMo-VL backbone on iMiGUE.

Method	mCA@1	mCA@5	MF1
Vanilla	0.0357	0.2084	0.0094
+ Cali (Lowrank)	0.1455	0.3717	0.0569
+ Cali (Blank)	0.2040	0.4943	0.0855
+ Cali (Aggregation)	0.2363	0.5436	0.1097

Appendix

Table 14. Comparison experiments across different calibration strategies.

Method	Acc@1	M	Train	Method	Acc@1	M	Train
SynSE [ICIP’21]	0.576	S	✓	SKI-FROSTER [AAAI’25]	0.685	R+T+S	✓
FS-VAE [ICCV’25]	0.679	S	✓	SKI-ViFiCLIP [AAAI’25]	0.775	R+T+S	✓
SkeletonContext [CVPR’26]	0.655	S+T	✓	Qwen2.5-VL-7B [5]	0.663	R	$\times$
XCLIP [ECCV’22]	0.578	R+T	$\times$	Zero-MELO ( Ours )	0.694	R	$\times$

Appendix

Table 15. Directly transfer Zero-MELO to the zeroshot task on NTU120-110/10 without manual intervention. M : modality. Train : trained on seen split. ACC@1 is reported for consistency with other methods. R/T/S denote RGB/text/skeleton.

Method	Time	Passes	Method	Time	Passes
Prompt-based Qwen2.5-VL	2.71	1	Zero-MELO ( Ours )	399.82	585
NLL-based Qwen2.5-VL	17.12	96	- TS	389.63	510
ZoomEye [47]	325.29	690	- Cali	9.52	75
- Context Learning	2.03	10	- Fusion	0.59	0
- ZoomEye Search	308.98	648
- Answer	14.18	32

Appendix

Table 16. Average runtime per iMiGUE sample. Time is in seconds, and Passes denotes MLLM calls.

Label $y$	Label-Conditioned Cue Set $\mathcal{C}(y)$	Label $y$	Label-Conditioned Cue Set $\mathcal{C}(y)$
Shaking body	$\{\texttt{torso},\texttt{legs}\}$	Spread legs	$\{\texttt{legs}\}$
Sitting straightly	$\{\texttt{torso}\}$	Closing legs	$\{\texttt{legs}\}$
Shrugging	$\{\texttt{torso}\}$	Crossing legs	$\{\texttt{legs}\}$
Turning around	$\{\texttt{torso},\texttt{legs}\}$	Stretching feet	$\{\texttt{feet},\texttt{legs}\}$
Rising up	$\{\texttt{torso},\texttt{legs}\}$	Retracting feet	$\{\texttt{feet},\texttt{legs}\}$
Bowing head	$\{\texttt{head},\texttt{neck}\}$	Tiptoe	$\{\texttt{feet},\texttt{legs}\}$

Appendix

Table 17. Label-conditioned cue sets

\mathcal{C}(y)

for MA-52.

Prompt type	Qwen2.5-VL	InternVL3.5 / MiMo-VL
Wrapper Prompts
Global wrapper	<video>	Main video frames: <video>
Zoom wrapper	<video> This is the main video, and the section enclosed by the red rectangle in each frame is the focus region. <video> This is the zoomed-in view of the focus region.	Main video frames (the section enclosed by the red rectangle in each frame is the focus region): <video> Zoomed-in video frames of the focus region: <video>
Single Cue Prompts
CQ (region)	In the current view, is the {cue} clearly visible and large enough to follow its motion over time? Answer Yes or No.	Based only on visible evidence, is the person’s {cue} visible in this view? Answer Yes or No.
Existence (region)	Is there a {cue} in the current view? Answer Yes or No.	Based only on visible evidence, can you see the person’s {cue} in this view? Answer Yes or No.

Appendix

Table 18. Prompt templates for region cues used by different backbones. Here, {cue} denotes the queried region cue.

Dataset	Label	Description
iMiGUE	Turtle neck	The movement of drawing the head and neck inward toward the shoulders, making the neck appear shortened.
iMiGUE	Bulging face, deep breath	The movement of taking a noticeable deep breath with visible expansion around the face and upper chest.
iMiGUE	Touching hat	The movement of bringing the hand close to and touching or adjusting the hat or its brim.
iMiGUE	Touching or scratching head	The movement of bringing the hand close to and touching or scratching the scalp or top of the head.
iMiGUE	Touching or scratching forehead	The movement of bringing the hand close to and touching or scratching the forehead.
iMiGUE	Covering face	The movement of covering a large portion of the face with one or both hands.

Appendix

Table 19. Descriptions used for verbalizer marginalization in iMiGUE and MA-52.

Micro-Gesture Recognition

See all 8 papers

FG-SGL: Fine-Grained Semantic Guidance Learning via Motion Process Decomposition for Micro-Gesture Recognition
Mar 17, 2026Jinsheng Wei, Zhaodi Xu, Guanming Lu +2Micro-Gesture Recognition Gesture Recognition
Nanjing University of Posts and Telecommunications, China · University of Oulu, Finland
A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition
Jun 11, 2026Haoran Zhang, Haokun Zhang, Pengyu Liu +3Micro-Gesture Recognition Unsupervised Domain Adaptation
School of Computer Science and Information Engineering, Hefei University of Technology (HFUT), Hefei, China · School of Computer Science, University of Auckland(UOA), Auckland, New Zealand
MA-Bench: Towards Fine-grained Micro-Action Understanding
Mar 27, 2026Kun Li, Jihao Gu, Fei Wang +3MLLM Evaluation Multimodal Large Language Models
CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology +2

Mar 17, 2026J/K move · Enter open · S save

FG-SGL: Fine-Grained Semantic Guidance Learning via Motion Process Decomposition for Micro-Gesture Recognition

Jinsheng Wei, Zhaodi Xu, Guanming Lu, Haoyu Chen+1

Nanjing University of Posts and Telecommunications, China · University of Oulu, Finland

Micro-Gesture Recognition Gesture Recognition Motion Representation Learning

Open paper AskPDF arXiv

Micro-gesture recognition (MGR) is challenging due to subtle inter-class variations. Existing methods rely on category-level supervision, which is insufficient for capturing subtle and localized motion differences. Thus, this paper proposes a Fine-Grained Semantic Guidance Learning (FG-SGL) framework that jointly integrates fine-grained and category-level semantics to guide vision--language models in perceiving local MG motions. FG-SA adopts fine-grained semantic cues to guide the learning of local motion features, while CP-A enhances the separability of MG features through category-level semantic guidance. To support fine-grained semantic guidance, this work constructs a fine-grained textual dataset with human annotations that describes the dynamic process of MGs in four refined semantic dimensions. Furthermore, a Multi-Level Contrastive Optimization strategy is designed to jointly optimize both modules in a coarse-to-fine pattern. Experiments show that FG-SGL achieves competitive performance, validating the effectiveness of fine-grained semantic guidance for MGR.

Abstract

Micro-Gesture Recognition

Explore similar work

FG-SGL: Fine-Grained Semantic Guidance Learning via Motion Process Decomposition for Micro-Gesture Recognition

A Multi-Modal Framework with Cross-Subject Pseudo-Labeling and Semantic Alignment for Micro-Gesture Recognition

MA-Bench: Towards Fine-grained Micro-Action Understanding