SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

Matea Tashkovska; Naël Ouerghemi; Haozhe Qi; Alexander Mathis

cs.CVOct 26, 2026

SOMA: A Benchmark for Fine-Grained Egocentric Behavior and Body-Motion Understanding

Authors: Matea Tashkovska, Naël Ouerghemi, Haozhe Qi, Alexander Mathis

Organizations: École Polytechnique Fédérale de Lausanne (EPFL), Switzerland

Abstract

Perceiving human body motion is critical for multimodal systems that understand and assist people. Yet current evaluations of Multimodal Large Language Models (MLLMs) largely focus on objects, scenes, and coarse actions, leaving their ability to reason about the body itself poorly assessed. This is especially challenging from the egocentric perspective, in which the body is only partially visible and motion must be inferred from occluded and temporally distributed cues. We introduce SOMA, a benchmark for fine-grained body-motion understanding from egocentric video, comprising 1,632 human-verified questions across three capability axes: body configuration, motion dynamics, and temporal reasoning. We evaluate 37 leading closed-source, open-source, and specialized egocentric MLLMs, complemented by an extensive human evaluation. We find that humans also find this task difficult from video alone (68.5% accuracy), yet clearly outperform the strongest MLLM (Qwen3.6-27B), which reaches 50.0% zero-shot accuracy. Our error analysis shows that MLLMs often misread the wearer's own body, misjudge events in time, and are prone to biases such as favoring the right hand or under-counting repetitions. To narrow the gap to human performance and support further progress, we introduce a training dataset of aligned video, body motion, and language supervision, and a multimodal baseline that jointly learns from these modalities. By identifying common failure modes and missing capabilities in current MLLMs, we hope SOMA will inspire future research on body-motion understanding and multimodal reasoning. Full benchmark, training data, and code will be released at: https://cnai.epfl.ch/soma

Figures & tables

Figure 1: Overview of SOMA. Left: Representative questions from the three capability axes, Body Configuration , Motion Dynamics , and Temporal Reasoning , grounded in egocentric video and body motion. Right: Performance of representative MLLMs across the twelve SOMA tasks compared with human performance (5 annotators), highlighting a human–model gap. Note: performance normalized for visualization purposes.

	Benchmark	#Clips	#QA	#Cat.	Attributes			QA Scope
	Benchmark	#Clips	#QA	#Cat.	Modality	Ego	Train	Body	Mot.	Temp.
Human- centric	BABEL-QA ( Endo et al., 2023 )	1,109	2,577	3		✗	✓	✓	✓	✓
	MoVid ( Chen et al., 2025 )	1,350	1,350	5	$\veebar$	✗	✓	✓	✓	✓
	MotionBench ( Hong et al., 2025 )	5,385	8,052	6		✗	✗	✓	✓	✓
	ExAct ( Yi et al., 2025 )	3,521	3,521	2		✗	✗	✓	✗	✓
Egocentric Video QA	EgoTaskQA ( Jia et al., 2022 )	2,000	40,000	4		✓	✓	✗	✗	✓

Table 1: Benchmark Comparison. #Clips : number of video or motion clips; #Cat. : number of categories; Input Modality : RGB video, 3D body pose (

\land

: both modalities;

\veebar

: either modality); Ego : egocentric perspective; Train : provides a training split; Body : targets the body, body parts, or posture; Mot. : fine-grained motion beyond coarse activity recognition; Temp. : temporal reasoning.

Figure 2: SOMA Benchmark Taxonomy. SOMA comprises 12 tasks organized into three capability axes: Body Configuration , Motion Dynamics and Temporal Reasoning . We show the distribution of questions across tasks, together with a representative question for each task. See Appendix A for more examples.

Figure 3: SOMA Data Construction Pipeline. We construct SOMA through task assignment, task-specific question generation, and quality control with blind-filtering and human validation.

Table 2: Comparison Results on SOMA. Accuracy (%) across the three capability axes and their twelve tasks. HU: Hand Use; BC: Bimanual Coordination; BP: Body Posture; CBS: Concurrent Body-Part State; RC: Repetition Counting; MD: Movement Direction; OMD: Opposite Motion Discrimination; AAI: Adjacent Action Identification; CO: Chronological Ordering; MEI: Missing Event Inference; NAP: Next Action Prediction; RER: Repeated Event Reasoning. The best model results are marked in bold , and the second-best are underlined . Human performance is the mean over five annotators; standard deviations are reported in Table 4 .

Figure 5: Distribution of Error Types. Root causes of incorrect answers across two representative MLLMs.

Backbone	Setting	Body	Mot.	Temp.	Avg.
Qwen3-VL-8B	Zero-Shot †	35.88	40.89	31.67	35.39
	Motion	27.32	28.12	29.02	28.21
	Video	44.99	43.75	37.13	41.49
	Video+Mot.	45.54	45.57	37.91	42.44
InternVL3.5-8B	Zero-Shot †	38.43	40.10	29.49	35.20
	Motion	30.97	28.39	29.02	29.54

Table 3: Effect of Body-Motion Input. Accuracy (%) per capability axis after fine-tuning with different input modalities. Per-task results in Table 11 . † Zero-shot backbones are evaluated under the same video preprocessing as the fine-tuned models (16 frames, matched resolution).

Appendix figures & tables17 assets

Supplementary material from the paper’s appendix.

Appendix

Body Configuration

Motion Dynamics

Temporal Reasoning

Avg.

CBS

OMD

AAI

MEI

NAP

RER

90.5

\pm

1.8

78.3

\pm

3.7

61.9

\pm

8.0

50.3

\pm

9.2

88.9

\pm

4.3

66.3

\pm

9.1

86.1

\pm

8.5

70.1

\pm

7.3

53.2

\pm

14.4

72.7

\pm

15.6

34.5

\pm

4.9

67.4

\pm

5.9

68.5

\pm

5.5

Appendix

Table 4: Variability of Human Performance. Accuracy (%) on each SOMA task, reported as mean

\pm

standard deviation across the five annotators. Task abbreviations follow Table 2 .

Models	Setting	Body Configuration				Motion Dynamics			Temporal Reasoning					Avg.
Models	Setting	HU	BC	BP	CBS	RC	MD	OMD	AAI	CO	MEI	NAP	RER	Avg.
Qwen3.8-27B	w/o CoT	72.1	47.7	46.1	44.1	36.6	40.5	58.6	44.3	34.7	48.3	39.0	37.0	47.0
Qwen3.8-27B	w/ o CoT	73.1	49.0	43.5	49.5	41.6	49.0	59.3	51.1	46.3	52.5	40.9	40.0	50.9
Gemma-4-31B	w/o CoT	67.2	41.1	39.1	46.8	37.6	34.6	50.7	38.2	38.8	30.5	40.2	38.0	43.1
Gemma-4-31B	w/ o CoT	58.2	42.4	42.6	44.1	25.7	39.2	47.9	43.5	41.5	39.8	33.5	36.0	42.2
InternVL3.5-38B	w/o CoT	52.2	44.4	37.4	32.4	36.6	30.1	59.3	42.0	29.9	45.8	17.7	38.0	39.0

Appendix

Table 5: Effect of Chain-of-Thought Prompting on SOMA. Accuracy (%) across the three capability axes. For each model, the top row is direct answering ( w/o CoT ) and the bottom row asks the model to reason step by step before answering ( w/ CoT ). The better of the two is marked in bold . Task abbreviations follow Table 2 .

Figure 28: Effect of temporal sampling density on accuracy, per capability axis.

Figure 29: Effect of test-time reasoning on accuracy, per capability axis.

Figure 30: Human and model performance across question difficulty levels. The human curve is the mean over five annotators with the band showing

\pm 1

SE.

Figure 35: Accuracy on left hand versus right hand questions, per model.

Figure 36: Direction of error on Repetition Counting.

Models	Body Configuration				Motion Dynamics			Temporal Reasoning					Avg.
Models	HU	BC	BP	CBS	RC	MD	OMD	AAI	CO	MEI	NAP	RER	Avg.
Random Chance	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0
Likelihood-Scored
MG-MotionLLM ( Wu et al., 2025a )	16.3	40.4	33.6	27.3	22.1	20.8	29.4	27.4	20.6	22.8	32.9	24.5	26.4
MotionGPT ( Jiang et al., 2023 )	23.0	26.7	33.6	30.0	26.3	19.5	31.0	25.8	20.6	28.9	27.2	22.4	26.0
MotionGPT3 ( Zhu et al., 2025a )	27.5	32.9	20.0	27.3	21.1	27.5	25.4	21.0	11.3	22.8	30.4	27.6	24.9

Appendix

Table 6: Performance of Motion-Only Baselines on SOMA. Accuracy (%) across the three capability axes and their twelve tasks, for models that receive body motion and no video. Scored on the 1,549 questions whose samples carry ground-truth motion. The four likelihood-scored models rank the options directly while Motion-SMD answers autoregressively through its own extractor. Task abbreviations follow Table 2 .

Task	Templates
Body Configuration
Hand Use	Which hand does the person use in this clip? In this clip, which hand performs the action? Which hand is the person using here? During this segment, which hand does the person use? Which of the following describes the person’s hand use in this clip? Which hand carries out the action in this clip?
Bimanual Coordination	Which arm does the distinctive movement in this clip? Which of the person’s arms is more actively moving? In this clip, which arm carries the notable movement? Which arm stands out as the moving one during this segment? Over this clip, which arm does the distinctive action? Which of the following describes the arms’ roles in this clip? During this segment, which arm is the active one? Which arm is doing the distinctive movement here?
Concurrent Body-Part State	While the person is {a} , what are their {l} doing? What are the person’s {l} doing while they are {a} ? As the person is {a} , what are their {l} doing? When the person {a} , what are their {l} doing? What are the person’s {l} doing while the person {a} ? While the person {a} , what are their {l} doing?
Body Posture	How many distinct body-posture states does the person move through in this clip, and what are they? What is the sequence of body postures the person moves through in this clip? What body posture is the person in at the end of the clip? Over the course of this clip, which of the following posture changes occurs?
Motion Dynamics

Appendix

Table 7: SOMA-Train Question Templates. The set of templates used to construct SOMA-Train, organized by capability axis. Questions are generated by randomly sampling templates and filling them with the corresponding clip annotations.

Training Input	Input	Blind	$\Delta_{\text{I}-\text{B}}$
None (blind)	–	28.1	–
Motion	28.2	25.6	+2.6
Video	41.5	27.4	+14.1
Video+Motion	42.4	25.8	+16.6
Chance (uniform)		25.0

Appendix

Table 8: Blind Transfer to SOMA (Qwen3-VL-8B, accuracy %). Each row fine-tunes the same LoRA recipe on SOMA-Train and is evaluated on SOMA with and without perceptual input (motion, video).

\Delta_{\text{I}-\text{B}}

is the modality–blind gap.

	Validation (194 takes)		Train (792 takes)
	Uncorrected	Corrected	Uncorrected	Corrected
EE4D-Motion	44.2	40.3	45.9	41.8
Ours	42.6	36.7	45.7	39.6
$\Delta$	$-1.6$	$-3.6$	$-0.2$	$-2.2$

Appendix

Table 9: Ego-Exo4D Refitting Quality. PA-MPJPE (mm, take-averaged) against Ego-Exo4D’s human annotations, over the non-climbing takes common to those annotations, EE4D-Motion and our fits: 827 train and 212 val, of which the 792 and 194 scored here pass the per-frame filters. Corrected adds the per-joint convention offset, fitted on train.

			EE4D-Motion †		Ours
Split	Takes	Hours	Takes	Video time	Takes	Video time
Train	2,232	119.6	91.0	70.5	95.9	93.1
Val	597	35.4	93.8	73.1	97.2	96.2

Appendix

Table 10: Ego-Exo4D Coverage. Share of non-climbing takes and video time carrying body motion, framewise on the 30 fps grid. † At 10 fps, equal video time carries half the samples.

Figure 37: Per-Question Body-Motion Coverage. Percentage of Ego-Exo4D questions with at least

x

% of their frames covered by fitted body motion at the target 20 fps rate. The dashed curve interpolates the 10 fps EE4D-Motion fits to 20 fps.

Setting	Body Configuration				Motion Dynamics			Temporal Reasoning					Avg.
Setting	HU	BC	BP	CBS	RC	MD	OMD	AAI	CO	MEI	NAP	RER	Avg.
Random Chance	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0
Qwen3-VL-8B
Zero-Shot †	47.0	28.1	38.2	25.5	37.0	34.2	51.1	35.7	33.6	32.2	23.3	36.7	35.4
+ Motion	19.7	24.0	38.2	33.6	23.0	25.5	34.8	29.4	33.6	25.2	26.4	30.6	28.2
+ Video	57.9	34.2	46.4	36.4	31.0	38.9	58.5	38.9	39.2	47.0	25.2	39.8	41.5

Appendix

Table 11: Fine-tuning with Body-Motion Input, Per Task. Accuracy (%) on SOMA after fine-tuning on SOMA-Train, by input modality. Task abbreviations follow Table 2 . † Zero-shot rows use the same video preprocessing as the fine-tuned models. Best per backbone in bold .

Input at inference	Body Configuration				Motion Dynamics			Temporal Reasoning					Avg.	$\Delta$
Input at inference	HU	BC	BP	CBS	RC	MD	OMD	AAI	CO	MEI	NAP	RER	Avg.	$\Delta$
Random Chance	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0	25.0
Qwen3-VL-8B , fine-tuned on video + motion
Video + Motion	53.0	34.2	51.8	41.8	32.0	38.9	63.0	35.7	39.2	47.8	28.9	41.8	42.4	–
Video	56.3	34.9	40.9	33.6	36.0	32.9	60.0	40.5	42.7	47.0	26.4	35.7	41.0	$-$ 1.4
Motion	25.1	27.4	40.9	30.9	27.0	31.5	33.3	32.5	29.4	29.6	20.1	25.5	29.1	$-$ 13.3

Appendix

Table 12: Test-Time Modality Ablation. Accuracy (%) of the same jointly fine-tuned Video+Mot. checkpoint for each backbone when evaluated with different input modalities at test time. Task abbreviations follow Table 2 ;

\Delta

denotes the change in Avg. relative to Video+Mot.

Artifact	License
Datasets
Nymeria ( Ma et al., 2024 )	CC BY-NC 4.0
Ego-Exo4D ( Grauman et al., 2024 )	Ego-Exo4D Model License
Evaluation framework
lmms-eval ( Zhang et al., 2025b )	MIT and Apache 2.0

Appendix

Table 13: Third-party artifacts used in this work and their licenses.

Human Motion Understanding

See all 17 papers

MA-Bench: Towards Fine-grained Micro-Action Understanding
Mar 27, 2026Kun Li, Jihao Gu, Fei Wang +3MLLM Evaluation Multimodal Large Language Models
CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology +2
Encoder-Free Human Motion Understanding via Structured Motion Descriptions
Apr 23, 2026Yao Zhang, Zhuchenyang Liu, Thomas Ploetz +1Human Motion Understanding
Aalto University, Espoo, Finland · Georgia Institute of Technology, Atlanta, GA, USA
Fine-grained Human Motion Understanding with Language Models
Jun 18, 2026Thomas Markhorst, Zhi-Yi Lin, Jouh Yeong Chew +2Human Motion Understanding
Delft University of Technology · Honda Research Institute Japan

Mar 27, 2026J/K move · Enter open · S save

MA-Bench: Towards Fine-grained Micro-Action Understanding

Kun Li, Jihao Gu, Fei Wang, Zhiliang Wu+2

CVLab, College of Information Technology, United Arab Emirates University · University College London · Hefei University of Technology+2

MLLM Evaluation Multimodal Large Language Models Multimodal Model Evaluation Human Activity Recognition

Open paper AskPDF arXiv

With the rapid development of Multimodal Large Language Models (MLLMs), their potential in Micro-Action understanding, a vital role in human emotion analysis, remains unexplored due to the absence of specialized benchmarks. To tackle this issue, we present MA-Bench, a benchmark comprising 1,000 videos and a three-tier evaluation architecture that progressively examines micro-action perception, relational comprehension, and interpretive reasoning. MA-Bench contains 12,000 structured question-answer pairs, enabling systematic assessment of both recognition accuracy and action interpretation. The results of 23 representative MLLMs reveal that there are significant challenges in capturing motion granularity and fine-grained body-part dynamics. To address these challenges, we further construct MA-Bench-Train, a large-scale training corpus with 20.5K videos annotated with structured micro-action captions for fine-tuning MLLMs. The results of Qwen3-VL-8B fine-tuned on MA-Bench-Train show clear performance improvements across micro-action reasoning and explanation tasks. Our work aims to establish a foundation benchmark for advancing MLLMs in understanding subtle micro-action and human-related behaviors. Project Page: https://MA-Bench.github.io

Abstract

Human Motion Understanding

Explore similar work

MA-Bench: Towards Fine-grained Micro-Action Understanding

Encoder-Free Human Motion Understanding via Structured Motion Descriptions

Fine-grained Human Motion Understanding with Language Models