MECO: A Multimodal Dataset for Emotion and Cognitive Understanding in Older Adults

Hongbin Chen; Jie Li; Wei Wang; Siyang Song; Xiao Gu; Jianqing Li; Wentao Xiang

cs.HCApr 3, 2026

MECO: A Multimodal Dataset for Emotion and Cognitive Understanding in Older Adults

Authors: Hongbin Chen, Jie Li, Wei Wang, Siyang Song, Xiao Gu, Jianqing Li, Wentao Xiang

Organizations: Nanjing Medical University Nanjing, China · University of Exeter Exeter, U.K. · University of Oxford Oxford, U.K.

Abstract

While affective computing has advanced considerably, multimodal emotion prediction in aging populations remains underexplored, largely due to the scarcity of dedicated datasets. Existing multimodal benchmarks predominantly target young, cognitively healthy subjects, neglecting the influence of cognitive decline on emotional expression and physiological responses. To bridge this gap, we present MECO, a Multimodal dataset for Emotion and Cognitive understanding in Older adults. MECO includes 42 participants and provides approximately 38 hours of multimodal signals, yielding 30,592 synchronized samples. To maximize ecological validity, data collection followed standardized protocols within community-based settings. The modalities cover video, audio, electroencephalography (EEG), and electrocardiography (ECG). In addition, the dataset offers comprehensive annotations of emotional and cognitive states, including self-assessed valence, arousal, six basic emotions, and Mini-Mental State Examination cognitive scores. We further establish baseline benchmarks for both emotion and cognitive prediction. MECO serves as a foundational resource for multimodal modeling of affect and cognition in aging populations, facilitating downstream applications such as personalized emotion recognition and early detection of mild cognitive impairment (MCI) in real-world settings. The complete dataset and supplementary materials are available at https://maitrechen.github.io/meco-page/.

Figures & tables

Figure 1. Overview of MECO dataset. (a) Emotion-inducing video stimuli for older adults. (b) Data acquisition protocol, encompassing cognitive assessment, synchronized multimodal recording, and self-assessed annotations. (c) Extracted multimodal signals and corresponding feature representations. (d) Downstream tasks for emotion and cognitive prediction. (e) Demographic characteristics of the study subjects, highlighting group-specific differences. A four-part overview of the MECO dataset. The first part shows emotion-eliciting video stimuli presented to older adults. The second part illustrates the data collection workflow, including Mini-Mental State Examination cognitive assessment, synchronized acquisition of multimodal signals using wearable sensors, and self-reported emotional annotations. The third part presents the recorded modalities and corresponding feature extraction processes across behavioral and physiological signals. The fourth part depicts downstream tasks, including classification and regression for both emotional and cognitive states.

Dataset	Age (Avg.)	#Subjects (M/F)	Length	Label	Primary Modality	Language	Source
IEMOCAP ( Busso et al., 2008 )	N/A	10 (5/5)	12 h	Emotion	Audio, Video	English	In the lab
DFEW ( Jiang et al., 2020 )	N/A	N/A	N/A	Emotion	Audio, Video	N/A	In the wild
DREAMER ( Katsigiannis and Ramzan, 2018 )	22–33 (26.6)	23 (14/9)	7 h	Emotion	EEG, ECG	N/A	In the lab
SEED-IV ( Zheng and Lu, 2015 )	18–30 (23.3)	15 (7/8)	30 h	Emotion	EEG, EOG	Chinese	In the lab
DEAP ( Koelstra et al., 2012 )	19–37 (26.9)	32 (16/16)	21 h	Emotion	Video, EEG	N/A	In the lab
MAHNOB-HCI ( Soleymani et al., 2012 )	19–40 (26.1)	27 (11/16)	12 h	Emotion	Audio, Video, EEG	English	In the lab

Table 1. Review of representative multimodal emotion datasets. "N/A" denotes information not available.

Figure 2. Overview of the experimental protocol. A schematic of the experimental protocol showing the sequential stages of the data collection process, including participant preparation, presentation of emotion-eliciting stimuli, synchronized recording of multimodal signals, and post-stimulus self-assessment and cognitive evaluation.

Session	Dimension	Single-Rater ICC(2,1)		Average-Rater ICC(2, $k$ )
Session	Dimension	ICC	95% CI	ICC	95% CI
Session 1	Valence	0.7038	[0.47, 0.94]	0.9901	[0.97, 1.00]
Session 1	Arousal	0.4272	[0.21, 0.82]	0.9691	[0.92, 0.99]
Session 2	Valence	0.6875	[0.45, 0.93]	0.9893	[0.97, 1.00]
Session 2	Arousal	0.3250	[0.15, 0.75]	0.9529	[0.88, 0.99]
Session 3	Valence	0.6182	[0.38, 0.91]	0.9855	[0.96, 1.00]

Table 2. Continuous emotion annotation ICCs across three sessions. Both single-rater consistency ICC(2,1) and average-rater reliability ICC(2,

k

) are highly significant (

p<0.001

Figure 3. Distribution of MECO dataset. (a-c) Global-level statistics, including 3- and 5-class discrete emotion distributions and the continuous valence-arousal space. (d, e) Subject-level emotion distributions under 3- and 5-class settings. A set of plots showing the data distribution in the MECO dataset. The first three panels present global statistics, including class distributions under different discrete emotion settings and the distribution of samples in the valence-arousal space. The last two panels illustrate subject-level variations, showing how emotion labels are distributed across individuals under the same classification schemes.

M	Feature	T1: SR (%)		T2: SA (%)		T3: ER (%)		T4: VR		T5: AR
M	Feature	UAR ${}_{\pm std}\uparrow$	WAR ${}_{\pm std}\uparrow$	UAR ${}_{\pm std}\uparrow$	WAR ${}_{\pm std}\uparrow$	UAR ${}_{\pm std}\uparrow$	WAR ${}_{\pm std}\uparrow$	CCC ${}_{\pm std}\uparrow$	MAE ${}_{\pm std}\downarrow$	CCC ${}_{\pm std}\uparrow$	MAE ${}_{\pm std}\downarrow$
V	AU	43.09 ±10.48	45.60 ±10.10	59.20 ±9.52	62.27 ±9.68	48.00 ±10.75	52.03 ±10.40	0.4686 ±0.1729	1.7899 ±0.4995	0.4916 ±0.1463	1.7007 ±0.4011
	HP	44.09 ±11.38	45.94 ±11.02	64.03 ±11.26	69.13 ±9.23	52.54 ±11.24	58.14 ±9.68	0.4558 ±0.1900	1.7639 ±0.4977	0.5247 ±0.1592	1.6262 ±0.5067
	EG	30.73 ±7.00	32.87 ±7.14	45.79 ±8.34	54.42 ±10.18	34.66 ±6.93	40.87 ±8.31	0.2153 ±0.1594	2.1855 ±0.6889	0.2314 ±0.1947	2.5147 ±1.4359
	DF	58.09 ±12.48	59.47 ±11.96	69.85 ±11.32	72.74 ±8.44	63.64 ±11.52	65.80 ±10.08	0.5972 ±0.1544	1.4106 ±0.3949	0.6347 ±0.1339	1.3270 ±0.3691
E	DE	30.54 ±7.55	33.38 ±8.05	45.85 ±7.01	53.44 ±7.38	35.10 ±6.95	41.41 ±8.12	0.2789 ±0.1548	2.0873 ±0.5138	0.2964 ±0.1718	2.0340 ±0.5656

Table 3. Results for the five emotion prediction tasks (T1–T5) on MECO dataset under the SD protocol. Best and second-best results are highlighted in bold and underlined, respectively. “M” denotes the modality: Video (V), EEG (E), and ECG (C).

M	Feature	T2: SA (%)		T3: ER (%)		T6: CR (%)		T7: MR
M	Feature	UAR ${}_{\pm std}\uparrow$	WAR ${}_{\pm std}\uparrow$	UAR ${}_{\pm std}\uparrow$	WAR ${}_{\pm std}\uparrow$	ACC ${}_{\pm std}\uparrow$	F1 ${}_{\pm std}\uparrow$	CCC ${}_{\pm std}\uparrow$	MAE ${}_{\pm std}\downarrow$
V	AU	41.30 ±2.32	49.64 ±1.90	25.90 ±0.76	29.60 ±2.89	60.84 ±2.44	54.28 ±3.64	0.0981 ±0.0838	3.2101 ±0.5792
	HP	36.66 ±1.30	47.74 ±2.55	22.67 ±1.25	28.44 ±3.32	60.51 ±4.73	51.23 ±3.89	0.0762 ±0.0357	3.4828 ±0.5989
	EG	35.34 ±0.37	49.34 ±1.12	22.37 ±1.20	30.46 ±4.65	62.46 ±2.99	45.55 ±4.00	0.0228 ±0.0280	3.6050 ±0.1252
	DF	40.21 ±1.09	45.49 ±1.12	24.56 ±1.04	28.07 ±1.61	57.89 ±12.55	52.57 ±11.39	0.2286 ±0.1805	3.4754 ±0.7802
E	DE	34.19 ±0.25	47.90 ±2.21	21.11 ±0.35	26.53 ±1.42	56.72 ±3.50	52.72 ±3.17	0.1401 ±0.0865	4.2936 ±2.0246

Table 4. Results for the two emotion (T2–T3) and cognitive (T6–T7) prediction tasks on MECO dataset under the SI protocol.

Explore similar work

SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark
Jul 20, 2026Sania Bano, Shahzad Ahmad, Santosh Kumar Vipparthi +2Emotion Recognition in Conversations Multimodal Emotion Recognition
CVPR Lab, Indian Institute of Technology Ropar, India · Østfold University College, Norway · CVPR Lab, SCSS, Trinity College Dublin, Ireland
From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding
Oct 8, 2026Jia Li, Yichao He, Yangchen Yu +6Multimodal Emotion Recognition Emotion Recognition
Hefei University of Technology · Nanyang Technological University
EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding
May 9, 2026Pengze Guo, Jingxi Liang, Zhiwen Xie +2Multimodal Large Language Models Multimodal Emotion Recognition
NLP2CT Lab, Department of Computer and Information Science, University of Macau · School of Computer Science, Central China Normal University · University of Macau

Jul 20, 2026J/K move · Enter open · S save

SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark

Sania Bano, Shahzad Ahmad, Santosh Kumar Vipparthi, Sukalpa Chanda+1

CVPR Lab, Indian Institute of Technology Ropar, India · Østfold University College, Norway · CVPR Lab, SCSS, Trinity College Dublin, Ireland

Emotion Recognition in Conversations Multimodal Emotion Recognition Cross-Dataset Generalization

Open paper AskPDF arXiv

Understanding human emotions in spoken conversations is a key challenge in affective computing, with applications in empathetic AI, human computer interaction, and mental health monitoring. However, existing datasets vary in scale, emotion distribution, modality alignment, and data partitioning strategies, which can influence reliable cross-dataset generalization and minority-emotion modeling. We introduce SpEmoC a Speaking segment Emotion for Conversations comprising 306,544 raw clips from 3,100 English language movies and TV series. From these, 30,000 high quality, class balanced clips are curated, featuring synchronized visual, audio, and textual modalities annotated for seven emotions through a hybrid pipeline that integrates pretrained models with human validation. SpEmoC uses strict movie- and series-level splits to prevent content overlap between split sets, allowing more reliable evaluation of model generalization. The dataset also maintains a near-balanced distribution across seven emotions, including minority classes such as Fear and Disgust, which supports more balanced learning across categories. Extensive experiments, including in-domain benchmarking, cross-dataset transfer, low-data training, class-imbalance analysis, and modality transfer show that balanced data and careful splitting lead to more stable performance across emotions when models are evaluated on other datasets. These results highlight the importance of dataset design for robust and transferable multimodal emotion recognition.

Abstract

Explore similar work

SpEmoC: A Balanced Speaker-Segment Multimodal Emotion Benchmark

From Surface to Depth: Towards Cognitive Appraisal Reasoning in Multimodal Emotion Understanding

EmoS: A High-Fidelity Multimodal Benchmark for Fine-grained Streaming Emotional Understanding