OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

Hymalai Bello; Lala Ray; Joanna Sorysz; Sungho Suh; Paul Lukowicz

cs.CVMar 2, 2026

OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

Authors: Hymalai Bello, Lala Ray, Joanna Sorysz, Sungho Suh, Paul Lukowicz

Organizations: DFKI Kaiserslautern · RPTU Kaiserslautern-Landau · Korea University

Abstract

Smart factories use advanced technologies to optimize production and increase efficiency. To this end, the recognition of worker activity allows for accurate quantification of performance metrics, improving efficiency holistically while contributing to worker safety. OpenMarcie is, to the best of our knowledge, the biggest multimodal dataset designed for human action monitoring in manufacturing environments. It includes data from wearables sensing modalities and cameras distributed in the surroundings. The dataset is structured around two experimental settings, involving a total of 36 participants. In the first setting, twelve participants perform a bicycle assembly and disassembly task under semi-realistic conditions without a fixed protocol, promoting divergent and goal-oriented problem-solving. The second experiment involves twenty-five volunteers (24 valid data) engaged in a 3D printer assembly task, with the 3D printer manufacturer's instructions provided to guide the volunteers in acquiring procedural knowledge. This setting also includes sequential collaborative assembly, where participants assess and correct each other's progress, reflecting real-world manufacturing dynamics. OpenMarcie includes over 37 hours of egocentric and exocentric, multimodal, and multipositional data, featuring eight distinct data types and more than 200 independent information channels. The dataset is benchmarked across three human activity recognition tasks: activity classification, open vocabulary captioning, and cross-modal alignment.

Figures & tables

Dataset	Industrial	Wearables	Ego	Exo	Multiview	Multi-Act.	Scale / Focus
InHARD [ 11 ]	✓	✓	x	✓	✓	x	HRC assembly, RGB+IMU
LARa [ 24 ]	✓	✓	x	✓	x	x	Logistics, IMU+mocap
OpenPack [ 37 ]	✓	✓	x	✓	x	x	Logistics, 50h+ IMU/IoT
Assembly101 [ 32 ]	✓	x	✓	✓	✓	x	Vision-only, procedural assembly
IKEA-ASM [ 7 ]	✓	x	x	✓	✓	x	Exo RGB-D, furniture assembly
HA4M [ 9 ]	✓	x	x	✓	x	x	Exo RGB-D+IR+pose

Table 1: Comparative table of human action recognition datasets in the industrial domain. Multi-actions indicates whether the dataset supports concurrent action labeling (e.g., walking while carrying), based on overlapping verb–object–tool annotations. Only OpenMarcie unifies wearables, egocentric + exocentric multiview video, multi-action labels, and full industrial coverage .

Figure 1: OpenMarcie is an open dataset for multimodal action recognition in industrial environments. Top Its applications include industrial process optimization, human activity understanding, and context awareness. Middle two assembly scenarios represent ad-hoc goal-oriented action and a procedural scenario to promote natural knowledge acquisition. Bottom multimodal data is collected with more than 200 independent channels, and a validation with three different benchmarks is done.

Figure 2: Experiment room setting with example views of the exocentric RGBD cameras.

Scenario	Modality	Position	Ch
Ad-Hoc (a)	IMU	Wrists, Forehead	21
	Magnetometer	Wrists, Forehead	9
	Barometer	Wrists, Head	3
	Temperature	Wrists, Forehead	3
	Spectrometer	Chest, R. Shoulder	24
	Thermal Camera	Chest, R. Shoulder	(8×8)×2

Table 2: Sensing modalities information for each scenario

Figure 3: Participant wearable setup and sensor signals. (a) Participant in the ad-hoc Scenario: bicycle assembly. (b) Participant in the procedural Scenario: 3D printer assembly. Wrist-mounted ZEDs provided only IMU and barometer data (stereo cameras disabled), while head and chest cameras captured visual streams. Sensor placements were adapted to the ergonomics and task demands of each scenario.

Figure 4: Egocentric and exocentric views of activity examples, accompanied by annotations (see Sec. 9 ). These include both soft and hard labels for two scenarios: (a) the ad-hoc bicycle assembly/disassembly task, and (b) the procedural 3D printer construction task.

Figure 5: Participants statistic. Top left Height distribution ranging from 150 to 193 cm. Top right Age distribution spanning 22 to 37 years. Bottom left Self-reported experience levels in assembly tasks, categorized as beginner, intermediate, and advanced. Bottom right academic level of participants, with engineers representing the majority and managerial roles accounting for only 3% of the sample.

Figure 6: Participants’ ownership and usage habits related to bicycles and 3D printers.

Figure 7: OpenMarcie activities distribution for the ad-hoc scenario and the procedural scenario.

Modality	Scenario (a)		Scenario (b)
Modality	Macro F1 ( $\uparrow$ )
	No Null	Null	No Null	Null
Inertial(I)	0.834 $\pm$ 0.007	0.811 $\pm$ 0.007	0.750 $\pm$ 0.015	0.674 $\pm$ 0.003
Acoustic(A)	0.489 $\pm$ 0.018	0.469 $\pm$ 0.017	0.425 $\pm$ 0.004	0.432 $\pm$ 0.005
Vision(V)	0.757 $\pm$ 0.011	0.729 $\pm$ 0.011	0.705 $\pm$ 0.004	0.655 $\pm$ 0.003
I + A	0.803 $\pm$ 0.012	0.782 $\pm$ 0.010	0.744 $\pm$ 0.004	0.666 $\pm$ 0.003

Table 3: Macro F1 scores for human activity recognition in Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly, with and without the null class.

Modality	Scenario (a)		Scenario (b)
Modality	Cosine Similarity ( $\uparrow$ )
	No Null	Null	No Null	Null
Inertial(I)	0.518 $\pm$ 0.023	0.501 $\pm$ 0.022	0.642 $\pm$ 0.002	0.640 $\pm$ 0.002
Acoustic(A)	0.361 $\pm$ 0.030	0.341 $\pm$ 0.018	0.316 $\pm$ 0.003	0.323 $\pm$ 0.004
Vision(V)	0.479 $\pm$ 0.016	0.463 $\pm$ 0.014	0.632 $\pm$ 0.002	0.631 $\pm$ 0.003
I + A	0.512 $\pm$ 0.021	0.493 $\pm$ 0.020	0.644 $\pm$ 0.002	0.641 $\pm$ 0.003

Table 4: Open vocabulary captioning results for Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly, including cosine similarity values with and without the null class.

Modality	Scenario (a)			Scenario (b)
Modality	R@1	R@5	Top-1	R@1	R@5	Top-1
Inertial (I) + Text (T)	0.324 $\pm$ 0.016	0.655 $\pm$ 0.025	0.481 $\pm$ 0.018	0.312 $\pm$ 0.016	0.642 $\pm$ 0.026	0.468 $\pm$ 0.019
Acoustic (A) + T	0.241 $\pm$ 0.014	0.583 $\pm$ 0.025	0.342 $\pm$ 0.016	0.227 $\pm$ 0.013	0.567 $\pm$ 0.022	0.329 $\pm$ 0.015
Vision (V) + T	0.437 $\pm$ 0.015	0.768 $\pm$ 0.017	0.556 $\pm$ 0.016	0.421 $\pm$ 0.013	0.751 $\pm$ 0.018	0.541 $\pm$ 0.014
I + A + T	0.347 $\pm$ 0.014	0.679 $\pm$ 0.019	0.495 $\pm$ 0.017	0.334 $\pm$ 0.015	0.663 $\pm$ 0.017	0.479 $\pm$ 0.018
A + V + T	0.412 $\pm$ 0.013	0.740 $\pm$ 0.020	0.533 $\pm$ 0.015	0.395 $\pm$ 0.014	0.723 $\pm$ 0.019	0.517 $\pm$ 0.014

Table 5: Cross-modal alignment results for Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly.

Figure 8: Distribution of participants by nationality, dominant hand and self identified sex.

ID	Sex	Age	Height	Dominant Hand	Academic Level	Demographic	Experience Level (1-3)
P1-B	F	33	160	R	Engineer	South America (Venezuela)	2
P2-B	F	26	176	R	Engineer	Europe (Poland)	3
P3-B	M	29	175	R	Computer Scientist	Europe (Germany)	1
P4-B	M	26	175	L	Engineer	South Asia (India)	1
P5-B	M	33	189	R	Computer Scientist	South America (Brazil)	1
P6-B	M	36	188	R	Engineer	East Africa (Rwanda)	1

Table 6: Participants’ metadata information for the Ad-hoc Scenario (a).

ID	Gender	Age	Height	Dominant Hand	Academic Level	Demographic	Experience Level (1-3)
P1-D	M	37	178	R	Engineer	East Asia (South Korea)	2
P2-D	F	25	159	R	Engineer	South East Asia Pacific (Nepal)	2
P3-D	M	34	193	R	Engineer	South America (Venezuela)	1
P4-D	F	26	165	R	Engineer	Europe (Italy)	1
P5-D	F	33	160	R	Engineer	South America (Venezuela)	2
P6-D	F	25	164	R	Engineer	South Asia (India)	1

Table 7: Participants metadata information for the procedural Scenario (b).

ID	SS (cm)	SR-WR (cm)	SL-WL (cm)
P1-D	51	55	55
P2-D	34	49	48
P3-D	51	61	61
P4-D	36	57	57
P5-D	36	54	54
P6-D	38	53	53

Table 8: Upper-body anthropometric measurements: Shoulder-to-Shoulder length (SS), Shoulder-Right to Wrist-Right length (SR-WR), and Shoulder-Left to Wrist-Left length (SL-WL), measured in cm. Values are presented for each participant along with Mean ± Standard Deviation.

Figure 9: Comparison of range and diversity of body motions across Scenario (a): bicycle assembly and Scenario (b): 3D printer assembly.

Acronym	Meaning
W	Walking
M	Move: Manipulating an object until the next action
U	Screw/Unscrew
C	Cycling
S	Sitting down
P	Pump: Pumping air into the bicycle tires

Table 9: Activities and objects acronyms for the Scenario (a): Bicycle assembly.

Modality	Scenario (a)		Scenario (b)
Modality	Cosine Similarity ( $\uparrow$ )
	No Null	Null	No Null	Null
I	0.137 $\pm$ 0.041	0.129 $\pm$ 0.038	0.168 $\pm$ 0.012	0.164 $\pm$ 0.011
A	0.098 $\pm$ 0.035	0.091 $\pm$ 0.033	0.103 $\pm$ 0.009	0.099 $\pm$ 0.010
V	0.152 $\pm$ 0.029	0.143 $\pm$ 0.027	0.181 $\pm$ 0.008	0.177 $\pm$ 0.009
I + A	0.141 $\pm$ 0.039	0.133 $\pm$ 0.036	0.172 $\pm$ 0.010	0.168 $\pm$ 0.011

Table 10: Open-vocabulary cosine similarity when training on VLM auto-labels vs. human annotations. VLM labels yield scores only marginally above chance, substantially below the manual-label baselines.

Figure 10: Object distributions in the bicycle and 3D printer scenarios from exocentric views, accompanied by scene visualizations illustrating object segmentation (masks) and human pose estimation in both environments.

Code	Tool	Usage in %
bh	Bare hand	72.5%
hx	Hex key	16.2%
sd	Screwdriver	3.3%
wr	Wrench	2.9%
hk	Hook	2.2%
pu	Pump	1.0%

Table 11: Tool vocabulary for Scenario (a) (Bicycle Assembly). Percentages are computed over the 1,769 tool-annotated action instances aggregated across all users.

Figure 11: Architectures for classification and regression in human activity recognition and open-vocabulary captioning benchmarks.

Figure 12: Architecture for cross-modal alignment benchmark.

Combination	Parameters	MACs	FLOPS
Text + Audio	781,568	783,616	$2.70\times 10^{9}$
Text + Video	2,098,432	2,100,480	$7.37\times 10^{9}$
Text + IMU	712,512	22,797,312	$2.07\times 10^{10}$
Text + Video + IMU	2,351,552	24,437,376	$2.00\times 10^{10}$
Audio + Video + Text	2,420,608	2,423,680	$5.91\times 10^{9}$
Audio + Text + IMU	1,034,688	23,120,512	$1.89\times 10^{10}$

Table 12: Computational Resources for Joint Latent Representations Across Modality Combinations.

Combination	Params	MACs	FLOPS
Audio only	339,200	340,224	$2.72\times 10^{9}$
Video only	1,656,064	1,657,088	$1.38\times 10^{10}$
IMU only	270,144	22,353,920	$2.39\times 10^{10}$
Audio + Video	2,092,928	3,307,392	$1.89\times 10^{10}$
Audio + IMU	707,008	24,004,224	$2.44\times 10^{10}$
Video + IMU	2,023,872	25,321,088	$2.57\times 10^{10}$

Table 13: Computational Resources for Classification Models Across Modality Combinations.

Combination	Params	MACs	FLOPS
Audio only	746,624	744,000	$5.95\times 10^{9}$
Video only	1,952,768	1,952,000	$1.63\times 10^{10}$
IMU only	566,848	22,648,832	$2.42\times 10^{10}$
Audio + Video	2,270,080	2,268,992	$1.70\times 10^{10}$
Audio + IMU	884,160	22,965,824	$2.33\times 10^{10}$
Video + IMU	2,090,368	24,173,824	$2.45\times 10^{10}$

Table 14: Computational Resources for Regression Models Across Modality Combinations.

Modality	Scenario (a)		Scenario (b)
Modality	Macro F1 ( $\uparrow$ )
	Without Null	With Null	Without Null	With Null
Acoustic-Anonymized Encoder	0.489 $\pm$ 0.018	0.469 $\pm$ 0.017	0.425 $\pm$ 0.004	0.432 $\pm$ 0.005
Acoustic-Non-Anonymized Encoder	0.509 $\pm$ 0.002	0.488 $\pm$ 0.001	0.460 $\pm$ 0.002	0.453 $\pm$ 0.003
Acoustic-Anonymized Mel-Spectrum	0.492 $\pm$ 0.003	0.473 $\pm$ 0.002	0.434 $\pm$ 0.003	0.430 $\pm$ 0.003
Acoustic-Non-Anonymized Mel-Spectrum	0.517 $\pm$ 0.004	0.493 $\pm$ 0.002	0.466 $\pm$ 0.003	0.455 $\pm$ 0.002

Table 15: Human activity recognition results for Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly, including macro F1 scores with and without the null class for Audio Extended Experiments.

Modality	Scenario (a)		Scenario (b)
Modality	Cosine Similarity ( $\uparrow$ )
	Without Null	With Null	Without Null	With Null
Acoustic-Anonymized Encoder	0.361 $\pm$ 0.030	0.341 $\pm$ 0.018	0.316 $\pm$ 0.003	0.323 $\pm$ 0.004
Acoustic-Non-Anonymized Encoder	0.375 $\pm$ 0.001	0.354 $\pm$ 0.001	0.328 $\pm$ 0.001	0.338 $\pm$ 0.002
Acoustic-Anonymized Mel-Spectrum	0.364 $\pm$ 0.003	0.348 $\pm$ 0.002	0.326 $\pm$ 0.002	0.322 $\pm$ 0.001
Acoustic-Non-Anonymized Mel-Spectrum	0.381 $\pm$ 0.002	0.359 $\pm$ 0.002	0.330 $\pm$ 0.001	0.340 $\pm$ 0.001

Table 16: Open vocabulary captioning results for Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly, including cosine similarity values with and without the null class for Audio Extended Experiments.

Modality	Scenario (a)			Scenario (b)
Modality	Recall@1 ( $\uparrow$ )	Recall@5 ( $\uparrow$ )	Top-1 ( $\uparrow$ )	Recall@1 ( $\uparrow$ )	Recall@5 ( $\uparrow$ )	Top-1 ( $\uparrow$ )
Acoustic + Text (Anonymized Encoder)	0.241 $\pm$ 0.014	0.583 $\pm$ 0.025	0.342 $\pm$ 0.016	0.227 $\pm$ 0.013	0.567 $\pm$ 0.022	0.329 $\pm$ 0.015
Acoustic + Text (Non-Anonymized Encoder)	0.251 $\pm$ 0.001	0.605 $\pm$ 0.002	0.355 $\pm$ 0.002	0.237 $\pm$ 0.001	0.589 $\pm$ 0.001	0.341 $\pm$ 0.001
Acoustic + Text (Anonymized Mel-Spectrum)	0.246 $\pm$ 0.001	0.595 $\pm$ 0.002	0.350 $\pm$ 0.001	0.233 $\pm$ 0.001	0.578 $\pm$ 0.002	0.336 $\pm$ 0.001
Acoustic + Text (Non-Anonymized Mel-Spectrum)	0.254 $\pm$ 0.001	0.613 $\pm$ 0.002	0.360 $\pm$ 0.001	0.238 $\pm$ 0.001	0.597 $\pm$ 0.001	0.345 $\pm$ 0.001

Table 17: Cross-modal alignment results for Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly, including recall@5, recall@1, and top-1 metrics.

Modality	Scenario (a)		Scenario (b)
Modality	Macro F1 ( $\uparrow$ )
	No Null	Null	No Null	Null
Inertial(I)	0.872 $\pm$ 0.008	0.847 $\pm$ 0.009	0.861 $\pm$ 0.006	0.836 $\pm$ 0.006
Acoustic(A)	0.783 $\pm$ 0.012	0.762 $\pm$ 0.013	0.772 $\pm$ 0.005	0.750 $\pm$ 0.006
Vision(V)	0.841 $\pm$ 0.009	0.817 $\pm$ 0.010	0.833 $\pm$ 0.004	0.808 $\pm$ 0.005
I + A	0.898 $\pm$ 0.007	0.871 $\pm$ 0.008	0.889 $\pm$ 0.004	0.863 $\pm$ 0.004

Table 18: Tool-contact/impact detection Macro F1 scores for Scenario (a): Bicycle Assembly and Scenario (b): 3D Printer Assembly, with and without the null class. Unlike general HAR, audio is complementary for detecting tool-contact events: adding acoustic data to V + I improves F1 from 0.905 to 0.923 in (a) and from 0.898 to 0.914 in (b).

Modality	Physical Signal	Assembly-Relevant Attributes	Contribution
IMU (Wrists, Head)	Acceleration, angular velocity, orientation	Tool manipulation, temporal segmentation	High
Magnetometer	Magnetic field orientation	Heading stabilization, orientation consistency	Supportive
Barometer (multi-position)	Air pressure (hPa)	Posture transitions, vertical displacement cues	Medium
Temperature (ambient)	Ambient temperature	Environmental context, barometric formula	Low (supporting)
Spectrometer	Material spectral reflectance	Material differentiation (metal vs. plastic)	Low (Exploratory)
Thermal Camera	Surface temperature distribution	Contact duration, friction heat	Low (Exploratory)

Table 19: Sensor Modalities and Their Intended Roles in OpenMarcie.

Modality	Scenario (a)		Scenario (b)
Modality	Macro F1 ( $\uparrow$ )
	No Null	Null	No Null	Null
Late Fusion
I + A + V	0.859 $\pm$ 0.010	0.831 $\pm$ 0.011	0.763 $\pm$ 0.003	0.676 $\pm$ 0.003
All	0.891 $\pm$ 0.008	0.862 $\pm$ 0.008	0.825 $\pm$ 0.003	0.731 $\pm$ 0.004
Perceiver IO

Table 20: Macro F1 scores for human activity recognition in Scenario (a) and Scenario (b) with and without the null class. Results are grouped by late fusion baselines, and Perceiver IO baseline with all modalilies including IMU, Audio, Video, Barometer, Temperature, Spectrometer, Thermal signals.

Modality	Scenario (a)		Scenario (b)
Modality	Cosine Similarity ( $\uparrow$ )
	No Null	Null	No Null	Null
Late Fusion
I + A + V	0.547 $\pm$ 0.020	0.519 $\pm$ 0.017	0.647 $\pm$ 0.001	0.646 $\pm$ 0.003
All	0.593 $\pm$ 0.015	0.563 $\pm$ 0.014	0.691 $\pm$ 0.002	0.690 $\pm$ 0.002
AnyMAL

Table 21: Open vocabulary captioning results for Scenario (a) and Scenario (b) with and without the null class. Results are grouped by late fusion baselines, and the AnyMAL LLM-based baseline.

Explore similar work

IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly
Apr 12, 2026Di Wen, Zeyun Zhong, David Schneider +18Human Activity Recognition Anomaly Detection
Karlsruhe Institute of Technology, Karlsruhe, Germany · ETH Zurich, Zurich, Switzerland · Italian Institute of Technology, Genova, Italy +1
OHRID-Retail: An Open Multimodal Dataset of Human Activity in Retail Environments
Sep 16, 2026Xiangrui Wang, Yuetong Wu, Jalen Beeman +6Human-Robot Interaction
Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL, USA. · Lane Department of Computer Science and Electrical Engineering, West Virginia University, Morgantown, WV, USA · Department of Mechanical, Materials and Aerospace Engineering, West Virginia University, Morgantown, WV, USA. +1
PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing
Aug 18, 2026Tengbo Yu, Jiahao Wu, Hanning Wang +4Robotic Data Collection Robot Manipulation Benchmarks
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University

Apr 12, 2026J/K move · Enter open · S save

IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly

Di Wen, Zeyun Zhong, David Schneider, Manuel Zaremski+17

Karlsruhe Institute of Technology, Karlsruhe, Germany · ETH Zurich, Zurich, Switzerland · Italian Institute of Technology, Genova, Italy+1

Human Activity Recognition Anomaly Detection

Open paper AskPDF arXiv

We introduce IMPACT, a synchronized five-view RGB-D dataset for deployment-oriented industrial procedural understanding, built around real assembly and disassembly of a commercial angle grinder with professional-grade tools. To our knowledge, IMPACT is the first real industrial assembly benchmark that jointly provides synchronized ego-exo RGB-D capture, decoupled bimanual annotation, compliance-aware state tracking, and explicit anomaly--recovery supervision within a single real industrial workflow. It comprises 112 trials from 13 participants totaling 39.5 hours, with multi-route execution governed by a partial-order prerequisite graph, a six-category anomaly taxonomy, and operator cognitive load measured via NASA-TLX. The annotation hierarchy links hand-specific atomic actions to coarse procedural steps, component assembly states, and per-hand compliance phases, with synchronized null spans across views to decouple perceptual limitations from algorithmic failure. Systematic baselines reveal fundamental limitations that remain invisible to single-task benchmarks, particularly under realistic deployment conditions that involve incomplete observations, flexible execution paths, and corrective behavior. The full dataset, annotations, and evaluation code are available at https://github.com/Kratos-Wen/IMPACT.

Abstract

Explore similar work

IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly

OHRID-Retail: An Open Multimodal Dataset of Human Activity in Retail Environments

PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing