IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly

Di Wen; Zeyun Zhong; David Schneider; Manuel Zaremski; Linus Kunzmann; Yitian Shi; Ruiping Liu; Yufan Chen

cs.CVApr 12, 2026

IMPACT: A Dataset for Multi-Granularity Human Procedural Action Understanding in Industrial Assembly

Authors: Di Wen, Zeyun Zhong, David Schneider, Manuel Zaremski, Linus Kunzmann, Yitian Shi, Ruiping Liu, Yufan Chen, +13 more

Organizations: Karlsruhe Institute of Technology, Karlsruhe, Germany · ETH Zurich, Zurich, Switzerland · Italian Institute of Technology, Genova, Italy · INSAIT, Sofia University, Sofia, Bulgaria

Abstract

We introduce IMPACT, a synchronized five-view RGB-D dataset for deployment-oriented industrial procedural understanding, built around real assembly and disassembly of a commercial angle grinder with professional-grade tools. To our knowledge, IMPACT is the first real industrial assembly benchmark that jointly provides synchronized ego-exo RGB-D capture, decoupled bimanual annotation, compliance-aware state tracking, and explicit anomaly--recovery supervision within a single real industrial workflow. It comprises 112 trials from 13 participants totaling 39.5 hours, with multi-route execution governed by a partial-order prerequisite graph, a six-category anomaly taxonomy, and operator cognitive load measured via NASA-TLX. The annotation hierarchy links hand-specific atomic actions to coarse procedural steps, component assembly states, and per-hand compliance phases, with synchronized null spans across views to decouple perceptual limitations from algorithmic failure. Systematic baselines reveal fundamental limitations that remain invisible to single-task benchmarks, particularly under realistic deployment conditions that involve incomplete observations, flexible execution paths, and corrective behavior. The full dataset, annotations, and evaluation code are available at https://github.com/Kratos-Wen/IMPACT.

Figures & tables

Figure 1. Overview of the IMPACT dataset and benchmark. Top: recording setup with two angle grinder models, instruction manual, synchronized ego–exo RGB-D views, gaze, audio, and cognitive load. Middle: task taxonomy covering temporal understanding, cross-view understanding, action forecasting, and state & reasoning. Bottom: multi-granularity annotations including step segments, bimanual actions, procedural steps, assembly states, and phases (normal, anomaly, recovery). A multi-part figure showing an industrial assembly dataset and benchmark. The top section displays components of two angle grinders and multiple synchronized camera views, including one egocentric and four exocentric RGB-D perspectives, along with gaze visualization and a cognitive load chart. The middle section groups tasks into temporal understanding, cross-view understanding, action forecasting, and state and reasoning. The bottom section presents timeline-based annotations, including step-level segments, bimanual left and right hand actions, procedural steps, assembly states, and phase labels indicating normal, anomaly, and recovery periods.

Model	Split	TAS-S										TAS-BL										TAS-BR
Model	Split	I3D ( Carreira and Zisserman, 2017 )					VideoMAEv2 ( Wang et al., 2023a )					I3D ( Carreira and Zisserman, 2017 )					VideoMAEv2 ( Wang et al., 2023a )					I3D ( Carreira and Zisserman, 2017 )					VideoMAEv2 ( Wang et al., 2023a )
		Acc $\uparrow$	Edit $\uparrow$	F1@10 $\uparrow$	F1@25 $\uparrow$	F1@50 $\uparrow$	Acc $\uparrow$	Edit $\uparrow$	F1@10 $\uparrow$	F1@25 $\uparrow$	F1@50 $\uparrow$	Acc $\uparrow$	Edit $\uparrow$	F1@10 $\uparrow$	F1@25 $\uparrow$	F1@50 $\uparrow$	Acc $\uparrow$	Edit $\uparrow$	F1@10 $\uparrow$	F1@25 $\uparrow$	F1@50 $\uparrow$	Acc $\uparrow$	Edit $\uparrow$	F1@10 $\uparrow$	F1@25 $\uparrow$	F1@50 $\uparrow$	Acc $\uparrow$	Edit $\uparrow$	F1@10 $\uparrow$	F1@25 $\uparrow$	F1@50 $\uparrow$
LTContext ( Bahrami et al., 2023 )	1	65.51	57.02	55.03	52.73	40.34	54.40	48.61	46.57	41.26	32.49	48.66	29.48	27.86	23.04	13.82	60.82	35.32	32.89	28.60	18.21	36.35	28.47	26.68	23.96	14.50	49.23	39.64	36.61	34.01	22.71
	2	63.75	55.23	53.25	50.84	38.81	53.52	50.36	47.86	43.39	34.09	22.86	17.34	15.38	12.47	6.14	23.05	14.09	11.88	8.51	4.52	17.66	16.72	14.63	12.42	7.06	20.81	14.73	12.68	12.64	7.01
	3	35.64	33.55	31.58	29.40	20.54	27.50	28.78	26.96	25.29	19.42	41.10	14.08	11.98	9.58	5.16	49.10	20.26	17.76	14.63	8.21	18.96	14.59	11.98	9.46	5.34	27.49	16.41	13.58	12.49	6.83
	4	17.83	15.88	13.73	12.26	7.45	29.36	24.73	22.66	20.96	13.81	48.12	24.61	21.55	18.16	11.72	46.37	28.61	24.86	19.98	13.65	16.56	14.65	12.37	9.88	5.86	17.79	15.35	12.95	10.57	6.29

Table 2. Temporal Understanding Benchmarks: Step-level (TAS-S) and Bimanual Atomic Actions (TAS-B)

Backbone	CV-TA (cosine kNN)									CV-SM Retrieval (cosine kNN)												CV-SM Classification (linear probe)
	Local			Global			Exo $\rightarrow$ Ego			Split 1			Split 2			Split 3			Split 4			Split 1		Split 2		Split 3		Split 4
	R@1 $\uparrow$	R@5 $\uparrow$	MdR $\downarrow$	R@1 $\uparrow$	R@5 $\uparrow$	MdR $\downarrow$	R@1 $\uparrow$	R@5 $\uparrow$	MdR $\downarrow$	R@1 $\uparrow$	R@5 $\uparrow$	mAP $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	mAP $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	mAP $\uparrow$	R@1 $\uparrow$	R@5 $\uparrow$	mAP $\uparrow$	Top-1 $\uparrow$	MF1 $\uparrow$	Top-1 $\uparrow$	MF1 $\uparrow$	Top-1 $\uparrow$	MF1 $\uparrow$	Top-1 $\uparrow$	MF1 $\uparrow$
I3D ( Carreira and Zisserman, 2017 )	4.95	21.50	13.67	0.69	2.24	333.00	3.79	18.86	14.17	5.46	22.56	5.36	5.08	20.90	5.53	5.53	24.03	6.78	5.78	22.08	5.41	39.99	40.41	40.98	41.88	18.01	14.57	41.62	41.64
VideoMAEv2 ( Wang et al., 2023a )	4.02	19.36	13.67	0.32	1.41	324.08	3.43	18.65	14.33	5.44	20.29	5.30	5.08	20.57	5.48	8.64	28.51	7.06	6.11	21.21	5.38	24.31	23.86	19.75	18.81	15.01	11.89	26.39	25.18
MViTv2 ( Li et al., 2022 )	7.38	29.18	10.17	1.40	5.35	162.25	5.12	21.38	13.00	6.24	24.47	5.54	5.82	23.52	5.80	6.37	25.21	7.13	6.38	24.87	5.58	54.10	53.97	53.44	54.17	23.35	20.51	54.83	54.87

Table 3. Cross-View Understanding Benchmarks: Temporal Correspondence (CV-TA) and Semantic Matching (CV-SM)

Model	Feat. Ext.	Overall			Left			Right
Model	Feat. Ext.	act $\uparrow$	verb $\uparrow$	noun $\uparrow$	act $\uparrow$	verb $\uparrow$	noun $\uparrow$	act $\uparrow$	verb $\uparrow$	noun $\uparrow$
AVT ( Girdhar and Grauman, 2021 )	I3D ( Carreira and Zisserman, 2017 )	17.34	42.61	54.20	17.33	45.32	50.81	15.10	38.28	50.95
AVT ( Girdhar and Grauman, 2021 )	VideoMAEv2 ( Wang et al., 2023a )	10.61	33.10	42.95	10.70	32.90	39.77	10.40	30.97	40.49
ScalAnt ( Zhong et al., 2026 )	I3D ( Carreira and Zisserman, 2017 )	18.41	44.44	53.42	18.36	45.80	49.30	15.70	40.92	50.52
ScalAnt ( Zhong et al., 2026 )	VideoMAEv2 ( Wang et al., 2023a )	13.89	38.80	48.39	13.32	40.58	46.75	13.16	36.18	45.61
Qwen3VL-8B ( Bai et al., 2025 )	–	4.86	6.99	32.14	5.95	6.90	33.28	4.26	6.96	31.53

Table 4. Action Forecasting Benchmarks.

Method	ASR Metrics			PSR Metrics
Method	Macro-F1 $\uparrow$	Trans-F1 $\uparrow$	Final-Acc $\uparrow$	POS $\uparrow$	F1 $\uparrow$	$\tau$ (s) $\downarrow$
MS-TCN++ ( Li et al., 2020 )	0.84	0.33	0.93	0.21	0.26	8.46
VideoMAEv2 ( Wang et al., 2023a )	0.38	0.01	0.33	0.00	0.01	24.20
STORM-PSR ( Schoonbeek et al., 2025 )	0.46	0.03	0.51	0.00	0.01	22.66
Gemini 3.1 Pro ( Team et al., 2023 )	0.44	0.10	0.81	0.36	0.23	19.26

Table 5. State & Reasoning Benchmarks.

Explore similar work

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects
May 13, 2026Danrui Li, Jiahao Zhang, Bernhard Egger +4Object Pose Estimation Robot Manipulation Benchmarks
Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany +1
PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing
Aug 18, 2026Tengbo Yu, Jiahao Wu, Hanning Wang +4Robotic Data Collection Robot Manipulation Benchmarks
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University
OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments
Mar 2, 2026Hymalai Bello, Lala Ray, Joanna Sorysz +2Human Activity Recognition Video Action Recognition
DFKI Kaiserslautern · RPTU Kaiserslautern-Landau · Korea University

May 13, 2026J/K move · Enter open · S save

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects

Danrui Li, Jiahao Zhang, Bernhard Egger, Moitreya Chatterjee+3

Rutgers, The State University of New Jersey, USA · The Australian National University, Australia · Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany+1

Object Pose Estimation Robot Manipulation Benchmarks Robotic Assembly Trajectory Prediction

Open paper AskPDF arXiv

Assembling objects from parts requires understanding multimodal instructions, linking them to 3D components, and predicting physically plausible 6-DoF motions for each assembly step. Existing datasets focus on simplified scenarios, overlooking shape complexities and assembly trajectories in industrial assemblies. We introduce AssemblyBench, a synthetic dataset of 2,789 industrial objects with multimodal instruction manuals, corresponding 3D part models, and part assembly trajectories. We also propose a transformer-based model, AssemblyDyno, which uses the instructional manual and the 3D shape of each part to jointly predict assembly order and part assembly trajectories. AssemblyDyno outperforms prior works in both assembly pose estimation and trajectory feasibility, where the latter is evaluated by our physics-based simulations.

Abstract

Explore similar work

AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects

PRISM: Precision and contact-rich Real-world Industrial Skill dataset with Multimodal sensing

OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments