From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs

Federico Toschi; Nicolò Brunello; Andrea Sassella; Vincenzo Scotti; Mark James Carman

cs.CVMar 20, 2026

From Instructions to Assistance: a Dataset Aligning Instruction Manuals with Assembly Videos for Evaluating Multimodal LLMs

Authors: Federico Toschi, Nicolò Brunello, Andrea Sassella, Vincenzo Scotti, Mark James Carman

Organizations: DEIB, Politecnico di Milano, Via Ponzio 34/5, 20133, Milano (MI), Italy · KASTEL, Karlsruhe Institute of Technology, Am Fasanengarten, 576131, Karlsruhe, Germany

Abstract

The recent advancements introduced by Large Language Models (LLMs) have transformed how Artificial Intelligence (AI) can support complex, real world tasks, pushing research outside the text boundaries towards multi modal contexts and leading to Multimodal Large Language Models (MLMs). Given the current adoption of LLM based assistants in solving technical or domain specific problems, the natural continuation of this trend is to extend the input domains of these assistants exploiting MLMs. Ideally, these MLMs should be used as real time assistants in procedural tasks, hopefully integrating a view of the environment where the user being assisted is, or even better sharing the same point of view via Virtual Reality (VR) or Augmented Reality (AR) supports, to reason over the same scenario the user is experiencing. With this work, we aim at evaluating the quality of currently openly available MLMs to provide this kind of assistance on technical tasks. To this end, we annotated a data set of furniture assembly with step by step labels and manual references: the Manual to Action Dataset (M2AD). We used this dataset to assess (1) to which extent the reasoning abilities of MLMs can be used to reduce the need for detailed labelling, allowing for more efficient, cost effective annotation practices, (2) whether MLMs are able to track the progression of assembly steps (3) and whether MLMs can refer correctly to the instruction manual pages. Our results showed that while some models understand procedural sequences, their performance is limited by architectural and hardware constraints, highlighting the need for multi image and interleaved text image reasoning.

Figures & tables

	Total	Avg.	Std.	Min	Max
No. of Annotations	$1228.0$
Frames per Step		$1365.3$
No. of annotated frames		$1\,676\,677.8$
No. of steps per video		$23.2$	$16.2$	$2.0$	$71.0$
Step duration [s]		$21.1$	$20.5$	$1.0$	$176.0$
Completion time [s]		$652.5$	$465.4$

Table 1: dataset summary statistics for the 53 furniture items considered.

Figure 3: Heatmap showcasing the trend in non-consecutive step transition lenghts. Warm colours indicate positive transitions (skipping), and cold colours indicate negative transitions (re-visiting).

Figure 4: Example of an IoU computation for IAA.

Model	Metric [%]
Model	Accuracy	Precision	Recall	F1
LLaVa-Video	$\mathbf{56.79}$	$59.13$	$56.79$	$53.84$
LLaVa-OneVision	$53.42$	$57.16$	$53.42$	$46.41$
MFuyu	$50.04$	$50.62$	$50.04$	$34.81$
MolMo	$50.12$	$56.84$	$50.12$	$33.88$
Ovis	$55.65$	$55.67$	$56.65$	$55.62$

Table 2: Step completion detection results ( best score , second-best score ).

Model	Metric [%]
Model	Accuracy	Precision	Recall	F1
LLaVa-Video	$\mathbf{59.16}$	$59.34$	$59.16$	$58.95$
LLaVa-OneVision	$50.00$	$50.00$	$50.00$	$35.12$
MFuyu	$50.40$	$51.25$	$50.40$	$40.31$
MolMo	50.00	25.00	50.00	33.33
Ovis	$50.04$	$58.34$	$50.04$	$33.49$

Table 3: Step detection results ( best score , second-best score ).

Model	Metric [%]
Model	Accuracy	Precision	Recall	F1
LLaVa-Video	23.77	$73.64$	$13.50$	$19.03$
LLaVa OneVision	$20.35$	$22.64$	$31.26$	$23.94$
MFuyu	$21.82$	$38.66$	$29.40$	$30.04$
MolMo	$\mathbf{79.56}$	$86.53$	$85.78$	$86.03$
Ovis	$14.49$	$38.18$	$9.02$	$12.82$

Table 4: Step identification results ( best score , second-best score ).

Multimodal Large Language Models

Multimodal large language models combine language models with image, video, audio, and other inputs for grounded understanding and generation.

See all 1,060 papers

AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly
Oct 6, 2026Zhiyuan Qi, Jierui Li, Yifan Shen +2Robotic Assembly 3D Spatial Reasoning
Tsinghua University · University of Illinois Urbana-Champaign · Xidian University
Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment
Apr 1, 2026Zhuchenyang Liu, Yao Zhang, Yu XiaoVision-Language Alignment VLM Robustness
Aalto University, Espoo, Finland
MUSE: A Unified Agentic Harness for MLLMs
Jun 2, 2026Jianglin Lu, Hailing Wang, Xu Ma +4Multimodal Reasoning Multimodal Agents
Northeastern University

Oct 6, 2026J/K move · Enter open · S save

AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly

Zhiyuan Qi, Jierui Li, Yifan Shen, Cheng Qian+1

Tsinghua University · University of Illinois Urbana-Champaign · Xidian University

Robotic Assembly 3D Spatial Reasoning VLMs for Robotics

Open paper AskPDF arXiv

Multimodal large language models (MLLMs) have made significant progress in visual understanding, but precise 3D spatial reasoning integrated with physical environment remains difficult. Furniture assembly requires not only recovering step-level operations from diagrammatic manuals, but also translating semantic attachment relations into 6D pose updates that enable parts to physically interact with the environment and previously assembled components. To study this problem, we propose AssemState, a zero-shot framework for manual and physical-state-guided furniture assembly. It firstly employs anchor-guided boundary assembly states to decompose manual pages into single-part operations and recover an assembly-tree. Then, it uses iterative after-state feedback refinement to guide successive (SE(3)) updates and corrections, and validates their physical plausibility through simulation-based release tests. Experiments show that compared with the strongest prior baseline, AssemState improves F1 from 38.58% to 62.80% and Tree Exact Match from 28.24% to 53.92% for assembly-tree recovery. On 243 independently evaluated part-level operations, our proposed iterative refinement improves judge-accepted operations from 0 to 5.3% and reduces mean Chamfer distance from 5.4111 to 1.7744. However, visually plausible candidate poses may still suffer from collision, floating, mirror-orientation errors, incomplete seating, and wrong-side attachment. These results show that AssemState improves operation-structure recovery and selected local pose metrics, while MLLMs remain limited for spatial relationship reasoning.

Abstract

Multimodal Large Language Models

Explore similar work

AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly

Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

MUSE: A Unified Agentic Harness for MLLMs