Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

Zhuchenyang Liu; Yao Zhang; Yu Xiao

cs.CVApr 1, 2026

Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment

Authors: Zhuchenyang Liu, Yao Zhang, Yu Xiao

Organizations: Aalto University, Espoo, Finland

Abstract

2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions. Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-video alignment; (2) architecture family predicts alignment accuracy more strongly than parameter count; (3) video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: https://ryenhails.github.io/IKEA-Bench/

Figures & tables

Figure 1. (a) Illustration of 2D-manual-based assembly guidance that leverages a VLM to align schematic diagrams with real-time camera feed. (b) The depiction gap: diagrams and video frames share no visual features despite depicting the same step. (c) The foundational alignment level (L1) remains unsolved. Three panels. Left: application scenario showing a 2D manual and camera feed input to a VLM. Center: the depiction gap between silhouette-style diagrams and photorealistic video. Right: three-level capability hierarchy with L1 highlighted and L2-L3 grayed out.

Figure 2. One example per task type. T1–T4 test cross-depiction alignment at increasing difficulty; D1–D2 isolate video and assembly instruction understanding respectively. Green borders mark correct options. Full prompt templates are in Suppl. A. Six panels in a 3x2 grid, each showing one task type with video frames, diagram options, question, and correct answer highlighted in green.

Task	Input	Format	$N$	Chance
T1	V+4P	4-MC (image)	320	25%
T2	V+1P	Binary	350	50%
T3	V+all P	4-MC (text)	334	25%
T4	V+4P	4-MC (image)	204	25%
D1	V+V	Binary	350	50%
D2	3P	4-MC (text)	65	25%

Table 1. Task summary.

Strategy	Plan modality	Execution	Depiction gap?
Visual	Diagram image	Video	Full gap
Visual+Text	Diagram + text	Video	Gap + text bridge
Text Only	Text description	Video	Eliminated

Table 2. Alignment strategies.

Family	Sizes	Architecture
Qwen2.5-VL	3B, 7B	Dense, M-RoPE
Qwen3-VL	2B, 8B, 30B-A3B	Dense/MoE, DeepStack
Qwen3.5	2B, 9B, 27B	Dense, GDN
InternVL3.5	2B, 8B, 38B	Dense, InternViT
Gemma3	4B, 12B, 27B	Dense
GLM-4.1V	9B	Dense, thinking

Table 3. Evaluated model families.

Family	Model	Size	T1 Match	T2 Verify	T3 Locate	T4 Predict	D1 Exec.	D2 Plan
Qwen2.5-VL	Qwen2.5-VL-3B	3B	42.8	51.1	35.6	28.9	48.3	52.3
	Qwen2.5-VL-7B	7B	49.1	50.9	35.0	36.8	46.0	53.8
Qwen3-VL	Qwen3-VL-2B	2B	42.2	50.0	29.6	34.8	50.0	26.2
	Qwen3-VL-8B	8B	53.1	56.6	49.4	39.7	58.3	58.5
	Qwen3-VL-30B-A3B	30B	48.8	58.3	50.6	34.3	60.0	56.9
Qwen3.5	Qwen3.5-2B	2B	44.4	56.6	32.9	36.3	51.4	36.9

Table 4. Main results under the Visual (baseline) setting. Best open-source per column in bold (top), proprietary in bold (bottom). Full results (V+T, Text Only) in Suppl. C.

Figure 3. T1 accuracy vs. model size (log scale). Different families scale at different rates, and generational improvements at

\sim

8B consistently exceed within-family scaling. Line chart of T1 accuracy vs model size showing non-parallel scaling curves across families.

Figure 4. Average accuracy across 17 models per task and strategy. D1 is invariant to strategy (hard video ceiling), while D2 shows massive recovery with text (+23.6 pp). Grouped bar chart showing 6 tasks under 3 strategies. D1 bars are nearly identical; D2 bars increase sharply from Visual to Text Only.

Task	V $\to$ V+T	V $\to$ Text	Interpretation
T1 (match)	$-$ 3.1	$-$ 5.1	Text hurts alignment
T2 (verify)	+1.6	$-$ 0.7	Roughly neutral
T3 (locate)	$-$ 3.2	$-$ 4.2	Text hurts alignment
T4 (predict)	+0.5	$-$ 1.7	Roughly neutral
D1 (video)	+0.1	+0.1	No effect (expected)
D2 (instr.)	+19.5	+23.6	Text massively helps

Table 5. Average strategy effect across 17 models.

	CKA		Probe Acc		Retrieval
Model	ViT	Merger	ViT	Merger	R@1	R@10
Qwen2.5-VL-7B	.006	.137	48.9	61.3	5.3	15.0
Qwen3-VL-8B	.001	.235	68.7	66.8	1.8	8.0
Qwen3.5-VL-9B	.006	.105	52.8	61.5	4.4	12.4
InternVL3.5-8B	.101	.075	52.7	68.3	0.0	8.8

Table 6. Layer 1: representation analysis across 4 models on 113 step pairs. CKA: diagram–video representational alignment (1 = identical, 0 = disjoint). Probe: video frame same/different step classification (chance = 50%). R@

K

: diagram

\to

video retrieval at the merger level (gallery = 2,546 frames).

Figure 5. Diagram influence on the LLM prediction representation, measured by cosine similarity between the last-token hidden state and diagram token hidden states. Adding text (V+T) reduces diagram influence in 3 of 4 models (

n

=100 T1 questions per model). Grouped bar chart of cosine similarity for 4 models under Visual vs V+T.

Figure 6. Per-modality attention share in Qwen3-VL-8B on T1 (

n

=100). Adding text halves diagram attention (

-

52%) and reduces video attention (

-

34%). Horizontal stacked bars showing attention redistribution from diagrams and video toward text under V+T.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Asset	Count
Furniture products (6 categories)	36 (29 used)
Assembly videos	98
Manual step diagrams (wordless PNG)	133
Text descriptions (LLM-generated)	132
Video frames (extracted)	2,569
Benchmark questions	1,623

Appendix

Table 7. Source dataset statistics (IKEA Manuals at Work).

Product	Category	T1	T2	T3	T4	D1	D2	Total
applaro_3	Bench	3	10	3	0	3	1	20
tjusig	Bench	20	16	22	12	21	3	94
applaro	Chair	3	10	3	0	3	1	20
bekvam_3017	Chair	30	16	30	24	23	3	126
grubban	Chair	8	16	8	6	13	2	53
ingolf	Chair	0	4	1	0	2	0	7

Appendix

Table 8. Per-product benchmark statistics. Products sorted by category.

Figure 7. D2 Visual vs. Text Only accuracy for 17 VLMs. All models fall below the diagonal: visual assembly instruction understanding consistently lags text-based reasoning. Marker size indicates parameter count. Scatter plot with all 17 models below the y=x diagonal. The shaded region below the diagonal is labeled Diagram Blindness.

Vision-Language Alignment

Vision-language alignment connects visual and textual representations so models can match, interpret, and ground language in images and video.

See all 351 papers

Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
May 20, 2026Aditya Chetan, Eric Cai, Peeyush Kushwaha +5VLM Evaluation Temporal Video Grounding
Cornell University · Independent researcher · MBZUAI +2
Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models
Sep 3, 2026Nicholas Sansoterra, Zishuo Zheng, Sachin KumarVLM Robustness Adversarial Attacks on VLMs
The Ohio State University
Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos
Jul 12, 2026Hao Zheng, Jinyi Huang, Tiantian Zheng +2VLM Adaptation Video Action Recognition
Department of Computer Engineering, New York University Abu Dhabi, UAE · Department of Mechanical and Mechatronics Engineering, The University of Auckland, New Zealand · Center for Quantum and Topological Systems, New York University Abu Dhabi, UAE +1

May 20, 2026J/K move · Enter open · S save

Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly

Aditya Chetan, Eric Cai, Peeyush Kushwaha, Bharath Raj Nagoor Kani+4

Cornell University · Independent researcher · MBZUAI+2

VLM Evaluation Temporal Video Grounding Visual Spatial Reasoning Temporal Video Understanding

Open paper AskPDF arXiv

The emergence of Large Vision-Language Models (LVLMs) has significantly advanced video understanding capabilities. However, existing benchmarks focus predominantly on coarse-grained tasks such as action segmentation, classification, captioning, and retrieval. Furthermore, these benchmarks often rely on entities that can be easily identified verbally, like household objects, animals, human subjects, etc., limiting their applicability to complex, in-the-wild video scenarios. But, many applications such as furniture assembly, cooking, etc., require step-by-step fine-grained spatio-temporal understanding of the video, which is not sufficiently evaluated in current benchmarks. To address this gap, we introduce Flat-Pack Bench, a novel benchmark centered on furniture assembly tasks. Our benchmark evaluates LVLMs on nuanced tasks, including temporal ordering of assembly actions, temporal localization of assembly state, understanding part mating, and tracking, using multiple-choice questions paired with visual prompts highlighting relevant parts as references for fine-grained questions. Our experiments reveal that state-of-the-art LVLMs struggle significantly with fine-grained spatio-temporal reasoning, highlighting their limitations in effectively leveraging temporal information from videos, limited tracking ability, and understanding of spatial interactions like physical contact.

Abstract

Vision-Language Alignment

Explore similar work

Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly

Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models

Compositional Context Fine-Tuning Vision-Language Model for Complex Assembly Action Understanding from Videos