From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly

Yanyi Chen; Min Deng

cs.ROJan 2, 2026

From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly

Authors: Yanyi Chen, Min Deng

Organizations: Department of Civil, Environmental, and Construction Engineering, Texas Tech University, Lubbock, TX 79409, USA

Abstract

Human-robot collaboration (HRC) in structured assembly requires reliable state estimation and adaptive task planning under noisy perception and human interventions. To address these challenges, we introduce a design-grounded human-aware planning framework for human-robot collaborative structured assembly. The framework comprises two coupled modules. Module I, Perception-to-Symbolic State (PSS), employs vision-language models (VLMs) based agents to align RGB-D observations with design specifications and domain knowledge, synthesizing verifiable symbolic assembly states. It outputs validated installed and uninstalled component sets for online state tracking. Module II, Human-Aware Planning and Replanning (HPR), performs task-level multi-robot assignment and updates the plan only when the observed state deviates from the expected execution outcome. It applies a minimal-change replanning rule to selectively revise task assignments and preserve plan stability even under human interventions. We validate the framework on a 27-component timber-frame assembly. The PSS module achieves 97% state synthesis accuracy, and the HPR module maintains feasible task progression across diverse HRC scenarios. Results indicate that integrating VLM-based perception with knowledge-driven planning improves robustness of state estimation and task planning under dynamic conditions.

Figures & tables

Symbol	Description
$\mathcal{D}$	Design artifact (e.g., CAD, BIM, or PDF)
$V$	Set of design components
$\mathbf{A}$	Multidimensional component array extracted from $\mathcal{D}$
$I^{\mathrm{des}}$	Design image rendered from $\mathcal{D}$
$O_{t}$	RGB-D observation at iteration $t$ , consisting of $I_{t}^{RGB}$ and $I_{t}^{Depth}$
$I_{t}^{RGB}$	RGB images at iteration $t$

TABLE I: Definition of notation.

Fig. 1: Overview of the proposed planning framework, showing the interaction between the Perception-to-Symbolic State module and Human-Aware Planning and Replanning module in a closed-loop perception-to-task-planning process.

Fig. 2: Overview of the Perception-to-Symbolic State module, which synthesizes a verified symbolic assembly state from design specifications and RGB-D observations.

Fig. 3: Structure of the multidimensional component array $\mathbf{A}$ .

Sub-array	Definition	Functional Role
$\mathbf{A}^{id}$	Defines component identifiers and reference keys.	Supports indexing across perception and planning.
$\mathbf{A}^{geo}$	Defines component geometry.	Supports design alignment and feasibility checking.
$\mathbf{A}^{type}$	Defines the functional type of each component.	Constrains interpretation in perception and action selection in planning.
$\mathbf{A}^{layer}$	Defines layering and grouping relations.	Supports multi-layer reasoning under occlusion.
$\mathbf{A}^{topo}$	Defines structural relations.	Supports dependency checking and admissible frontier computation.
$\mathbf{A}^{phys}$	Defines physical properties.	Supports capability-aware task allocation.

TABLE II: Definition and functional roles of the component array

\mathbf{A}

Fig. 4: Design-aligned perceptual representation pipeline for converting raw RGB-D observations into structured geometric primitives.

Fig. 5: Example layout for the case study. The configuration includes two heterogeneous collaborative robots, an overhead RGB-D camera, and the design coordinate frame registered to the workspace.

Fig. 6: Pipeline for generating the ontological representation.

Fig. 7: Representative example of the arrays in $\mathbf{A}$ for the component window_1_header_layer_1 .

Fig. 8: Design-aligned perceptual representation pipeline results across four evaluation groups (G1-G4).

Fig. 9: VLM-based symbolic inference workflow, illustrating the interaction between the Match Agent, the Filter Agent, and rule-based consistency reconciliation.

Fig. 10: Closed-loop execution trace of the full timber-frame wall assembly, illustrating task progression and replanning events.

	condition	design	Appearance confounders	Peripheral clutter	Human occlusion
G1	Indoor	Gable-wall	0.16	0.42	10%
G2	Indoor	Gable-wall	6.18	7.27	29%
G3	Outdoor	Gable-wall	0.33	2.30	3%
G4	Indoor	Stair-wall	3.34	3.55	15%

TABLE III: Mean perceptual complexity descriptors by group (G1-G4).

Model	G1			G2			G3			G4
	Base	Grounded	Verified	Base	Grounded	Verified	Base	Grounded	Verified	Base	Grounded	Verified
GPT-5	52%	95%	95%	51%	89%	98% $\uparrow$	49%	83%	83%	60%	74%	81%
Gemini-3-Pro	42%	82%	82%	46%	74%	83%	61%	48% $\downarrow$	48%	51%	80%	85% $\uparrow$
Claude-Opus-4.5	24%	71%	71%	21%	40%	85%	28%	34%	34%	26%	55%	64%

TABLE IV: VLM symbolic matching correct rate by group and inference stage (%).

Fig. 11: Match-stage logistic regression by VLM: ORs with 95% CI

		Team-I: heterogeneous (2 robots)				Team-II: homogeneous (3 robots)
Human	Planner	Cycles	Workload	Plan Edit Dist.	Time (ms)	Cycles	Workload	Plan Edit Dist.	Time (ms)
No-Int.	HPR	$\mathbf{17.00\pm 0.00}$	$\mathbf{4.24\pm 0.00}$	—	$\mathbf{1.11\pm 0.22}$	$\mathbf{14.00\pm 0.00}$	3.51 $\pm$ 0.00	—	$\mathbf{1.08\pm 0.26}$
	Greedy	$\mathbf{17.00\pm 0.00}$	$\mathbf{4.24\pm 0.00}$	—	1.24 $\pm$ 0.30	$\mathbf{14.00\pm 0.00}$	4.93 $\pm$ 0.00	—	1.20 $\pm$ 0.33
	M-MILP	18.00 $\pm$ 0.00	$\mathbf{4.24\pm 0.00}$	—	19.16 $\pm$ 3.14	$\mathbf{14.00\pm 0.00}$	$\mathbf{0.58\pm 0.00}$	—	30.74 $\pm$ 6.53
Random	HPR	17.87 $\pm$ 5.16	3.52 $\pm$ 1.01	$\mathbf{0.00}$	$\mathbf{0.96\pm 0.19}$	14.03 $\pm$ 4.09	2.70 $\pm$ 0.71	$\mathbf{0.00}$	$\mathbf{0.96\pm 0.25}$
	Greedy	$\mathbf{17.82\pm 5.28}$	3.48 $\pm$ 0.93	0.13	1.12 $\pm$ 0.35	14.71 $\pm$ 5.68	3.64 $\pm$ 0.90	0.73	1.21 $\pm$ 0.44

TABLE V: Factorial planning benchmark: mean

\pm

std over 100 trials.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Processing Stage	Parameter	Value
Multimodal Scene	Number of clusters ( $K$ )	5
Segmentation	Feature weights ( $w_{x},w_{y},w_{z}$ )	0.6, 0.6, 0.2
	Depth norm. percentiles	1%-99%
Geometric	Morphological gradient kernel	$3\times 3$
Primitive	Canny thresholds (wood mask)	30, 100
Extraction	Canny thresholds (RGB)	50, 150

Appendix

Table 17

Explore similar work

CoViLLM: An Adaptive Human-Robot Collaborative Assembly Framework Using Large Language Models
Mar 12, 2026Jiabao Zhao, Jonghan Lim, Hongliang Li +1Human-Robot Collaboration Robotic Assembly
The Pennsylvania State University, University Park, PA, USA
Robust Assembly State Reasoning from Action Recognition for Human-Robot Collaboration
Jun 18, 2026James Fant-Male, Roel PietersHuman-Robot Collaboration Robotic Assembly
Cognitive Robotics group, Unit of Automation Technology and Mechanical Engineering, Tampere University, 33720, Tampere, Finland
Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction
Feb 16, 2026Taichi Kato, Takuya Kiyokawa, Namiko Saito +1Language-Conditioned Robot Manipulation Large Language Model-Based Robot Planning
aDepartment of Systems Science, School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan · bDepartment of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan · cMicrosoft Research Asia, Shinagawa, Tokyo, Japan +1

Mar 12, 2026J/K move · Enter open · S save

CoViLLM: An Adaptive Human-Robot Collaborative Assembly Framework Using Large Language Models

Jiabao Zhao, Jonghan Lim, Hongliang Li, Ilya Kovalenko

The Pennsylvania State University, University Park, PA, USA

Human-Robot Collaboration Robotic Assembly Generalization in Robotic Manipulation Large Language Model-Based Robot Planning

Open paper AskPDF arXiv

With increasing demand for mass customization, traditional manufacturing robots that rely on rule-based operations lack the flexibility to accommodate customized or new product variants. Human-Robot Collaboration has demonstrated potential to improve system adaptability by leveraging human versatility and decision-making capabilities. However, existing Human-Robot Collaborative frameworks typically depend on predefined perception-manipulation pipelines, limiting their ability to autonomously generate task plans for new product assembly. In this work, we propose CoViLLM, an adaptive human-robot collaborative assembly framework that supports the assembly of customized and previously unseen products. CoViLLM combines depth-camera-based localization for object position estimation, human operator classification for identifying new components, and a Large Language Model for assembly task planning based on natural language instructions. The framework is validated on the NIST Assembly Task Board for known, customized, and new product cases. Experimental results show that the proposed framework enables flexible collaborative assembly by extending Human-Robot Collaboration beyond predefined product and task settings.

Abstract

Explore similar work

CoViLLM: An Adaptive Human-Robot Collaborative Assembly Framework Using Large Language Models

Robust Assembly State Reasoning from Action Recognition for Human-Robot Collaboration

Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction