From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly

Yanyi Chen; Min Deng

cs.ROJan 2, 2026

From Perception to Symbolic Task Planning: Vision-Language Guided Human-Robot Collaborative Structured Assembly

Authors: Yanyi Chen, Min Deng

Organizations: Department of Civil, Environmental, and Construction Engineering, Texas Tech University, Lubbock, TX 79409, USA

Abstract

Human-robot collaboration (HRC) in structured assembly requires reliable state estimation and adaptive task planning under noisy perception and human interventions. To address these challenges, we introduce a design-grounded human-aware planning framework for human-robot collaborative structured assembly. The framework comprises two coupled modules. Module I, Perception-to-Symbolic State (PSS), employs vision-language models (VLMs) based agents to align RGB-D observations with design specifications and domain knowledge, synthesizing verifiable symbolic assembly states. It outputs validated installed and uninstalled component sets for online state tracking. Module II, Human-Aware Planning and Replanning (HPR), performs task-level multi-robot assignment and updates the plan only when the observed state deviates from the expected execution outcome. It applies a minimal-change replanning rule to selectively revise task assignments and preserve plan stability even under human interventions. We validate the framework on a 27-component timber-frame assembly. The PSS module achieves 97% state synthesis accuracy, and the HPR module maintains feasible task progression across diverse HRC scenarios. Results indicate that integrating VLM-based perception with knowledge-driven planning improves robustness of state estimation and task planning under dynamic conditions.

Figures & tables

Symbol	Description
$\mathcal{D}$	Design artifact (e.g., CAD, BIM, or PDF)
$V$	Set of design components
$\mathbf{A}$	Multidimensional component array extracted from $\mathcal{D}$
$I^{\mathrm{des}}$	Design image rendered from $\mathcal{D}$
$O_{t}$	RGB-D observation at iteration $t$ , consisting of $I_{t}^{RGB}$ and $I_{t}^{Depth}$
$I_{t}^{RGB}$	RGB images at iteration $t$

TABLE I: Definition of notation.

Fig. 1: Overview of the proposed planning framework, showing the interaction between the Perception-to-Symbolic State module and Human-Aware Planning and Replanning module in a closed-loop perception-to-task-planning process.

Fig. 2: Overview of the Perception-to-Symbolic State module, which synthesizes a verified symbolic assembly state from design specifications and RGB-D observations.

Fig. 3: Structure of the multidimensional component array $\mathbf{A}$ .

Sub-array	Definition	Functional Role
$\mathbf{A}^{id}$	Defines component identifiers and reference keys.	Supports indexing across perception and planning.
$\mathbf{A}^{geo}$	Defines component geometry.	Supports design alignment and feasibility checking.
$\mathbf{A}^{type}$	Defines the functional type of each component.	Constrains interpretation in perception and action selection in planning.
$\mathbf{A}^{layer}$	Defines layering and grouping relations.	Supports multi-layer reasoning under occlusion.
$\mathbf{A}^{topo}$	Defines structural relations.	Supports dependency checking and admissible frontier computation.
$\mathbf{A}^{phys}$	Defines physical properties.	Supports capability-aware task allocation.

TABLE II: Definition and functional roles of the component array

\mathbf{A}

Fig. 4: Design-aligned perceptual representation pipeline for converting raw RGB-D observations into structured geometric primitives.

Fig. 5: Example layout for the case study. The configuration includes two heterogeneous collaborative robots, an overhead RGB-D camera, and the design coordinate frame registered to the workspace.

Fig. 6: Pipeline for generating the ontological representation.

Fig. 7: Representative example of the arrays in $\mathbf{A}$ for the component window_1_header_layer_1 .

Fig. 8: Design-aligned perceptual representation pipeline results across four evaluation groups (G1-G4).

Fig. 9: VLM-based symbolic inference workflow, illustrating the interaction between the Match Agent, the Filter Agent, and rule-based consistency reconciliation.

Fig. 10: Closed-loop execution trace of the full timber-frame wall assembly, illustrating task progression and replanning events.

	condition	design	Appearance confounders	Peripheral clutter	Human occlusion
G1	Indoor	Gable-wall	0.16	0.42	10%
G2	Indoor	Gable-wall	6.18	7.27	29%
G3	Outdoor	Gable-wall	0.33	2.30	3%
G4	Indoor	Stair-wall	3.34	3.55	15%

TABLE III: Mean perceptual complexity descriptors by group (G1-G4).

Model	G1			G2			G3			G4
	Base	Grounded	Verified	Base	Grounded	Verified	Base	Grounded	Verified	Base	Grounded	Verified
GPT-5	52%	95%	95%	51%	89%	98% $\uparrow$	49%	83%	83%	60%	74%	81%
Gemini-3-Pro	42%	82%	82%	46%	74%	83%	61%	48% $\downarrow$	48%	51%	80%	85% $\uparrow$
Claude-Opus-4.5	24%	71%	71%	21%	40%	85%	28%	34%	34%	26%	55%	64%

TABLE IV: VLM symbolic matching correct rate by group and inference stage (%).

Fig. 11: Match-stage logistic regression by VLM: ORs with 95% CI

		Team-I: heterogeneous (2 robots)				Team-II: homogeneous (3 robots)
Human	Planner	Cycles	Workload	Plan Edit Dist.	Time (ms)	Cycles	Workload	Plan Edit Dist.	Time (ms)
No-Int.	HPR	$\mathbf{17.00\pm 0.00}$	$\mathbf{4.24\pm 0.00}$	—	$\mathbf{1.11\pm 0.22}$	$\mathbf{14.00\pm 0.00}$	3.51 $\pm$ 0.00	—	$\mathbf{1.08\pm 0.26}$
	Greedy	$\mathbf{17.00\pm 0.00}$	$\mathbf{4.24\pm 0.00}$	—	1.24 $\pm$ 0.30	$\mathbf{14.00\pm 0.00}$	4.93 $\pm$ 0.00	—	1.20 $\pm$ 0.33
	M-MILP	18.00 $\pm$ 0.00	$\mathbf{4.24\pm 0.00}$	—	19.16 $\pm$ 3.14	$\mathbf{14.00\pm 0.00}$	$\mathbf{0.58\pm 0.00}$	—	30.74 $\pm$ 6.53
Random	HPR	17.87 $\pm$ 5.16	3.52 $\pm$ 1.01	$\mathbf{0.00}$	$\mathbf{0.96\pm 0.19}$	14.03 $\pm$ 4.09	2.70 $\pm$ 0.71	$\mathbf{0.00}$	$\mathbf{0.96\pm 0.25}$
	Greedy	$\mathbf{17.82\pm 5.28}$	3.48 $\pm$ 0.93	0.13	1.12 $\pm$ 0.35	14.71 $\pm$ 5.68	3.64 $\pm$ 0.90	0.73	1.21 $\pm$ 0.44

TABLE V: Factorial planning benchmark: mean

\pm

std over 100 trials.

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Processing Stage	Parameter	Value
Multimodal Scene	Number of clusters ( $K$ )	5
Segmentation	Feature weights ( $w_{x},w_{y},w_{z}$ )	0.6, 0.6, 0.2
	Depth norm. percentiles	1%-99%
Geometric	Morphological gradient kernel	$3\times 3$
Primitive	Canny thresholds (wood mask)	30, 100
Extraction	Canny thresholds (RGB)	50, 150

Appendix

Table 17

Explore similar work

Mar 12, 2026

CoViLLM: An Adaptive Human-Robot Collaborative Assembly Framework Using Large Language Models

With increasing demand for mass customization, traditional manufacturing robots that rely on rule-based operations lack the flexibility to accommodate customized or new product variants. Human-Robot Collaboration has demonstrated potential to improve system adaptability by leveraging human versatility and decision-making capabilities. However, existing Human-Robot Collaborative frameworks typically depend on predefined perception-manipulation pipelines, limiting their ability to autonomously generate task plans for new product assembly. In this work, we propose CoViLLM, an adaptive human-robot collaborative assembly framework that supports the assembly of customized and previously unseen products. CoViLLM combines depth-camera-based localization for object position estimation, human operator classification for identifying new components, and a Large Language Model for assembly task planning based on natural language instructions. The framework is validated on the NIST Assembly Task Board for known, customized, and new product cases. Experimental results show that the proposed framework enables flexible collaborative assembly by extending Human-Robot Collaboration beyond predefined product and task settings.

Jiabao Zhao, Jonghan Lim, Hongliang Li +1

The Pennsylvania State University, University Park, PA, USA

Human-Robot Collaboration Robotic Assembly Generalization in Robotic Manipulation Large Language Model-Based Robot Planning

Ask PDF

Jun 18, 2026

Robust Assembly State Reasoning from Action Recognition for Human-Robot Collaboration

Human Action Recognition (HAR) is frequently investigated in Human-Robot Collaboration (HRC) research to understand what actions have been performed and hence the state of a collaborative task. Accurately tracking an assembly state from HAR is however not fully investigated, and in realistic scenarios is not a trivial task. This research systematically investigates and compares methods for tracking assembly state using action recognition inputs. Investigations using two diverse datasets and five state tracking approaches, including logic-based, Hidden Markov Model (HMM), and neural network (NN) methods, show that optimal approaches are not uniform across different tasks and that different methods fail under different circumstances. Testing is performed using both simulated inputs with varying noise levels and realistic inputs from a HAR model. Results show NN and HMM methods can perform well in tasks with limited variability, but for other scenarios logic-based approaches can be more robust. Methods which model expected action duration are also important for tasks with repeated actions where no additional sensing is provided.

James Fant-Male, Roel Pieters

Cognitive Robotics group, Unit of Automation Technology and Mechanical Engineering, Tampere University, 33720, Tampere, Finland

Human-Robot Collaboration Robotic Assembly

Ask PDF

Feb 16, 2026

Replanning Human-Robot Collaborative Tasks with Vision-Language Models via Semantic and Physical Dual-Correction

Human-robot collaborative assembly requires robots to interpret ambiguous corrective instructions while producing physically executable motions. Vision-language models (VLMs) provide semantic reasoning but may select logically inconsistent targets or misjudge execution outcomes. We propose a replanning framework that maps human instructions to Action Target candidates, including grasp poses and tool selections, and combines an Internal Correction Model for pre-execution logical verification with an External Correction Model for post-execution visual verification. The framework integrates VLM reasoning with 6-DoF grasp generation and collision-free trajectory planning. Simulation ablations show configuration-dependent effects: internal correction improves candidate validity, whereas external correction enables recovery for a low-latency VLM but can reduce success when visual verification produces false negatives. Experiments with an upper-body humanoid robot achieved 66.7% success in real-world object fixation, 100% in initial tool selection, and 75.0% in corrective tool selection. These results demonstrate interactive replanning across spatial and semantic collaborative tasks while identifying visual-state verification as a key limitation.

Taichi Kato, Takuya Kiyokawa, Namiko Saito +1

aDepartment of Systems Science, School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan · bDepartment of Systems Innovation, Graduate School of Engineering Science, The University of Osaka, 1-3 Machikaneyama, Toyonaka, Osaka, Japan · cMicrosoft Research Asia, Shinagawa, Tokyo, Japan +1

Language-Conditioned Robot Manipulation Large Language Model-Based Robot Planning Human-Robot Collaboration Robotic Assembly VLMs for Robotics

Ask PDF