From Language to Motion: Task-Conditioned Focal-Stack Trajectory Integration for Microscopic Robots

Junjie Xie; Chuxuan He; Junkai Huang; Heng Zhang; Angen Ye; Yujia Song; Yuqing Li; Pengsong Zhang

cs.ROOct 8, 2026

From Language to Motion: Task-Conditioned Focal-Stack Trajectory Integration for Microscopic Robots

Authors: Junjie Xie, Chuxuan He, Junkai Huang, Heng Zhang, Angen Ye, Yujia Song, Yuqing Li, Pengsong Zhang, +1 more

Organizations: Institute of Automation, Chinese Academy of Sciences · Zhejiang Gongshang University · Safe AI Lab, Carnegie Mellon University · Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Ontario, M5S 3G8, Canada

Abstract

Microscopic robots require accurate task geometry despite changes in language, parts, and focus. We present a semantic-to-physical framework that maps instructions to constrained geometric operators, reuses frozen open-vocabulary perception, and integrates locally reliable focal-plane trajectories by confidence weighting and dynamic programming. Calibrated multi-view geometry connects 2-D paths to physical execution. Prompt, unseen-part, and geometry reconfiguration tests yield 6.30-6.59-pixel RMSE. Relative to part-specific U-Net training with 20-100 labels, the proposed zero-new-label configuration takes 15 rather than 72-165 min. Across nine part-illumination conditions, trajectory-space integration reduces RMSE from 14.41 to 6.28 pixels (56.4%) and P95 error from 20.07 to 8.13 pixels (59.5%) compared with image-first multi-focus fusion. An ablation isolates the roles of confidence and path-wise selection. In representative robot experiments, target-region coverage improves from 83.5% to 92.9%. Dispensing provides a measurable physical trace, not a task-specific limitation of the method.

Figures & tables

Fig. 1: System overview. (A) CAD-rendered examples illustrate task-level geometric reconfiguration through language prompts. (B) Trajectory candidates and local evidence are extracted at individual focal planes before integration in path space. (C) Calibrated multi-view geometry bridges the recovered 2-D paths to physical execution on the microscopic robot.

Fig. 2: Semantic-to-geometry reconfiguration. (a) Traditional part-specific retraining versus a prompt-driven workflow. (b) Natural-language instructions are converted into constrained task specifications. (c) Reusable frozen detection/segmentation models and lightweight geometry operators recover 2-D task geometry. (d) The same visual backbone serves multiple geometry requests without task-specific detector retraining.

Fig. 3: Confidence-weighted trajectory-level multi-focal integration. (a) Local trajectories and reliability are estimated separately for each focal plane. (b) Image-first and trajectory-first processing paths. (c) Confidence-weighted hypotheses and dynamic-programming selection in ordered path space. (d) The final 2-D trajectory preserves focal-source information.

Fig. 4: Real-system physical validation. (a) Microscopic manipulation platform and auxiliary setup. (b) Complementary microscope views. (c) Robot–part interaction during execution. (d) Deposited physical traces with local image-software dimensional annotations. Dispensing is used as a trace-preserving measurement task; target-relative performance is summarized separately in Table V .

Condition	RMSE (px) $\downarrow$	P95 (px) $\downarrow$
Canonical wording	$6.30\pm 0.03$	$8.08\pm 0.03$
Formal paraphrase	$6.34\pm 0.03$	$8.12\pm 0.04$
Semantic paraphrase	$6.43\pm 0.03$	$8.25\pm 0.03$
Colloquial wording	$6.57\pm 0.03$	$8.37\pm 0.06$
Unseen part D	$6.39\pm 0.03$	$8.19\pm 0.03$
Unseen part E	$6.50\pm 0.03$	$8.26\pm 0.04$

TABLE I: Language and task reconfiguration.

Method	Labels	RMSE	P95	Adapt. time
U-Net (100)	100	$5.47$	$7.15$	$165$ min
U-Net (50)	50	$5.78$	$7.52$	$119$ min
U-Net (20)	20	$6.46$	$8.26$	$72$ min
Ours	0	$6.28$	$8.13$	$15$ min

TABLE II: Task-specific supervision versus reconfiguration cost.

Method	RMSE	P95	Mean	Max
Image-first MFIF	$14.41\pm 0.04$	$20.07\pm 0.09$	$13.50\pm 0.07$	$20.09\pm 0.06$
Ours	$\mathbf{6.28\pm 0.04}$	$\mathbf{8.13\pm 0.05}$	$\mathbf{5.93\pm 0.05}$	$\mathbf{9.25\pm 0.05}$
Reduction	56.4%	59.5%	56.1%	54.0%

TABLE III: Measured trajectory error across nine part–illumination conditions.

Variant	RMSE	P95	Source switches
Pointwise max	$7.00$	$8.81$	$60.6$
Confidence only	$6.75$	$8.53$	$48.8$
DP only	$6.56$	$8.33$	$11.2$
Confidence + DP	$\mathbf{6.28}$	$\mathbf{8.13}$	$9.0$

TABLE IV: Confidence/DP ablation.

Upstream trajectory	Rel. trace error	Coverage	Out-of-region
Image-first MFIF	$1.000$	$83.5\pm 1.1\%$	$12.5\pm 0.5\%$
Ours	$0.519$	$92.9\pm 1.1\%$	$5.4\pm 0.5\%$

TABLE V: Representative physical validation.

Explore similar work

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation
Jun 22, 2026Ali Alabbas, Dipshikha Das, Camillo Murgia +3Zero-Shot Robotic Manipulation Language-Conditioned Robot Manipulation
Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland
MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning
Sep 21, 2026Mingke Lu, Anxing Xiao, David HsuMultimodal Grounding Robot Task Planning
School of Computing and Smart Systems Institute, National University of Singapore, Singapore · Department of Electrical and Computer Engineering, University of California, Los Angeles, Los Angeles, CA, USA
Robostral Navigate
Jul 22, 2026Abdelaziz Bounhar, Abhijeet Somani, Aditi Kabra +273Robot Policy Learning Robot Navigation

Jun 22, 2026, cs.ROJ/K move · Enter open · S save

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

Ali Alabbas, Dipshikha Das, Camillo Murgia, Sainul Ansary+2

Department of Electronic, Software and Advanced Manufacturing Engineering, Atlantic Technological University, Galway, Ireland · Atlantic Technological University, Galway, Ireland

Zero-Shot Robotic Manipulation Language-Conditioned Robot Manipulation Large Language Model-Based Robot Planning Robotic Manipulation VLMs for Robotics

Open paper AskPDF arXiv

This paper presents a modular training-free framework for zero-shot, language-guided robotic manipulation in semi-structured environments. The architecture bridges the gap between high-level reasoning and low-level kinematics by decomposing the vision-action pipeline into three stages: visual perception, semantic interpretation, and task execution. To overcome the spatial ambiguity and semantic hallucinations inherent in standard Vision-Language Models (VLMs), the perception module employs FastSAM and Set-of-Mark (SoM) prompting to dynamically generate grounded, alphanumeric visual anchors. The same foundation model then operates purely as a Large Language Model (LLM) to act as a semantic router, translating unconstrained human directives into verifiable, reconfigurable configurations. Finally, these configurations are dynamically parsed by a Task Orchestrator into MoveIt Task Constructor (MTC) to generate collision-free trajectories. The framework is evaluated across two zero-shot experimental setups: unconstrained open-world sequential manipulation and dense relational spatial reasoning, achieving a 62% end-to-end task success rate across both scenarios, demonstrating its capacity to reliably execute complex physical actions without domain-specific training or manual coordinate programming.

Abstract

Explore similar work

Bridging Semantics and Kinematics: A Modular Framework for Zero-Shot Robotic Manipulation

MIGU: Multimodal Instruction Grounding under Uncertainty for Manipulation Planning

Robostral Navigate