Task-Specialized Language-Conditioned Models for Fine-Grained Robotic Manipulation

Chanho Kim; Soyun Choi

cs.ROOct 31, 2026

Task-Specialized Language-Conditioned Models for Fine-Grained Robotic Manipulation

Authors: Chanho Kim, Soyun Choi

Organizations: Department of Immersive Media Engineering Sungkyunkwan University South Korea

Abstract

Fine-grained robotic manipulation requires instruction grounding, robust perception, and precise contact control. We describe our submission to the MetaFine Open Competition at CoRL 2026. Five independently trained models use a common architecture with frozen DINOv2 and CLIP encoders, trainable language-conditioned feature modulation, and task-specific action heads. We improve demonstrations through contact-aware grasp filtering, instruction-matched data, and expert correction of insertion failures. At inference, task-dependent strategies include photometric test-time augmentation, stochastic action averaging, and uncertainty-triggered replanning. Our submission achieves an official overall score of 0.914 and ranks third among 15 teams. We analyze the models and their task-specific behavior.

Figures & tables

Figure 1: Overall architecture of our approach. Frozen DINOv2 and CLIP encoders provide language-conditioned visual features through trainable FiLM modules. A multimodal Transformer drives action heads that produce end-effector targets for T1–T4 or joint-space actions for T5.

Task	Head / action	Episodes	Training emphasis	Inference
T1	$\ell_{1}$ / EE	5,024	Contact audit, rollouts	TTA, residual
T2	FM / EE	1,200	Directional demos	FM mean, replan
T3	$\ell_{1}$ / EE	4,800	Paired target grounding	TTA
T4	FM / EE	4,644	Color-conditioned demos	FM mean
T5	FM / joint	6,723	DAgger-style correction	FM mean, replan

Table 1: Task-specific training and deployment settings. Episode counts include rendered camera views before applying training masks.

Task	$P$	$U$	$B$	Score
T1 Grasp part	0.735	0.775	0.775	0.762
T2 Move mug	0.967	1.000	1.000	0.989
T3 Toggle switch	0.988	1.000	1.000	0.996
T4 Place blocks	1.000	1.000	1.000	1.000
T5 Insert letter	0.816	0.825	0.825	0.822
Overall	–	–	–	0.914

Table 2: Official MetaFine evaluation results. Scores are rounded for display.

Panel of Figure 2 — Figure 2: Task-dependent effects of replanning. Episode success rates with and without uncertainty-triggered replanning for (a) T4 block placement and (b) T5 letter insertion.

Language-Conditioned Robot Manipulation

Language-conditioned robot manipulation connects natural-language instructions to robot perception, planning, and physical actions.

See all 242 papers

FineCraft: Enhancing VLA Models for Fine-Grained Manipulation
Nov 2, 2026Shuaike Zhang, Ziheng Quan, Junhao Ran +3Robotic Manipulation Language-Conditioned Robot Manipulation
Harbin Institute of Technology, Shenzhen · Xi’an Jiaotong University · Shenzhen Loop Area Institute HITSZ Robotics
Learning from an Imitable Teacher: Competition Solution for the MetaFine Challenge
Oct 31, 2026Yanze Zhang, Hao Liu, Haochang Hao +3Robotic Manipulation Learning from Demonstration
World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation
Aug 5, 2026Yuhao Pan, Haosong Peng, Zhengshen Zhang +8Language-Conditioned Robot Manipulation Contact-Rich Robotic Manipulation
1The Hong Kong University of Science and Technology · 2National University of Singapore · 3Nanyang Technological University +4

Nov 2, 2026J/K move · Enter open · S save

FineCraft: Enhancing VLA Models for Fine-Grained Manipulation

Shuaike Zhang, Ziheng Quan, Junhao Ran, Muhe Ding+2

Harbin Institute of Technology, Shenzhen · Xi’an Jiaotong University · Shenzhen Loop Area Institute HITSZ Robotics

Robotic Manipulation Language-Conditioned Robot Manipulation VLA Model Adaptation Closed-Loop Control of VLA Models Robot Manipulation Benchmarks

Open paper AskPDF arXiv

Fine-grained manipulation couples instruction grounding, local visual precision, and sustained execution. We present FineCraft, a framework that adapts pretrained models for fine-grained manipulation in the MetaFine Challenge. A task-routed mixture of policies selects independently adapted DINOv2-DiT or pretrained

π_{0.5}

policies through a common observation-action interface. Local visual features help identify the instructed object parts, demonstrations and expert continuations teach the required actions, and new observations guide replanning during execution. FineCraft ranks fourth with an overall score of

0.878

, exceeding the published reference on all five tasks. Official results and local diagnostics characterize performance across visual perturbations, instruction changes, and multi-stage manipulation. Code is available at https://github.com/eqcy/FineCraft

Abstract

Language-Conditioned Robot Manipulation

Explore similar work

FineCraft: Enhancing VLA Models for Fine-Grained Manipulation

Learning from an Imitable Teacher: Competition Solution for the MetaFine Challenge

World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation