Embodied Task Agents Can Perform Fine Manipulation through Tactile In-Context Learning

Yubang Wang; Zezheng Huai; Yifei Zhang; Jinhao Wu; Yitong Chen; Haozhe Zhang; Sixian Li; Jingjing Gong

cs.ROOct 20, 2026

Embodied Task Agents Can Perform Fine Manipulation through Tactile In-Context Learning

Authors: Yubang Wang, Zezheng Huai, Yifei Zhang, Jinhao Wu, Yitong Chen, Haozhe Zhang, Sixian Li, Jingjing Gong,

Organizations: Fudan University · Shanghai Innovation Institute · Tencent · Jilin University ·

Abstract

Can embodied task agents use tactile experience to perform fine manipulation without task-specific model training? We investigate this capability through Tactile-ETA, a system that incorporates tactile experience into an embodied task agent through in-context learning. Successful expert demonstrations show how contact changes during manipulation. We pair each recorded robot movement with its camera views and bilateral tactile observations. We keep tactile observations at the start and end of each movement and add two neighboring time points when the tactile images change markedly in between. The agent combines these demonstrations with current vision, tactile observations, and proprioception to select actions through general-purpose robot tools, without updating model parameters. Across four simulated manipulation tasks, adding tactile observations to the same two expert demonstrations increases average success from 15.75% to 24.00%, with current sensing and execution interfaces held fixed. Further experiments show that performance depends on the sensory content and representation of demonstrations, while comparing agent models on Lift Bottle reveals substantial differences under the same system configuration. On the real-robot tasks, demonstrations with aligned visual and tactile observations increase the task success rate from 0% to 60%. These findings demonstrate the feasibility of using tactile experience in context for fine manipulation and suggest a path toward extending embodied task agents to additional sensory inputs.

Figures & tables

Figure 1: Tactile-ETA performs fine manipulation by conditioning a frozen embodied task agent on (a) action-aligned expert demonstrations that preserve vision, motion, and tactile observations, together with (b) current observations and execution feedback—without parameter updates.

Figure 2: Tactile context construction. (Left) Keep tactile endpoints and, when the threshold is met, the peak-change pair. (Right) Combine tactile observations, vision, and recorded robot movements into ordered demonstrations for robot manipulation.

Task	No demos (vision)	No demos (vision + tactile)	Visual–motion demos	Tactile-ETA (ours)
Insert Tube	3	15	15	32
Lift Can	0	1	3	2
Lift Bottle	7	3	27	36
Pull Out Key	2	6	18	26
Average success (%)	3.00	6.25	15.75	24.00

Table 1: Success rates (%) on four UniVTAC tasks, with 100 trials per task and condition. Bold denotes the best result in each row.

Figure 3: Two Insert Tube rollouts with different demonstration contexts. Visual–motion demonstrations lead to continued advancement and failure; tactile-augmented demonstrations guide retreat and lateral correction, followed by success. Curves show measured TCP displacement along each rollout’s initial approach direction.

Context	Lift Bottle	Pull Out Key
Motion records	24	10
Visual–motion	27	18
Tactile–motion	26	26
Tactile-ETA (ours)	36	26

Table 2: Demonstration content ablation. Success rates (%); bold marks the best per task.

Representation	Success (%)
No demo tactile	27
Endpoints	27
All frames	2
Selected (ours)	36

Table 3: Tactile representation ablation on Lift Bottle. Success rates (%).

Figure 4: Intermediate tactile information missed by endpoint-only sampling. Columns show the start, a selected intermediate frame, and the end of an Insert Tube demonstration segment, with synchronized RGB observations above and right-fingertip tactile images below. Although the endpoint tactile patterns look similar, the intermediate frame exhibits a wider blue band in the highlighted region. Endpoint-only sampling omits this intermediate pattern, whereas Selected retains it in the demonstration context. Green boxes and arrows identify the same image region for comparison. One of the two selected intermediate frames is shown.

Figure 5: Agent model comparison on Lift Bottle: 100 trials per model with the same two demonstrations. GPT-6 results are reused from Table 1 .

Figure 6: Real-world tasks and results. (a) Recorded interactions illustrate wiping and chip pickup. (b) Task completions over five trials per configuration. Both receive current vision and tactile observations; Tactile-ETA additionally receives one multimodal human demonstration.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Setting	Current tactile	Demonstration records	Demonstration vision	Demonstration tactile
No demos (vision)	–	–	–	–
No demos (vision + tactile)	Yes	–	–	–
Motion records	Yes	Yes	–	–
Visual–motion	Yes	Yes	Yes	–
Tactile–motion	Yes	Yes	–	Yes
Tactile-ETA	Yes	Yes	Yes	Yes

Appendix

Table 4: Input configurations. Numerical demonstration records contain recorded robot movements, robot states, timing, and episode outcomes. All settings retain current camera observations, proprioception, and ordinary execution feedback. Motion records and Tactile–motion are additional content ablations evaluated on Lift Bottle and Pull Out Key.

Setting	Value
Physics / control frequency	120 / 60 Hz (decimation 2)
Maximum control steps per motion	80
Admitted non-preview motions	30 per episode
MCP calls	100 per episode
Total operator wall time	3600 s
Post-terminal observation/completion grace	Up to 300 s within total budget

Appendix

Table 5: Execution settings in the retained prompt and configuration records.

Parameter	Value
Tracking width	160 pixels
ROI $(x_{0},y_{0},x_{1},y_{1})$	$(0.08,0.08,0.92,0.92)$ of image size
Tracking grid	3 rows $\times$ 4 columns
Patch radius / patch size	6 / $13\times 13$ pixels
Search radius	4 pixels ( $9\times 9$ candidates)
Minimum patch standard deviation	8.0 grayscale levels

Appendix

Table 6: Tactile selection parameters. Spatial tracking settings use resized pixels; the motion threshold uses original-image pixels.

Case	Task	Setting
Target arrival without success	Lift Bottle	Tactile–motion
Uprighting (success)	Lift Bottle	Selected
Uprighting (failure)	Lift Bottle	Endpoints
Supplementary success	Lift Bottle	Endpoints
Supplementary failure	Lift Bottle	Selected
Contact adjustment	Insert Tube	Visual–motion

Appendix

Table 7: Cases included in the qualitative analyses.

Figure 7: Stage transitions in two Pull Out Key episodes. The motion-record agent reverses after three forward rotations; the tactile–motion agent performs a fourth before reversing and pulling. Images and message excerpts show the recorded sequence. Initial conditions differ; outcomes are native labels.

Task	1-shot	2-shot	4-shot
Insert Tube	22	32	23
Lift Can	2	2	3
Lift Bottle	31	36	2
Pull Out Key	35	26	48
Average	22.50	24.00	19.00

Appendix

Table 8: Success rate (%) with 1, 2, or 4 complete demonstrations. Each task–setting pair contains 100 evaluable query episodes. The shaded column denotes the 2-shot setting used in the main comparisons.

Tool	Arguments and purpose
review_demonstrations	No arguments; retrieve demonstration context.
observe	No arguments; obtain current observations.
move_to	Absolute xyz_m in world metres or relative delta_mm in world/current grip_site axes; approach_world and jaw_world specify TCP $+Z$ and $+X$ ; optional gripper, preview, and preview identifier.
mark_point	view , integer pixel u,v , optional point_id ; identify an image point.
report_issue	Required message ; report an issue.
finish_episode	Optional reason ; end voluntarily or acknowledge termination.

Appendix

Table 9: Robot interface. Auxiliary tools expose no online native task score.

Robotic Manipulation

Robotic manipulation studies how robots perceive, plan, and control physical interactions to grasp, move, assemble, and use objects.

See all 1,685 papers

Jul 2, 2026

Imagining the Sense of Touch: Touch-Informed Manipulation via Imagined Tactile Representations

Tactile sensing can substantially improve contact-rich robotic manipulation, yet its practical deployment remains limited by the fragility, calibration requirements, and maintenance burden of tactile hardware. This raises a fundamental question: can robots benefit from tactile knowledge without requiring tactile sensors at deployment? We present TacImag, a tactile imagination framework that predicts tactile observations from vision and proprioception and uses the generated signals to guide manipulation policies. Trained from paired visuotactile demonstrations, TacImag enables touch-informed manipulation using only visual observations at test time. We evaluate TacImag on six simulated and four real-world manipulation tasks. Across simulation and real-world experiments, imagined tactile observations consistently improve manipulation performance without requiring tactile hardware. In real-world experiments, imagined force fields improve contact-sensitive tasks by 44.4% on average, whereas imagined tactile images improve texture-sensitive tasks by 23.3%, revealing that the effectiveness of tactile imagination depends strongly on the relationship between tactile representation and task requirements. Our results further suggest that tactile imagination does not simply recover missing tactile measurements. Instead, it acts as a form of contact-aware supervision that transforms subtle visual interaction cues into representations that are easier for manipulation policies to exploit.

Zhiyuan Zhang, Adeesh Desai, Jyun-Chi Hu +7

Department of Industrial Engineering, Purdue University, West Lafayette, IN 47907, USA · Department of Mechanical Engineering, Texas A&M University, College Station, TX 77843, USA

Visual-to-Tactile Prediction Contact-Rich Robotic Manipulation Robotic Tactile Sensing

Ask PDF

Oct 20, 2026

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

General-purpose multimodal models have shown promise as direct robot action policies, yet whether they grasp the physics underlying real-world interactions and can apply that understanding to guide action remains underexplored. We investigate this question using contact-rich manipulation as a representative testbed, examining tactile understanding as a concrete instance of this broader capability. Contact-rich manipulation requires interpreting tactile feedback and adjusting actions after obstruction. Existing learned tactile policies acquire these capabilities through specialized pretraining and per-task optimization, with substantial data and training costs. We present GPT-as-Tactile-Policy, which summarizes expert demonstrations into task guidance and tactile references for contact assessment and corrective action. GPT-6-Astra combines this context with multimodal observations and interaction history to generate control commands and adjust actions through contact feedback. Without additional robotics-specific pretraining or per-task policy fine-tuning, our method achieves 93.8% mean success across five UniVTAC tasks. Overall performance is comparable to state-of-the-art policies, with higher success on selected tasks. On Insert Hole, tactile feedback improves success from 31% to 99%. Execution analysis further demonstrates contact-guided probing and correction, supporting manipulation progress through explicit expert experience and online feedback.

Ziyong Wang, Yupei Lin, Runxuan Huang +3

Sun Yat-sen University · X-Era AI Lab · The Hong Kong Polytechnic University

Tactile-Vision-Language-Action Models Robot Policy Learning Contact-Rich Robotic Manipulation Tactile Sensing for Robotic Manipulation

Ask PDF

May 21, 2026

TacO: Benchmarking Tactile Sensors for Object Manipulation

Vision-based learning from demonstrations has achieved remarkable success in enabling robots to perform manipulation tasks and high-level semantic reasoning, yet it remains insufficient for complex, contact-rich manipulation. While there is broad agreement that tactile sensing improves manipulation, there is no empirical guidance on which tactile sensors are best suited for which manipulation tasks. In this paper, we provide a systematic, task-driven evaluation of tactile sensors for robot manipulation and propose a framework for selecting and evaluating sensors based on manipulation policy performance. Separate manipulation policies are trained for tactile sensors of four distinct modalities: visual, acoustic, magnetic, and resistive, across three tasks: pick-and-place with unknown mass, object reorientation, and plug insertion. For each task, an analysis of how sensor properties such as spatial resolution, shear sensing, and tactile representation, and the inherent material friction affect task performances is done. Rather than tactile sensing being universally beneficial in the same way, our results show that the usefulness of tactile information depends strongly on sensor modality, material properties, and the specific manipulation tasks. All of the tactile sensors, code, data, and hardware setup will be publicly available on the project website.

Anya Zorin, Zilin Si, Myungsun Park +11

UC San Diego United States · CMU · UC San Diego +2

Tactile Sensing Contact-Rich Robotic Manipulation Robot Manipulation Benchmarks Tactile Sensing for Robotic Manipulation

Ask PDF

Abstract

Robotic Manipulation

Explore similar work

Imagining the Sense of Touch: Touch-Informed Manipulation via Imagined Tactile Representations

GPT-6-Astra Understands Touch and Outperform Vision--Tactile--Language--Action Models

TacO: Benchmarking Tactile Sensors for Object Manipulation