Predict the Swing, Match the Speed: Robotic Pipe Stabbing With Learned Stereo Vision and Residual Reinforcement Learning

Mohamed Ashraf Mostafa; Taher El-Fakharany; Samir Khaled

cs.ROOct 27, 2026

Predict the Swing, Match the Speed: Robotic Pipe Stabbing With Learned Stereo Vision and Residual Reinforcement Learning

Authors: Mohamed Ashraf Mostafa, Taher El-Fakharany, Samir Khaled

Organizations: Mining and Petroleum Engineering Department, Faculty of Engineering, Al-Azhar University, Cairo, Egypt · Petroleum Department, Faculty of Engineering and Technology, Future University in Egypt (FUE), Cairo, Egypt

Abstract

A joint of pipe run into the hole swings from the elevator over the well and strikes a fixed stabbing guide at whatever speed the swing has. StabBot, a robotic stabbing system, predicts the swing and meets the pipe at its own speed. A pendulum Kalman filter tuned from the pipe tally tracks the joint in stereo images. A six-axis arm with an open V-shaped roller head closes in at matched velocity, confirms the last millimetres by touch, aims the pin over the box when the elevator hangs off centre, and follows the block down with lateral compliance. A ResNet-18 keypoint network locates the pipe, pin nose and box mouth in 3-D, and a residual reinforcement-learning policy refines the capture. On 200 paired random jobs with nine sizes from 2-3/8-in. tubing to 20-in. casing, StabBot seated every joint with the pipe touching only rollers, against 74% for a conventional fixed guide. Median contact speed fell from 18.4 to 2.3 cm/s and median peak head-pipe force from 11.3 to 1.4 kN, and the force was lower in all 200 jobs (Wilcoxon $p < 10^{-30}$ ). The network locates the pipe with 0.4-mm median error in 2.3 ms per stereo pair. With the network in the loop, success stayed at 100% under 20 mm of added noise, 300-ms delay, 85% frame loss, 20-mm calibration offset, 20-cm elevator offset and 40-cm swings. All results come from a 1:4 Froude-scaled MuJoCo model and are reported at full scale.

Figures & tables

Panel of Figure 1 — Figure 1: StabBot on a land-rig drill floor Left: the arm stands beside the V-door lane, the joint hangs from the elevator over the well and the box is held in the slips. Right: the open V head has met the swinging 5-in. drillpipe at its own speed and closed its two rotary clamps; the pipe rests on free rollers only. Every frame is a replay of the recorded physics, rendered at full scale.

Work	M	S	V	T	L
Floorhand + tagline	✓	✗	$\circ$	✓	✗
Passive funnel [ 10 ]	✗	✗	✗	✗	✗
Robot + stabbing guide [ 2 , 3 , 5 ]	✗	✗	✗	$\circ$	✗
Remote stabber [ 6 ]	✗	✗	✗	✗	✗
Pivot-pointing tailing [ 12 ]	✓	$\circ$	✗	✗	✗
Stick-up vision [ 13 , 14 ]	✗	✗	✗	✗	$\circ$

Table 1: StabBot against the closest published work

Quantity	Scale factor	Model	Full scale
Length (joint)	1/4	2.4 m	9.6 m
Time (swing period)	1/2	2.54 s	5.07 s
Speed	1/2	model = full scale / 2
Mass (heaviest joint)	1/64	21 kg	1,343 kg
Force	1/64	model = full scale / 64
Camera rate	2	60 Hz	30 Hz

Table 2: Froude scaling between the model and full scale

Item	Setting
Keypoint network	ResNet-18 (ImageNet), three heatmaps of 64 $\times$ 64
Heatmap target	Gaussian, $\sigma$ = 1.5 px
Network training	AdamW, learning rate $10^{-3}$ , weight decay $10^{-4}$ , one-cycle schedule (10% warm-up), 30 epochs, batch 64
Data	20,000 training / 2,000 validation / 2,000 test stereo pairs
Filter measurement noise	6 mm (not retuned for the network)
Residual policy	MLP 64 $\times$ 64, PPO (Stable-Baselines3)

Table 3: Main parameters of the keypoint network, the filter and the residual policy

Figure 2: Overview of StabBot The vision network gives 3-D landmarks, from which the pendulum filter predicts the pipe. The model-based controller captures, centres and stabs, and the residual policy adds a small, smoothed correction during capture. The rig inputs are ones that drill floor robots already have, namely the pipe tally, the drawworks height, cameras and a wrist force sensor.

Panel of Figure 3 — Figure 3: The guide head designed for this work (a) Rendered design with the clamps closed on 5-in. drillpipe. (b) Exploded view: flange adapter and 6-axis F/T sensor, mount block, V plates, six face rollers on standoffs, outrigger brackets, two 270 ∘ torque-limited rotary actuators and clamp arms ending in rollers of the same part. (c) General arrangement. (d) Section through the rollers with every size seated, from 2 $\frac{3}{8}$ -in. tubing to 20-in. casing: each touches rollers only, and the clamps close on every diameter. The full set of four A3 sheets is in the repository.

Panel of Figure 4 — Figure 4: One job with StabBot 5-in. drillpipe, 20-cm swing at the pin, seen from the side (top) and from beyond the pipe (bottom). (a) The robot watches the swing; (b) the V meets the pipe at its speed; (c) the clamps close; (d) the block slacks off and the pin enters the box; (e) the clamps open and the head backs off. The frames replay the recorded physics in a plain studio setting.

Method	Seated, no hits	Contact speed, cm/s	Peak force, kN	Time to
	(95% CI)	(med / p95)	(med / p95)	seat, s
Static guide ∗	74.0% (67.5–79.6)	18.4 / 53.4	11.3 / 24.7	12.3
Reactive	93.0% (88.6–95.8)	5.3 / 30.5	4.0 / 13.4	13.6
Reactive+guarded	97.5% (94.3–98.9)	5.4 / 30.1	3.8 / 14.1	13.6
Swing-aware	100% (98.1–100)	3.2 / 11.8	1.9 / 6.3	14.6
StabBot	100% (98.1–100)	2.3 / 5.5	1.4 / 2.5	14.1

Table 4: Main benchmark: 200 paired jobs, keypoint network in the loop

Figure 5: Benchmark over 200 paired jobs (a) Jobs seated with the pipe touching only rollers, Wilson 95% intervals. (b, c) Contact speed and peak head–pipe force (box: interquartile range; whiskers: 5th–95th percentile; log scale; median printed). (d) Peak force per job, fixed guide against StabBot: every point lies below the diagonal, including the 52 jobs the fixed guide failed (

\times

Group	Jobs	Seated, no hits	Contact speed, cm/s	Peak force, kN
		(fixed / StabBot)	(median, fixed / StabBot)	(median, fixed / StabBot)
2 $\frac{3}{8}$ -in. tubing	25	15 / 25	20.0 / 1.4	1.8 / 0.69
3 $\frac{1}{2}$ -in. drillpipe	25	19 / 25	17.4 / 0.8	6.9 / 1.34
5-in. drillpipe	23	19 / 23	21.7 / 1.8	8.5 / 0.97
6 $\frac{5}{8}$ -in. drillpipe	23	21 / 23	24.0 / 2.8	12.0 / 1.33
7-in. casing	23	17 / 23	17.6 / 3.0	10.3 / 1.41

Table 5: Benchmark by pipe size and by swing amplitude, fixed guide against StabBot

Figure 6: One recorded job, fixed guide against StabBot 5-in. drillpipe, 20-cm swing at the pin, keypoint network in the loop. Top: position along the swing direction of the pipe axis at head height and of the V seat. Middle: pipe–seat relative speed (symmetric log scale). Bottom: net head–pipe force sampled at 25 frames/s and the running peak over capture and centring (dashed), which also records impacts shorter than a frame. Shading marks the job phases. The fixed guide waits at well centre and is struck at 47.5 cm/s; StabBot follows the pipe, reaches it at 1.7 cm/s and holds it while it centres.

Step	Contact speed	Peak force	Success
+ guarded approach	$+7\%$ ( $p$ = 0.02)	$-4\%$ ( $p$ = 0.02)	93.0 $\to$ 97.5%
+ prediction and matched speed	$\mathbf{-41\%}$ ( $p$ = $1{\times}10^{-7}$ )	$\mathbf{-51\%}$ ( $p$ = $3{\times}10^{-16}$ )	97.5 $\to$ 100%
+ residual RL	$\mathbf{-27\%}$ ( $p$ = $3{\times}10^{-8}$ )	$\mathbf{-22\%}$ ( $p$ = $4{\times}10^{-19}$ )	100 $\to$ 100%
Static guide $\to$ StabBot	$\mathbf{-87\%}$ ( $p$ = $9{\times}10^{-31}$ )	$\mathbf{-87\%}$ ( $p$ = $1{\times}10^{-34}$ )	74 $\to$ 100%

Table 6: Paired effect of each component (200 jobs)

Figure 7: Robustness of the full system with the keypoint network in the loop First 100 jobs, one error at a time with the controllers unchanged. The two swing-aware methods, whose curves overlap in most panels, seat 100% of joints through every error except 30 mm of added noise; the last panel shows that noise raises their force while reactive tracking misses. The fixed guide (dashed) does not track the pipe. Swing levels are ranges at the pin.

Figure 8: Elevator 20 cm off centre, same job Without tilt compensation the pin lands on the box face 1.5 cm off (left); with it the joint seats 0.1 cm off (right).

Landmark	Median	p95	Within 6 mm
Pipe axis at head height	0.4	1.1	–
Pin nose	0.7	1.6	–
Box mouth	0.5	1.5	–
Pin-to-box offset	0.7	1.8	99.9%

Table 7: Keypoint network on 2,000 held-out scenes (3-D error, mm)

Panel of Figure 9 — Figure 9: Keypoint vision on held-out and unseen scenes Top left: predicted (green) and true (red) keypoints on held-out stereo pairs across sizes and robot poses. Bottom left: the same scene under unseen conditions (dark, bright, low sun, yellow and white pipe, red floor, motion blur, sensor noise, JPEG, haze, glare, occluding bar). Right: pipe error for v1 and the final v2 against the 95th-percentile error of the simulated camera used to design the controllers.

Quantity	Fixed guide	StabBot	Effect
Capture missed	10.5%	0%	no re-stab
Pipe struck steel, then seated	15.5%	0%	pipe on rollers only
Peak head–pipe force, median / p95	11.3 / 24.7 kN	1.4 / 2.5 kN	7.8 $\times$ / 9.9 $\times$ lower
Contact speed, median	18.4 cm/s	2.3 cm/s	7.8 $\times$ lower
Impact energy at first contact	–	–	95 $\times$ lower (median per job)
Wrist force, median	8.1 kN	2.8 kN	66% lower

Table 8: Measured effect per connection, StabBot against the fixed guide

Contribution	What is new	Evidence
Swing-aware capture	Online pendulum estimate from the pipe tally and the camera; capture at the pipe’s own speed, finished by touch	100% seated vs 74%; contact speed and force 7.8 $\times$ lower
Tilt compensation	Pin aimed over the box from the estimated elevator offset	43% $\to$ 100% seated at a 20-cm offset
One head for nine sizes	Open V on free rollers with 20-mm clearance on every other part	2 $\frac{3}{8}$ -in. tubing to 20-in. casing, all seated
3-D keypoint vision	Pipe, pin and box located from stereo images	0.4-mm median error, 2.3 ms per pair
Residual learning	Learned correction on top of the model-based capture	a further 22% lower peak force
Paired benchmark	Every method on the same 200 jobs, with robustness and ablation	100% under delay, frame loss, calibration and elevator errors

Table 9: Contributions of this work and the evidence for each

$c$	Rest position about which the joint swings	m
$F_{k}$ , $F_{\max}$	Peak head–pipe force in a decision step, and over the episode	kN
$k$	Filter step or decision index	–
$\mathbf{n}$	Unit vector along the approach direction	–
$o$	Offset of the elevator from well centre	m
$p$ , $v$	Position and velocity of the pipe axis at head height	m, m/s
$\hat{\mathbf{p}}$	Predicted pipe position	m

Table 19

Dynamic Robotic Manipulation

Dynamic robotic manipulation studies how robots act on moving or deformable objects and respond to fast-changing physical interactions.

See all 94 papers

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception
May 11, 2026Evans Han, Yunfan Jiang, Yingke Wang +6Robot Policy Learning Visuomotor Policy Learning
1Stanford University · 2Northwestern University · 3Lambda, Inc
InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality
Sep 21, 2026Zhenghua Ma, Xinpan Meng, Zeyu Liu +4RL for Robotics Contact-Rich Robotic Manipulation
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China, and the State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China · PaXini AI Technology (Beijing) Co., Ltd., Beijing 100080, China
Facet-0: A Robotic Foundation Model for Contact-Rich Precise Manipulation
Sep 1, 2026Haoyuan Deng, Haichao Liu, Wenkai Guo +6Robot Skill Learning Contact-Rich Robotic Manipulation
1PINE Lab, Nanyang Technological University, Singapore

May 11, 2026J/K move · Enter open · S save

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

Evans Han, Yunfan Jiang, Yingke Wang, Haoyue Xiao+5

1Stanford University · 2Northwestern University · 3Lambda, Inc

Robot Policy Learning Visuomotor Policy Learning Robotic Manipulation Stereo Vision

Open paper AskPDF arXiv

Recent advances in robot imitation learning have produced powerful visuomotor policies that manipulate diverse objects from visual inputs. However, monocular observations lack depth information, which is critical for precise manipulation in cluttered or geometrically complex scenes. Explicit depth maps and point clouds are often noisy and fragile in real-world manipulation. We introduce StereoPolicy, a visuomotor policy learning framework that directly leverages synchronized stereo image pairs to improve geometric reasoning without constructing explicit 3D representations. StereoPolicy processes each image with pretrained 2D vision encoders and fuses left-right features through a cross-attention-based Stereo Transformer, capturing spatial correspondence and disparity cues implicitly. The framework integrates with diffusion-based and pretrained vision-language-action (VLA) policies, delivering consistent improvements over RGB, RGB-D, point cloud, and multi-view baselines across three simulation benchmarks and seven real-robot tabletop and bimanual mobile manipulation tasks. Our results show that stereo vision bridges 2D pretrained representations and 3D geometric understanding for robotic manipulation.

Abstract

Dynamic Robotic Manipulation

Explore similar work

StereoPolicy: Improving Robotic Manipulation Policies via Stereo Perception

InsertAnything: Generalizable Contact-Rich Precision Insertion from Simulation to Reality

Facet-0: A Robotic Foundation Model for Contact-Rich Precise Manipulation