Time-Indexed Skill Success Models for Latency-Aware Skill Selection and Timing in Dynamic Manipulation
Organizations: Virtual Interaction Research Lab (VIRaL), School of Media, Technological University Dublin, Dublin, Ireland. · School of Food Science and Environmental Health, Technological University Dublin, Dublin, Ireland.
Abstract
Robots that manipulate moving objects act on observations that are outdated when the chosen skill starts, and the best skill can change within a few hundred milliseconds. Skill competence models, however, predict success for a launch at the moment of observation. We propose time-indexed skill success models, which predict whole-task success when a skill is launched after a given delay. Trained on rollouts branched at a grid of launch delays, they let a receding-horizon planner select both the skill and its launch time under fixed or random latency. In a MuJoCo study in which a UR5e captures a sliding puck with three skills, the planner exceeded a static competence model by 2.8 to 6.2 percentage points at 100 to 300 ms of latency, including when that model was evaluated on point or sampled forecasts of the launch state. Across five complete runs (three training seeds and two distribution shifts), all 80 paired 95% intervals against four baselines excluded zero. On delayed launches, the time-indexed model was calibrated (expected calibration error 0.009 against 0.239), and pausing the world during latency overstated success by up to 23 points. On a real UR5e that captures a rolling ball after 0.2 s of injected latency, a static model trained in a hardware-matched simulator predicted 99% success on a prespecified subset of 35 delayed launches that it rated at least 0.8. Of these, 63% succeeded (95% interval 46 to 77%), and the time-indexed model predicted 56%. In this single-skill, predictable regime, a matched physics-based forecast was also well calibrated.
Figures & tables
| Fixed [ms] | Random | ||||||
| Planner | 0 | 100 | 150 | 200 | 300 | Heavy | Unif. |
| B0 feasibility rule | 63.6 | 55.9 | 48.1 | 46.5 | 41.6 | 50.1 | 48.5 |
| B1 static | 63.6 | 56.9 | 50.3 | 47.5 | 43.3 | 51.7 | 50.1 |
| B1r myopic wait | 61.8 | 54.9 | 47.3 | 45.4 | 40.5 | n/a | n/a |
| B1p point forecast | 63.7 | 55.8 | 49.3 | 46.6 | 43.9 | 50.0 | 48.5 |
| B1s sampled forecasts | 65.0 | 57.0 | 51.9 | 49.1 | 42.8 | 52.1 | 51.1 |
| ECE | Confident | ||
|---|---|---|---|
| Predictor | InterceptGrasp | Block | subset (%) |
| Time-indexed | 0.009 [0.006, 0.016] | 0.006 [0.005, 0.013] | 72.0 |
| Static | 0.239 [0.226, 0.254] | 0.234 [0.221, 0.249] | 98.8 |
| Static + temperature | 0.119 [0.108, 0.131] | 0.133 [0.119, 0.148] | 66.8 |
| Static, point forecast | 0.126 [0.118, 0.135] | 0.092 [0.083, 0.101] | 71.5 |
| Static, sampled forecasts | 0.059 [0.051, 0.066] | 0.053 [0.045, 0.061] | 70.0 |
| Group | Parameter | Value |
|---|---|---|
| Scene | channel | inner width 0.48 m ( 0.26 to 0.74 m in the robot frame), launch line m, open exit edge m, rails 3 cm high |
| puck | cylinder, diameter 40 mm, height 24 mm, mass 50 g | |
| bin | centre m, cm | |
| gripper | parallel jaw, tool-centre offset 160 mm, stroke 50 mm, V-groove pads, 0.10 m/s per finger, 60 N, 20 ms actuation delay | |
| physics | MuJoCo / MuJoCo Warp, time step 2 ms | |
| Robot | arm | UR5e (MuJoCo Menagerie), joint speed /s, tool speed m/s, tool acceleration m/s 2 |
| Model | Mean | Median | 95th pct. | 99th pct. | Max. | Quantiles used by the -dist planners |
|---|---|---|---|---|---|---|
| Uniform, 50 to 300 ms | 175 | 175 | 287 | 297 | 300 | 66, 97, 128, 159, 191, 222, 253, 284 |
| Heavy-tailed, , cap 500 ms | 145 | 130 | 262 | 364 | 500 | 79, 95, 109, 123, 138, 158, 187, 249 |
| [s] | WaitPick only | InterceptGrasp only | Block only | Several | None |
|---|---|---|---|---|---|
| 0.00 | 7.3 | 11.0 | 11.1 | 36.2 | 34.4 |
| 0.10 | 7.8 | 9.9 | 8.8 | 33.9 | 39.7 |
| 0.20 | 7.9 | 7.8 | 6.8 | 33.5 | 44.0 |
| 0.30 | 6.9 | 6.7 | 5.2 | 32.9 | 48.4 |
| 0.45 | 7.0 | 6.0 | 2.1 | 31.4 | 53.5 |
| 0.60 | 6.2 | 4.1 | 0.5 | 30.9 | 58.2 |
| Injected latency [ms] | |||||
|---|---|---|---|---|---|
| Comparison | 0 | 100 | 150 | 200 | 300 |
| TI-flat B0 feasibility rule | 2.2 [1.1, 3.3] | 3.9 [2.7, 5.0] | 7.2 [5.8, 8.6] | 6.4 [5.1, 7.8] | 6.8 [5.3, 8.2] |
| TI-flat B1 static model | 2.2 [1.1, 3.2] | 2.9 [1.8, 3.9] | 5.1 [3.7, 6.2] | 5.3 [4.0, 6.7] | 5.1 [3.8, 6.5] |
| TI-flat B1p static, point forecast | 2.1 [0.9, 3.2] | 4.0 [2.6, 5.3] | 6.0 [4.7, 7.4] | 6.2 [4.9, 7.8] | 4.5 [3.2, 5.9] |
| TI-flat B1s static, sampled forecasts | 0.9 [ 0.1, 1.9] | 2.8 [1.6, 4.0] | 3.4 [2.2, 4.6] | 3.8 [2.5, 5.1] | 5.7 [4.3, 6.9] |
| TI-flat TI-fac | 0.2 [ 0.8, 1.3] | 1.6 [0.4, 2.5] | 0.2 [ 0.9, 1.2] | 0.6 [ 0.6, 1.8] | 1.2 [0.2, 2.2] |
| Planner | Latency | Heavy-tailed | Uniform |
|---|---|---|---|
| B0 feasibility rule | – | 50.1 | 48.5 |
| B1 static | – | 51.7 | 50.1 |
| B1p point forecast | realised | 50.2 | 49.3 |
| B1p point forecast | distribution | 50.0 | 48.5 |
| B1s sampled forecasts | realised | 52.2 | 51.1 |
| B1s sampled forecasts | distribution | 52.1 | 51.1 |
| Comparison | Condition | Positive | Range [pts] |
|---|---|---|---|
| TI-flat B1s | fixed, 100 to 300 ms | 20/20 | 2.3 to 6.3 |
| TI-flat B1s | fixed, 0 ms | 3/5 | 0.9 to 1.5 |
| TI-flat B1p | fixed, 100 to 300 ms | 20/20 | 3.0 to 7.1 |
| TI-flat B1p | fixed, 0 ms | 5/5 | 1.9 to 3.9 |
| TI-flat B1 | fixed, 100 to 300 ms | 20/20 | 2.1 to 6.9 |
| TI-flat B1 | fixed, 0 ms | 5/5 | 1.8 to 2.7 |
| Predictor | InterceptGrasp, F | InterceptGrasp, G | Block, F | Block, G |
|---|---|---|---|---|
| Time-indexed | 0.009 [0.006, 0.016] | 0.008 [0.006, 0.014] | 0.006 [0.005, 0.013] | 0.006 [0.005, 0.012] |
| Static | 0.239 [0.226, 0.254] | 0.197 [0.184, 0.211] | 0.234 [0.221, 0.249] | 0.116 [0.105, 0.130] |
| Static + temperature | 0.119 [0.108, 0.131] | 0.142 [0.130, 0.152] | 0.133 [0.119, 0.148] | 0.084 [0.076, 0.097] |
| Static, point forecast | 0.126 [0.118, 0.135] | 0.082 [0.074, 0.089] | 0.092 [0.083, 0.101] | 0.057 [0.051, 0.065] |
| Static, sampled forecasts | 0.059 [0.051, 0.066] | 0.051 [0.044, 0.059] | 0.053 [0.045, 0.061] | 0.028 [0.023, 0.035] |
| Skill, target | Predictor | ECE | ECE (mass) | Brier | Reliability | Resolution | NLL | Over-conf. |
|---|---|---|---|---|---|---|---|---|
| InterceptGrasp, F | Time-indexed | 0.009 | 0.009 | 0.075 | 0.000 | 0.125 | 0.235 | 0.7 |
| InterceptGrasp, F | Static | 0.239 | 0.234 | 0.241 | 0.089 | 0.047 | 2.194 | 14.4 |
| InterceptGrasp, F | Static + temperature | 0.119 | 0.120 | 0.167 | 0.019 | 0.053 | 0.494 | 11.9 |
| InterceptGrasp, F | Static, point forecast | 0.126 | 0.119 | 0.135 | 0.019 | 0.084 | 1.069 | 3.3 |
| InterceptGrasp, F | Static, sampled forecasts | 0.059 | 0.057 | 0.113 | 0.004 | 0.091 | 0.514 | 3.6 |
| InterceptGrasp, G | Time-indexed | 0.008 | 0.008 | 0.063 | 0.000 | 0.098 | 0.196 | 0.5 |
| All stale, [ms] | Avoidable stale, [ms] | |||||||
|---|---|---|---|---|---|---|---|---|
| Planner | 100 | 150 | 200 | 300 | 100 | 150 | 200 | 300 |
| B0 feasibility rule | 5.4 | 12.2 | 12.0 | 14.2 | 3.0 | 7.3 | 7.3 | 7.9 |
| B1 static model | 9.5 | 15.3 | 15.2 | 17.2 | 1.7 | 5.9 | 6.2 | 6.3 |
| B1p static, point forecast | 7.0 | 10.0 | 9.2 | 6.7 | 1.6 | 3.9 | 4.0 | 2.7 |
| B1s static, sampled forecasts | 8.1 | 10.8 | 9.7 | 7.5 | 1.1 | 2.5 | 2.1 | 1.7 |
| TI-fac (ours) | 8.3 | 10.2 | 10.0 | 8.9 | 0.3 | 1.5 | 1.7 | 0.9 |
| [ms] | ||||
|---|---|---|---|---|
| Planner | 100 | 150 | 200 | 300 |
| Always WaitPick | 0.2 | 0.2 | 0.2 | 0.1 |
| Always InterceptGrasp | 3.7 | 7.2 | 8.7 | 11.1 |
| B0 feasibility rule | 8.1 | 15.9 | 17.5 | 22.4 |
| B1 static model | 7.4 | 13.9 | 16.8 | 20.9 |
| B1p static, point forecast | 8.4 | 14.8 | 17.5 | 20.2 |
| vs B1, [ms] | vs B1s, [ms] | ||||||
|---|---|---|---|---|---|---|---|
| Band | 150 | 200 | 300 | 150 | 200 | 300 | |
| slow, 0.30 to 0.50 | 579 | 7.9 | 9.0 | 6.9 | 2.2 | 4.3 | 3.1 |
| medium, 0.50 to 0.75 | 726 | 5.0 | 5.1 | 6.2 | 3.6 | 4.3 | 8.0 |
| fast, 0.75 to 1.00 | 695 | 2.7 | 2.6 | 2.4 | 4.2 | 2.7 | 5.3 |
| Predictor | Conf. delayed | Brier | Brier [95%] | AUC |
|---|---|---|---|---|
| Static | 0.99 | 0.260 | 0.230 [0.110, 0.360] | 0.84 |
| , forecast state | 0.60 | 0.020 | 0.011 [ 0.021, 0.003] | 1.00 |
| at launch † | 0.61 | 0.006 | 0.024 [ 0.056, 0.005] | 1.00 |
| Time-indexed | 0.56 | 0.031 | 1.00 | |
| Observed | 0.63 [0.46, 0.77] |
| Group | Item | Value |
|---|---|---|
| Hardware | robot | UR5e, Cartesian servo streaming at 125 Hz (look-ahead 0.05 s, gain 1000) |
| tool | Robotiq Hand-E, fingers fully open (53 mm), bag behind the fingers; lateral clearance 5.2 mm | |
| task | golf ball (42.7 mm) rolling on a 58.4 cm by 48.7 cm board inclined by about , released by hand from rest | |
| perception | RealSense D455f, 90 frames/s, colour detection, robot-calibrated board frame (about 1 mm) | |
| estimation | extended Kalman filter on a fitted acceleration field (25 parameters, fit RMS 2.5 mm on 72 rolls) | |
| skill | lateral move along the catch line to the predicted crossing, time-optimal with m/s and m/s 2 , hand between and 44.5 cm |
| World | [s] | Launches | Brier, forecast | Brier, time-indexed | Difference | Launches for 80% power |
|---|---|---|---|---|---|---|
| nominal | 0.2 | 6,119 | 0.0268 | 0.0257 | 0.0011 | 14,339 |
| nominal | 0.3 | 6,154 | 0.0297 | 0.0298 | 0.0001 | no gap |
| nominal | 0.4 | 6,074 | 0.0380 | 0.0359 | 0.0021 | 6,407 |
| doubled perturbations | 0.2 | 6,083 | 0.0458 | 0.0429 | 0.0030 | 4,411 |
| doubled perturbations | 0.3 | 6,160 | 0.0596 | 0.0554 | 0.0042 | 3,155 |
| doubled perturbations | 0.4 | 6,155 | 0.0771 | 0.0706 | 0.0065 | 2,043 |
Robot Task Planning
Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.
See all 246 papers