Time-Indexed Skill Success Models for Latency-Aware Skill Selection and Timing in Dynamic Manipulation

Mohsen Omidi; Maria Chiara Leva; Brian Vaughan

cs.RONov 5, 2026

Time-Indexed Skill Success Models for Latency-Aware Skill Selection and Timing in Dynamic Manipulation

Authors: Mohsen Omidi, Maria Chiara Leva, Brian Vaughan

Organizations: Virtual Interaction Research Lab (VIRaL), School of Media, Technological University Dublin, Dublin, Ireland. · School of Food Science and Environmental Health, Technological University Dublin, Dublin, Ireland.

Abstract

Robots that manipulate moving objects act on observations that are outdated when the chosen skill starts, and the best skill can change within a few hundred milliseconds. Skill competence models, however, predict success for a launch at the moment of observation. We propose time-indexed skill success models, which predict whole-task success when a skill is launched after a given delay. Trained on rollouts branched at a grid of launch delays, they let a receding-horizon planner select both the skill and its launch time under fixed or random latency. In a MuJoCo study in which a UR5e captures a sliding puck with three skills, the planner exceeded a static competence model by 2.8 to 6.2 percentage points at 100 to 300 ms of latency, including when that model was evaluated on point or sampled forecasts of the launch state. Across five complete runs (three training seeds and two distribution shifts), all 80 paired 95% intervals against four baselines excluded zero. On delayed launches, the time-indexed model was calibrated (expected calibration error 0.009 against 0.239), and pausing the world during latency overstated success by up to 23 points. On a real UR5e that captures a rolling ball after 0.2 s of injected latency, a static model trained in a hardware-matched simulator predicted 99% success on a prespecified subset of 35 delayed launches that it rated at least 0.8. Of these, 63% succeeded (95% interval 46 to 77%), and the time-indexed model predicted 56%. In this single-skill, predictable regime, a matched physics-based forecast was also well calibrated.

Figures & tables

Panel of Figure 1 — Fig. 1: Task and timing problem. Left: A UR5e must capture a puck that slides towards an open edge and place it in a bin. Every 0.1 s, it chooses one of three skills and an optional extra wait $\delta$ , and a decision on the observation at time $t$ takes effect at $t+\ell+\delta$ , where $\ell$ is the latency. Right: For slow launches, InterceptGrasp is the best skill if launched at once, but WaitPick is the best after 200 ms (oracle rollouts of the held-out episodes).

	Fixed $\ell$ [ms]					Random $\ell$
Planner	0	100	150	200	300	Heavy	Unif.
B0 feasibility rule	63.6	55.9	48.1	46.5	41.6	50.1	48.5
B1 static	63.6	56.9	50.3	47.5	43.3	51.7	50.1
B1r myopic wait	61.8	54.9	47.3	45.4	40.5	n/a	n/a
B1p point forecast	63.7	55.8	49.3	46.6	43.9	50.0	48.5
B1s sampled forecasts	65.0	57.0	51.9	49.1	42.8	52.1	51.1

TABLE I: Whole-task success (%) on the 2,000 held-out episodes under fixed latency

\ell

and under random per-decision latency known only through its distribution. Best entries in bold. Always-

k

, which launches one fixed skill at once, reaches at most 44.3% at 0 ms (slightly below the oracle rollouts, as every planned launch is charged at least one 4 ms control period) and 36.8% at 300 ms.

Panel of Figure 3 — Fig. 2: Paired differences in whole-task success between TI-flat and each baseline on the 2,000 held-out episodes, with 95% bootstrap intervals (markers offset horizontally). TI-flat exceeds every baseline at 100 ms and above.

	ECE		Confident
Predictor	InterceptGrasp	Block	subset (%)
Time-indexed	0.009 [0.006, 0.016]	0.006 [0.005, 0.013]	72.0
Static	0.239 [0.226, 0.254]	0.234 [0.221, 0.249]	98.8
Static + temperature	0.119 [0.108, 0.131]	0.133 [0.119, 0.148]	66.8
Static, point forecast	0.126 [0.118, 0.135]	0.092 [0.083, 0.101]	71.5
Static, sampled forecasts	0.059 [0.051, 0.066]	0.053 [0.045, 0.061]	70.0

TABLE II: ECE of the skill-success predictors on delayed launches (

\tau>0

) of the held-out episodes, with 95% episode-bootstrap intervals. The last column gives the mean prediction (%) on the confident subset, the 3,228 InterceptGrasp launches with delays of 150 to 300 ms that the static model rated at least 0.8.

Panel of Figure 5 — Fig. 3: InterceptGrasp skill-success predictors on held-out launches. (a) Reliability on delayed launches. The static model is over-confident at both ends, a refitted temperature only flattens it, and the time-indexed model is close to the diagonal. (b) ECE against the launch delay. Block shows the same pattern.

Panel of Figure 6 — Fig. 4: Hardware test. (a) Set-up: a UR5e with an open Hand-E gripper captures a golf ball rolling down an inclined board (blue arrow) at the catch line (red, dashed). (b) ECE against the launch delay in the hardware-matched simulator, with nominal and doubled ( $2\times$ ) perturbations. (c) Mean predicted capture probability on the 35 delayed test launches ( $\tau=0.2$ s) that the static model rated at least 0.8, with the observed rate (line) and its 95% Wilson interval (band). † Static model on the state estimated at launch (an oracle).

Group	Parameter	Value
Scene	channel	inner width 0.48 m ( $x$ 0.26 to 0.74 m in the robot frame), launch line $y=0.45$ m, open exit edge $y=-0.35$ m, rails 3 cm high
	puck	cylinder, diameter 40 mm, height 24 mm, mass 50 g
	bin	centre $(0.08,0.40)$ m, $14\times 14$ cm
	gripper	parallel jaw, tool-centre offset 160 mm, stroke 50 mm, V-groove pads, 0.10 m/s per finger, 60 N, 20 ms actuation delay
	physics	MuJoCo / MuJoCo Warp, time step 2 ms
Robot	arm	UR5e (MuJoCo Menagerie), joint speed $\leq 180^{\circ}$ /s, tool speed $\leq 1$ m/s, tool acceleration $\leq 6$ m/s 2

TABLE S1: Parameters of the simulated task, the data and the models (configuration of training seed 0; shifts A and B change only the rows marked).

Model	Mean	Median	95th pct.	99th pct.	Max.	Quantiles used by the -dist planners
Uniform, 50 to 300 ms	175	175	287	297	300	66, 97, 128, 159, 191, 222, 253, 284
Heavy-tailed, $40+90\,e^{0.55n}$ , cap 500 ms	145	130	262	364	500	79, 95, 109, 123, 138, 158, 187, 249

TABLE S2: Random latency models (milliseconds).

Panel of Figure 9 — Fig. S1: Whole-task success of each skill against its launch delay, by launch speed (oracle rollouts of the 2,000 held-out episodes). The dynamic skills lose value within a few hundred milliseconds, whereas WaitPick barely depends on timing, so the best skill changes with the delay.

$\tau$ [s]	WaitPick only	InterceptGrasp only	Block only	Several	None
0.00	7.3	11.0	11.1	36.2	34.4
0.10	7.8	9.9	8.8	33.9	39.7
0.20	7.9	7.8	6.8	33.5	44.0
0.30	6.9	6.7	5.2	32.9	48.4
0.45	7.0	6.0	2.1	31.4	53.5
0.60	6.2	4.1	0.5	30.9	58.2

TABLE S3: Share of held-out episodes (%) by the set of skills that complete the task when launched after delay

\tau

(oracle rollouts).

Fig. S2: Skills that would complete the task in each held-out episode, plotted over launch speed and heading, for launch delays of 0, 0.3 and 0.6 s (oracle rollouts). Block-only episodes (fast launches) vanish first; WaitPick-only episodes persist.

Panel of Figure 12 — Fig. S3: Main comparison on the 2,000 held-out episodes. (a) All planners lose success as latency grows, and the time-indexed planners lose the least. (b) Paired differences between TI-flat and each baseline. TI-flat exceeds every baseline at 100 ms and above, including the static model averaged over sampled forecasts (B1s). Markers are offset horizontally for legibility.

	Injected latency $\ell$ [ms]
Comparison	0	100	150	200	300
TI-flat $-$ B0 feasibility rule	$+$ 2.2 [1.1, 3.3]	$+$ 3.9 [2.7, 5.0]	$+$ 7.2 [5.8, 8.6]	$+$ 6.4 [5.1, 7.8]	$+$ 6.8 [5.3, 8.2]
TI-flat $-$ B1 static model	$+$ 2.2 [1.1, 3.2]	$+$ 2.9 [1.8, 3.9]	$+$ 5.1 [3.7, 6.2]	$+$ 5.3 [4.0, 6.7]	$+$ 5.1 [3.8, 6.5]
TI-flat $-$ B1p static, point forecast	$+$ 2.1 [0.9, 3.2]	$+$ 4.0 [2.6, 5.3]	$+$ 6.0 [4.7, 7.4]	$+$ 6.2 [4.9, 7.8]	$+$ 4.5 [3.2, 5.9]
TI-flat $-$ B1s static, sampled forecasts	$+$ 0.9 [ $-$ 0.1, 1.9]	$+$ 2.8 [1.6, 4.0]	$+$ 3.4 [2.2, 4.6]	$+$ 3.8 [2.5, 5.1]	$+$ 5.7 [4.3, 6.9]
TI-flat $-$ TI-fac	$+$ 0.2 [ $-$ 0.8, 1.3]	$+$ 1.6 [0.4, 2.5]	$+$ 0.2 [ $-$ 0.9, 1.2]	$+$ 0.6 [ $-$ 0.6, 1.8]	$+$ 1.2 [0.2, 2.2]

TABLE S4: Paired differences in whole-task success (percentage points) with 95% episode-bootstrap intervals. Bold entries have intervals that exclude zero. The last row compares models with the same factored structure, which isolates the effect of the launch-delay index.

Planner	Latency	Heavy-tailed	Uniform
B0 feasibility rule	–	50.1	48.5
B1 static	–	51.7	50.1
B1p point forecast	realised	50.2	49.3
B1p point forecast	distribution	50.0	48.5
B1s sampled forecasts	realised	52.2	51.1
B1s sampled forecasts	distribution	52.1	51.1

TABLE S5: Random per-decision latency. Top: whole-task success (%). Planners are told either the realised latency of each decision or only its distribution. Bottom: paired differences (points) with 95% intervals. Bold intervals exclude zero.

Panel of Figure 15 — Fig. S4: Mechanism and evaluation protocol. (a) Share of episodes in which a planner launched InterceptGrasp, the skill found no feasible intercept when it started, and WaitPick or Block launched at the same delay would have completed the task. (b) Success with the world paused during latency and computation minus success with the clock running, on identical episodes.

Panel of Figure 16 — Fig. S5: Replication over five complete runs, each with its own training data, models and evaluation (training seeds 0, 1 and 2, shift A with faster and wider launches, and shift B with slower and noisier perception). Every comparison at 100 ms and above has its 95% interval above zero.

Comparison	Condition	Positive	Range [pts]
TI-flat $-$ B1s	fixed, 100 to 300 ms	20/20	$+$ 2.3 to $+$ 6.3
TI-flat $-$ B1s	fixed, 0 ms	3/5	$+$ 0.9 to $+$ 1.5
TI-flat $-$ B1p	fixed, 100 to 300 ms	20/20	$+$ 3.0 to $+$ 7.1
TI-flat $-$ B1p	fixed, 0 ms	5/5	$+$ 1.9 to $+$ 3.9
TI-flat $-$ B1	fixed, 100 to 300 ms	20/20	$+$ 2.1 to $+$ 6.9
TI-flat $-$ B1	fixed, 0 ms	5/5	$+$ 1.8 to $+$ 2.7

TABLE S6: Replication over five complete runs (training seeds 0, 1, 2; parameter shifts A and B). Cells are runs

\times

latencies (fixed) or runs

\times

latency models (random). In the random rows the planners know only the latency distribution; the last row compares TI-flat told the realised latency with TI-flat told only its distribution. Positive means that the 95% interval is above zero.

Predictor	InterceptGrasp, F	InterceptGrasp, G	Block, F	Block, G
Time-indexed	0.009 [0.006, 0.016]	0.008 [0.006, 0.014]	0.006 [0.005, 0.013]	0.006 [0.005, 0.012]
Static	0.239 [0.226, 0.254]	0.197 [0.184, 0.211]	0.234 [0.221, 0.249]	0.116 [0.105, 0.130]
Static + temperature	0.119 [0.108, 0.131]	0.142 [0.130, 0.152]	0.133 [0.119, 0.148]	0.084 [0.076, 0.097]
Static, point forecast	0.126 [0.118, 0.135]	0.082 [0.074, 0.089]	0.092 [0.083, 0.101]	0.057 [0.051, 0.065]
Static, sampled forecasts	0.059 [0.051, 0.066]	0.051 [0.044, 0.059]	0.053 [0.045, 0.061]	0.028 [0.023, 0.035]

TABLE S7: Calibration on delayed launches (

\tau>0

) of held-out episodes. Top: expected calibration error (15 equal-width bins) with 95% episode-bootstrap intervals, for skill success (F) and whole-task success (G). Bottom: mean predictions of each model against the observed rate (%) on launches 150 to 300 ms after the observation on which the static model predicts at least 80% skill or task success.

Panel of Figure 19 — Fig. S6: Calibration of skill-success predictors on held-out launches. (a, b) Reliability on delayed launches (equal-mass bins). The static model is severely over-confident, a refitted temperature only flattens it, and the time-indexed model is close to the diagonal. (c, d) ECE against the launch delay. A single temperature improves calibration at long delays at the cost of calibration at zero delay, and forecast-based corrections reduce the error without removing it. The sampled-forecast predictor appears in (c, d) only.

Skill, target	Predictor	ECE	ECE (mass)	Brier	Reliability	Resolution	NLL	Over-conf.
InterceptGrasp, F	Time-indexed	0.009	0.009	0.075	0.000	0.125	0.235	$+$ 0.7
InterceptGrasp, F	Static	0.239	0.234	0.241	0.089	0.047	2.194	$+$ 14.4
InterceptGrasp, F	Static + temperature	0.119	0.120	0.167	0.019	0.053	0.494	$+$ 11.9
InterceptGrasp, F	Static, point forecast	0.126	0.119	0.135	0.019	0.084	1.069	$-$ 3.3
InterceptGrasp, F	Static, sampled forecasts	0.059	0.057	0.113	0.004	0.091	0.514	$-$ 3.6
InterceptGrasp, G	Time-indexed	0.008	0.008	0.063	0.000	0.098	0.196	$+$ 0.5

TABLE S8: Full calibration metrics on delayed launches (report half of the held-out episodes): ECE with equal-width and equal-mass bins, Brier score with Murphy’s reliability and resolution terms, negative log-likelihood, and mean prediction minus observed rate (points).

	All stale, $\ell$ [ms]				Avoidable stale, $\ell$ [ms]
Planner	100	150	200	300	100	150	200	300
B0 feasibility rule	5.4	12.2	12.0	14.2	3.0	7.3	7.3	7.9
B1 static model	9.5	15.3	15.2	17.2	1.7	5.9	6.2	6.3
B1p static, point forecast	7.0	10.0	9.2	6.7	1.6	3.9	4.0	2.7
B1s static, sampled forecasts	8.1	10.8	9.7	7.5	1.1	2.5	2.1	1.7
TI-fac (ours)	8.3	10.2	10.0	8.9	0.3	1.5	1.7	0.9

TABLE S9: Stale InterceptGrasp launches (the skill finds no feasible intercept when it starts) as a share of episodes (%), and the avoidable subset in which WaitPick or Block launched at the same delay would have completed the task.

	$\ell$ [ms]
Planner	100	150	200	300
Always WaitPick	$-$ 0.2	$-$ 0.2	$+$ 0.2	$-$ 0.1
Always InterceptGrasp	$+$ 3.7	$+$ 7.2	$+$ 8.7	$+$ 11.1
B0 feasibility rule	$+$ 8.1	$+$ 15.9	$+$ 17.5	$+$ 22.4
B1 static model	$+$ 7.4	$+$ 13.9	$+$ 16.8	$+$ 20.9
B1p static, point forecast	$+$ 8.4	$+$ 14.8	$+$ 17.5	$+$ 20.2

TABLE S10: Success with the world frozen during latency and computation minus success with the clock running (points), on the same episodes.

		vs B1, $\ell$ [ms]			vs B1s, $\ell$ [ms]
Band	$n$	150	200	300	150	200	300
slow, 0.30 to 0.50	579	$+$ 7.9	$+$ 9.0	$+$ 6.9	$+$ 2.2	$+$ 4.3	$+$ 3.1
medium, 0.50 to 0.75	726	$+$ 5.0	$+$ 5.1	$+$ 6.2	$+$ 3.6	$+$ 4.3	$+$ 8.0
fast, 0.75 to 1.00	695	$+$ 2.7	$+$ 2.6	$+$ 2.4	$+$ 4.2	$+$ 2.7	$+$ 5.3

TABLE S11: TI-flat minus B1 and minus B1s by launch-speed band [m/s] (points). Bold entries have 95% intervals above zero.

Predictor	Conf. delayed	Brier	$\Delta$ Brier [95%]	AUC
Static $F(z)$	0.99	0.260	$+$ 0.230 [0.110, 0.360]	0.84
$F$ , forecast state	0.60	0.020	$-$ 0.011 [ $-$ 0.021, $-$ 0.003]	1.00
$F$ at launch †	0.61	0.006	$-$ 0.024 [ $-$ 0.056, $-$ 0.005]	1.00
Time-indexed $G$	0.56	0.031		1.00
Observed	0.63 [0.46, 0.77]

TABLE S12: Hardware test on 49 held-out launches. The confident delayed subset contains the 35 launches with

\tau=0.2

s and a static prediction of at least 0.8 (mean prediction, and observed rate with its 95% Wilson interval).

\Delta

Brier is the difference from the time-indexed model, with 95% intervals from 5,000 bootstrap resamples of launches. † Uses the frames recorded during the delay, which are not available when deciding.

Group	Item	Value
Hardware	robot	UR5e, Cartesian servo streaming at 125 Hz (look-ahead 0.05 s, gain 1000)
	tool	Robotiq Hand-E, fingers fully open (53 mm), bag behind the fingers; lateral clearance 5.2 mm
	task	golf ball (42.7 mm) rolling on a 58.4 cm by 48.7 cm board inclined by about $3^{\circ}$ , released by hand from rest
	perception	RealSense D455f, 90 frames/s, colour detection, robot-calibrated board frame (about 1 mm)
	estimation	extended Kalman filter on a fitted acceleration field (25 parameters, fit RMS 2.5 mm on 72 rolls)
	skill	lateral move along the catch line to the predicted crossing, time-optimal with $v\leq 0.5$ m/s and $a\leq 2$ m/s 2 , hand between $y=12$ and 44.5 cm

TABLE S13: Hardware test: set-up, hardware-matched simulator, calibration and protocol.

World	$\tau$ [s]	Launches	Brier, forecast	Brier, time-indexed	Difference	Launches for 80% power
nominal	0.2	6,119	0.0268	0.0257	$+$ 0.0011	14,339
nominal	0.3	6,154	0.0297	0.0298	$-$ 0.0001	no gap
nominal	0.4	6,074	0.0380	0.0359	$+$ 0.0021	6,407
doubled perturbations	0.2	6,083	0.0458	0.0429	$+$ 0.0030	4,411
doubled perturbations	0.3	6,160	0.0596	0.0554	$+$ 0.0042	3,155
doubled perturbations	0.4	6,155	0.0771	0.0706	$+$ 0.0065	2,043

TABLE S14: Forecast against time index inside the launch window of the hardware protocol (launch 0.3 to 0.7 s before the line) in the hardware-matched simulator. The last column gives the number of paired launches needed to detect the difference with a two-sided test at the 5% level and 80% power.

Robot Task Planning

Robot task planning turns goals and observations into executable actions while coordinating skills, motion, uncertainty, and safety.

See all 246 papers

MOSAIC: Skill-Centric Manipulation Planning with Physics Simulation
Apr 23, 2025Itamar Mishani, Yorai Shaoul, Maxim LikhachevLong-Horizon Robotic Manipulation Robot Task Planning
Robotics Institute, School of Computer Science, Carnegie Mellon University, United States
RoboCoach: World Models as Active Coaches for Compositional Robot Skills
Sep 30, 2026Jiajun Liu, Yifan Chen, Yichao Liu +7Robot Policy Learning Robot Policy Adaptation
Renmin University of China · Tsinghua University · Shanghai Qizhi Institute +2
Evolving Skill Modules under a Fixed Planner: Versioning, Rollback, and Runtime Governance for Long-Lived Robot Systems
Apr 9, 2026Xue Qin, Simin Luan, Cong Yang +1Robotic Policy Evaluation Robot Safety
School of Software, Harbin Institute of Technology, Harbin, China · School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China · School of Future Science and Engineering, Soochow University, Suzhou, China

Apr 23, 2025J/K move · Enter open · S save

MOSAIC: Skill-Centric Manipulation Planning with Physics Simulation

Itamar Mishani, Yorai Shaoul, Maxim Likhachev

Robotics Institute, School of Computer Science, Carnegie Mellon University, United States

Long-Horizon Robotic Manipulation Robot Task Planning

Open paper AskPDF arXiv

Planning long-horizon manipulation motions using a set of predefined skills is a central challenge in robotics; solving it efficiently could enable general-purpose robots to tackle novel tasks by flexibly composing generic skills. Solutions to this problem lie in an infinitely vast space of parameterized skill sequences -- a space where common incremental methods struggle to find sequences that have non-obvious intermediate steps. Some approaches reason over lower-dimensional, symbolic spaces, which are more tractable to explore but may be brittle and are laborious to construct. In this work, we introduce MOSAIC, a skill-centric, multi-directional planning approach that targets these challenges by reasoning about which skills to employ and where they are most likely to succeed, by utilizing physics simulation to estimate skill execution outcomes. Specifically, MOSAIC employs two complementary skill families: Generators, which identify ``islands of competence'' where skills are demonstrably effective, and Connectors, which link these skill-trajectories by solving boundary value problems. By focusing planning efforts on regions of high competence, MOSAIC efficiently discovers physically-grounded solutions. We demonstrate its efficacy on complex long-horizon problems in both simulation and the real world, using a diverse set of skills including generative diffusion models, motion planning algorithms, and manipulation-specific models. Visit skill-mosaic.github.io for demonstrations and examples.

Abstract

Robot Task Planning

Explore similar work

MOSAIC: Skill-Centric Manipulation Planning with Physics Simulation

RoboCoach: World Models as Active Coaches for Compositional Robot Skills

Evolving Skill Modules under a Fixed Planner: Versioning, Rollback, and Runtime Governance for Long-Lived Robot Systems