RoboScope Kit: Unified Interfaces for Fast and Scalable Embodied-AI Evaluation

Deng-Ke Wu; Yifan Jin; Yida Yang; Lei Qian; Zhaoye Zhou; Hongru Zhang; Kai Ye; Shuyuan Hu

cs.RONov 2, 2026

RoboScope Kit: Unified Interfaces for Fast and Scalable Embodied-AI Evaluation

Authors: Deng-Ke Wu, Yifan Jin, Yida Yang, Lei Qian, Zhaoye Zhou, Hongru Zhang, Kai Ye, Shuyuan Hu,

Organizations: School of AI, Shanghai Jiao Tong University · Central South University · Tongji University · ShanghaiTech University ·

Abstract

Simulation benchmarks are the standard way to compare robot policies, but each benchmark brings its own evaluator, observation layout, and action convention, and evaluation is slow: 500 RoboTwin Easy episodes with one simulator and one model on one GPU take 359.7 minutes. Profiling the official StarVLA evaluator shows that simulation occupies CPU cores while the GPU averages 14.3% utilization. RoboScope Kit wraps each model once in a Python class called a Policy, and each benchmark once in an adapter around the benchmark's own evaluator, so the benchmark's rules and scoring stay unchanged. Scheduling runs simulator workers and Policy replicas as separate pools, sizes both from measured probes, and shards work deterministically across GPU instances; runtime optimization renders only the observations an action-chunk Policy reads. Adapters cover six benchmark families. On LIBERO-Long with a StarVLA checkpoint on one 48 GiB RTX 4090, the concurrency search selects twelve simulator workers sharing three Policy replicas, which evaluate 500 episodes 4.45 $\times$ faster than the official StarVLA evaluation with equivalent success rates, 2.78 $\times$ faster than RLinf, and 1.52 $\times$ faster than vla-eval at its tuned configuration. With CUDA MPS, which lets the replicas run their kernels concurrently, the search selects sixteen workers sharing four replicas, 6.16 $\times$ faster than StarVLA, 3.85 $\times$ faster than RLinf, and 2.10 $\times$ faster than vla-eval. On RoboTwin Easy, four GPUs with chunk-boundary observation complete the 500 episodes in 14.4 minutes, 25.04 $\times$ faster than the single-GPU baseline, with identical success outcomes and terminal steps.

Figures & tables

Figure 1: RoboScope Kit connects Policies and benchmark adapters. After interface validation, acceleration has two parts: runtime optimization removes unconsumed simulation work inside each worker, and scheduling coordinates $K$ simulator workers that exchange observations and actions with $g$ Policy replicas. Each episode yields one episode record. The Policy class holds the model-side processing.

Figure 2: The official StarVLA LIBERO evaluator and its policy server on 50 LIBERO-Long episodes (five per task) on one RTX 4090. (a) Execution timeline over a 2-s window. The simulator runs in the evaluator process: physics and control on CPU cores, camera rendering on the GPU (each tick is one render of about 0.4 ms); the policy server runs on the GPU. (b) Over the whole run: evaluator time, the share of time the policy server is handling a request, and mean GPU utilization (the share of time a GPU kernel is executing).

Name	Definition	Setting
$\textsc{Probe}(K,g)$ fails	A worker is missing or fails, an out-of-memory error occurs, episodes do not finish within $\min(\tau_{c},\text{remaining }\tau)$ , or success on the episodes shared with the first probe falls below $(1-\delta)$ times the first probe’s	$\tau_{c}=15$ min, $\tau=30$ min, $\delta=0.15$
$\textsc{WorkerCap}(m^{\ast})$	$K^{\ast}+\lfloor(C-1-c(m^{\ast}))/\hat{c}\rfloor$ , where $\hat{c}$ is the CPU cost per worker fitted from the probes with the fewest and most workers at $g=G$ ; before two such probes, $\max(G,\lfloor(C-1)/2\rfloor)$ ; at most 64	–
ReplicaCap	$\min(16,C,G\lfloor(V-r_{v})/v\rfloor)$ , with $v$ measured by the first probe	$r_{v}=2$ GiB reserved per GPU
$\textsc{FitsMemory}(K,g)$	The busiest GPU’s peak memory, extrapolated from the incumbent with the replica footprint and the fitted memory per worker, stays below $V-r_{v}$	–
$\textsc{WithinLimits}(m)$	$c(m)\leq C-1$ , $u(m)\leq u_{\max}$ , and $t(m)\leq t_{\max}$ when $t$ is measured	$u_{\max}=0.95$ , $t_{\max}=0.40$
$\textsc{Knee}(m,m^{\ast})$	$q(m)\geq\bigl(1+\kappa\,(K(m)/K^{\ast}-1)\bigr)\,q(m^{\ast})$	$\kappa=0.25$

Table 1: Predicates and parameters of Algorithm 1 .

C

is the CPU quota in cores and

V

the memory per GPU;

q(m)

c(m)

u(m)

t(m)

, and

v(m)

are the measurements of probe

m

, and

K(m)

its worker count.

Figure 3: Work distribution and worker recovery. Workers atomically claim queued items and publish partial progress or completed results. After a worker exit or allocator out-of-memory error, the Control Runtime requeues unfinished work, discards its partial records, and starts a replacement worker, which replays the item from its first episode. Only completed items enter aggregation. Policy-replica exit has a different failure boundary: it aborts the evaluation (Section 3.7 ).

Figure 4: LIBERO-Long on one RTX 4090 with RoboScope Kit (Search), which selects

K12/g3

, and RoboScope Kit (Search+MPS), which selects

K16/g4

under CUDA MPS. (a) Evaluation time of each system: bars are medians and circles the launches; labels give the speedup over the official StarVLA evaluation. (b) Success rate minus that of the official StarVLA evaluation, with a 95% paired task-cluster bootstrap interval; the shaded band is the

\pm 2

-point equivalence margin. Table 8 lists the values.

Figure 5: RoboTwin Easy ablations. (a) Throughput of the Simulation Runtime ladder on one RTX 4090; each row adds one change to the row above. (b) Evaluation time of the 500 episodes along the measured configurations. Labels give each value and, in parentheses, the cumulative speedup over the first row. Tables 15 and 16 list the values.

Figure 6: Execution timeline of RoboScope Kit simulator workers (W) and Policy replicas on the 50 LIBERO-Long episodes of Figure 2 , one RTX 4090, with the categories of that figure. (a)–(c) A 2-s window of one run per configuration; rendering ticks mark the end of each simulation step. (d) Worker time, replica busy time, and GPU utilization over the whole run, mean of three runs.

Figure 7: Grid search and diagonal concurrency search on LIBERO-Long, one RTX 4090. Grid lines are full evaluations at fixed

K/g

, one launch per configuration, one line per replica count

g

, labelled at its end; the search path joins the steady-state probes in the order the search ran them, mean of three launches, with

g

marked where no grid line has that replica count; filled triangles are probes the search kept and hollow triangles probes it rejected; the star is the formal run at the selected configuration (median of three). (a) CUDA MPS off, on the sweep host with a 20-core quota: the grid takes eighteen launches and 5.4 h of evaluation, the search seven probes and 8.4 min (median) and selects

K12/g3

, within 1.2% of the fastest grid configuration,

K16/g4

. (b) CUDA MPS on, on the cross-system host with 12 pinned cores: the grid takes eighteen launches and 5.0 h of evaluation, the search seven probes and 8.4 min (median) and selects

K16/g4

, the fastest grid configuration;

K20/g5

is skipped because its predicted memory exceeds the GPU. GPU utilization of the grid configurations is in Tables 18 and 14 .

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

Method	Lifecycle transition	Policy behavior
load(context)	Created $\rightarrow$ Loaded	Load one checkpoint replica and declare capabilities
reset(context)	Loaded $\rightarrow$ Episode active	Allocate state keyed by episode_id
act(observation)	Episode active $\rightarrow$ Episode active	Produce one action or action chunk from canonical features
act_batch	Episode active $\rightarrow$ Episode active	Serve an observation batch with native batching when available
end_episode(id)	Episode active $\rightarrow$ Loaded	Release state for one episode
unload()	Loaded $\rightarrow$ Released	Release model and process-wide resources

Appendix

Table 2: Policy lifecycle methods and ownership.

Figure 8: Policy lifecycle within one replica. The checkpoint is loaded once; each episode adds and later removes its own state, and several episodes may be active at a time.

Benchmark	Protocol	Required canonical features	Action contract
LIBERO	libero-v1	images.primary $256\times 256\times 3$ uint8; images.wrist ; 16-D float32 proprioception; instruction	7-D float32 end-effector delta plus gripper
LIBERO-Plus	libero-plus-v1	Same named contract as LIBERO	7-D float32 end-effector delta plus gripper
RoboTwin	robotwin2-v1	Head, left-wrist, and right-wrist RGB; dual-arm joint state; instruction	14-D float32 absolute dual-arm joint and gripper target
RoboCasa	robocasa-v1	Primary RGB; optional wrist RGB; GR-1 state; instruction	29-D float32 arms, hands, and waist target
CALVIN	calvin-v1	Static RGB; gripper RGB; proprioception; instruction	7-D float32 relative end-effector and gripper action
MetaWorld	metaworld-v1	images.primary $480\times 480\times 3$ ; 4-D end-effector position and gripper aperture; instruction	4-D float32 end-effector delta and gripper command

Appendix

Table 3: Canonical features and actions.

Panel of Figure 9 — Figure 9: Two levels of parallel evaluation. Each instance runs a fixed subset of work items with its own queue, simulator workers, and Policy replicas; the shard results are validated and merged into one benchmark result.

System	Search space	Selected	Runs
vla-eval	Shards $\{8,\ldots,24\}$ , servers $\{1,2,3\}$ , maximum batch $\{4,\ldots,24\}$ , wait $\{0,5,20\}$ ms; 100-episode probes, then 500-episode runs (Table 5 )	24 shards, 1 server, batch 24, wait 5 ms	23
RLinf	Start: 10 environments, 1 pipeline stage; environments $\{4,8,12,16,20\}$ ; then 2 and 4 pipeline stages at 20 environments; 30-episode probes	20 environments, 1 pipeline stage, 1 rollout worker (30/30)	8

Appendix

Table 4: Offline tuning of the LIBERO-Long baselines.

Stage	Shards	Servers	Batch	Wait (ms)	Episodes	Successes	Evaluation (s)	Steady steps/s	Busy cores
0	10	2	10	5	500	451	1,215.9	–	–
1	10	1	10	5	100	90	317.1	107.9	9.4
1	10	2	5	5	100	91	278.5	110.4	9.8
1	8	1	8	5	100	88	355.2	86.0	7.7
1	8	2	4	5	100	87	356.6	88.2	8.0
1	12	1	12	5	100	88	318.9	123.6	10.7

Appendix

Table 5: vla-eval tuning runs on LIBERO-Long, in execution order. Stage 0 is the earlier 500-episode run.

	$K1/g1$	$K4/g1$	$K12/g2$
Evaluation time (s)	374.1	130.2	82.7
Robot control and forward kinematics	37.1	28.6	16.0
Physics integration	10.4	7.9	4.4
Observable updates and step bookkeeping	2.9	2.3	1.3
Rendering	2.9	3.1	3.3
Episode reset	11.9	9.7	5.7

Appendix

Table 6: Simulator-worker time by stage (% of worker wall time) and Policy-replica measurements, mean of three runs.

Stage	Wall (ms)	Kernel time (ms)	GPU busy (%)	Kernels
Image processing and tokenization	4.7	0.0	0	0
Vision encoder	23.9	9.7	41	1,486
Language model	24.6	14.7	60	1,641
Action head	0.6	0.4	–	30
Whole inference	57.7	26.1	45	3,220

Appendix

Table 7: One StarVLA LIBERO inference by model stage at

K1/g1

(medians over 273 inferences after 20 warmup inferences).

System	Success (%)	$\Delta$ success vs. StarVLA, pp [95% CI]	Agreement with StarVLA (%)	Evaluation time (s)	Throughput (steps/s)	Speedup vs. StarVLA
StarVLA	90.2	reference		2,990	45.8	1.00
vla-eval	89.8	$-0.4$ $[-1.6,\,+0.7]$	91.7	1,016	133.5	2.93
RLinf	90.3	$+0.1$ $[-2.6,\,+3.3]$	88.1	1,882	74.2	1.60
RoboScope Kit (Search)	89.9	$-0.3$ $[-0.8,\,0.0]$	97.9	671	204.6	4.45
RoboScope Kit (Search+MPS)	89.9	$-0.3$ $[-1.3,\,+0.8]$	96.5	485	282.5	6.16

Appendix

Table 8: Speed and outcome agreement in the cross-system comparison.

Launch	MPS	Successes	Search (s)	Eval. (s)	E2e (s)	Steps/s	CPU cores	GPU util. mean (%)	Peak GPU mem. (GiB)
Search 1, second host	Off	448	506.0	679.0	1,192.6	202.5	7.8	97.0	33.6
Search 2, second host	Off	447	504.1	675.2	1,187.0	204.6	7.8	97.2	33.6
Search 3, second host	Off	448	506.9	679.7	1,193.9	202.5	7.8	96.8	33.5
Fixed 1, first host	Off	449	–	671.2	737.6	204.9	7.7	96.8	33.8
Fixed 2, first host	Off	448	–	673.5	739.1	204.6	7.8	96.6	33.7
Fixed 3, first host	Off	451	–	669.3	731.0	204.6	7.8	96.6	33.7

Appendix

Table 9: RoboScope Kit (Search) and (Search+MPS) launches, with CPU and GPU use during the formal evaluation.

Order	System	Launch	Successes	Evaluation (s)	End-to-end (s)	Steps/s
1	StarVLA	1	451	2,989.9	3,116.5	45.8
2	RLinf	1	453	1,881.7	1,934.8	74.2
3	RLinf	2	452	1,907.5	1,955.3	72.0
4	StarVLA	2	451	2,968.0	2,983.5	46.1
5	RLinf	3	450	1,806.7	1,857.8	77.3
6	StarVLA	3	451	2,997.9	3,012.5	45.6

Appendix

Table 10: StarVLA and RLinf launches in execution order (same host and GPU).

Order	System	Successes	Evaluation (s)	End-to-end (s)	Steps/s
1	RoboScope Kit $K8/g2$	450	762.2	805.5	179.1
2	RoboScope Kit $K8/g2$	451	744.4	785.5	183.4
3	RoboScope Kit $K8/g2$	445	756.4	804.4	181.3
4	RoboScope Kit $K8/g2$	448	754.6	841.5	181.8
5	RoboScope Kit $K8/g2$	449	745.8	789.2	183.2
6	vla-eval	446	1,007.0	1,023.6	135.1

Appendix

Table 11: Launches of RoboScope Kit at a fixed

K8/g2

, the fastest configuration of the sweep with two replicas, and of vla-eval at its tuned configuration, in execution order.

Workload	$K/g$	$n$	Success/run	Eval. time (s)	Steps/s
LIBERO-Long	1/1	2	439/500	$3{,}069\pm 54$	$45.49\pm 0.81$
RoboTwin Easy	1/1	3	231/500	$21{,}582\pm 389$	$9.79\pm 0.17$
RoboTwin Easy	4/4	2	235/500	$5{,}646\pm 193$	$36.86\pm 1.26$

Appendix

Table 12: Initial fixed-configuration runs. Each run has 500 episodes. Timing and throughput are means

\pm

sample standard deviations;

n

counts runs.

Workload	$K/g$	Search (s)	Eval. (s)	E2e (s)	Successes	Peak GPU mem. (GiB)
LIBERO-Long	12/3	506	679	1,193	448/500	33.6
	12/3	504	675	1,187	447/500	33.6
	12/3	507	680	1,194	448/500	33.5
LIBERO-Long,	16/4	504	485	997	452/500	44.7
CUDA MPS,	16/4	505	487	1,000	447/500	44.5
12 pinned cores	16/4	502	483	994	449/500	44.4

Appendix

Table 13: Concurrency search launches on one RTX 4090, with a 20-core quota or, with CUDA MPS, 12 pinned cores on the cross-system host. Evaluation time excludes the search; end-to-end time is the wall time of the launch.

$K/g$	Successes	Eval. (s)	E2e (s)	Steps/s	CPU cores	GPU util. mean (%)	Peak GPU mem. (GiB)
1/1	451	3,655.4	3,676.2	37.4	1.2	15.0	9.6
2/1	453	1,908.3	1,932.5	71.6	2.2	29.5	10.5
4/1	450	1,177.9	1,204.5	115.9	3.4	48.5	11.5
8/1	451	1,166.5	1,190.5	117.2	3.5	49.5	13.5
12/1	448	1,171.4	1,192.2	117.6	3.6	49.8	16.1
16/1	451	1,186.3	1,208.9	115.5	3.5	49.0	18.0

Appendix

Table 14: LIBERO-Long at fixed

K/g

under CUDA MPS with 12 pinned cores, one launch per configuration (the grid of Figure 7 b).

Variant	Added Simulation Runtime behavior	Observations	Setup (s)	Wall (s)	Steps/s
00 official-like	Official-like path	2,100	46.9	237	8.88
01 shared context	Reuse Engine and Renderer	2,100	45.2	226	9.31
02 planner warmup	Skip cuRobo initialization	2,100	1.4	180	11.63
03 chunk boundary	Observe at 16-step chunk boundaries	132	1.4	71.9	29.23
04 passive force	Hoist duplicate passive force	132	1.7	64.8	32.38

Appendix

Table 15: Cumulative Simulation Runtime ladder on one RTX 4090.

Change	Configuration	Runs	Eval. time (s)	Factor
Baseline	$K1/g1$ , one GPU, stride 1	3	21,582	–
Four GPUs	$K4/g4$ , four GPUs, 80 cores	3	5,934	3.64 $\times$
Workers	$K16/g4$ , same hardware	1	3,951	1.50 $\times$
Stride and run setup	$K16/g4$ , stride 16	1	862	4.58 $\times$
Total				25.04 $\times$

Appendix

Table 16: Evaluation time for the 500-episode RoboTwin Easy manifest along the measured configurations. Each factor divides the previous row’s time by the current row’s.

Measurement	Value
Evaluation time	862 s
Evaluation phase (metrics recorder)	855 s
Policy loading	65.8 s
Runtime preparation	0.2 s
End-to-end time	928 s
Successes/episodes	231/500

Appendix

Table 17: Timing and outcome of the RoboTwin Easy stride-16 run (

K16/g4

, four RTX 4090, 80 cores).

Configuration	Successes	Evaluation time (s)	Steps/s	GPU util. (%)	Speedup vs. $K1/g1$	Efficiency per worker	Success agreement (%)	Terminal agreement (%)
Search $\rightarrow$ $K12/g3$	447.7	679.0	202.5	97.0	5.25	0.44	97.1	78.5
$K1/g1$	451	3,559.6	38.4	15.4	1.00	1.00	100.0	100.0
$K2/g1$	452	1,883.8	72.5	29.4	1.89	0.94	99.8	98.4
$K4/g1$	451	1,213.6	112.5	45.7	2.93	0.73	98.4	87.0
$K8/g1$	446	1,220.5	112.4	45.8	2.92	0.36	97.0	85.2
$K12/g1$	446	1,216.1	113.7	46.3	2.93	0.24	97.8	78.6

Appendix

Table 18: LIBERO-Long single-GPU worker and Policy-replica scaling (one launch per configuration).

Benchmark	Suites	Default work item	External benchmark assets
LIBERO	libero_spatial , libero_object , libero_goal , libero_10 , libero_90	Episodes of one task	Pinned Python environment, simulator, dataset, initial states
LIBERO-Plus	libero_spatial (2,402 tasks), libero_object (2,518), libero_goal (2,591), libero_10 (2,519)	Episodes of one variant	Pinned LIBERO-Plus environment, simulator archive, dataset, initial states
RoboTwin	demo_clean , demo_randomized ; 50 tasks	One task episode window	RoboTwin source and environment, checkpoint, validated seed cache
RoboCasa	tabletop ; 24 GR-1 tasks	One task with multiple episodes	GR-1 simulator and task source, dataset, EGL environment
CALVIN	task_D_D , task_ABC_D , task_ABCD_D	Consecutive sequences	CALVIN source, dataset, config, official 1,000-sequence manifest
MetaWorld	mt50 ; 50 tasks	One task with all its episodes	Pinned MT50 environment and protocol artifact

Appendix

Table 19: Adapter catalog.

Robotic Policy Evaluation

Robotic policy evaluation measures whether learned controllers succeed reliably, generalize across conditions, recover from failures, and behave safely.

See all 139 papers

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation
Jul 1, 2026Byeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo +4World Model Learning Robotic Policy Evaluation
1KAIST · 2Config
vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models
Mar 14, 2026Suhwan Choi, Yunsung Lee, Yubeen Park +4VLM Evaluation Vision-Language-Action Models
Allen Institute for AI (AI2) · Seoul National University
RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild
Nov 2, 2026Tan Sang Nguyen, Yi Ru Wang, Shailesh Shailesh +24Robotic Policy Evaluation Embodied Agent Benchmarks
National University of Singapore · University of Washington · Stanford University +6

Jul 1, 2026J/K move · Enter open · S save

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

Byeongguk Jeon, Seonghyeon Ye, JaeHyeok Doo, Sungdong Kim+3

1KAIST · 2Config

World Model Learning Robotic Policy Evaluation World Model Evaluation Video World Models

Open paper AskPDF arXiv

Video world models are emerging as a scalable alternative for evaluating generalist robot policies, bypassing the physical constraints and engineering burdens of real-world deployment. However, evaluating policies with video world models remains challenging, as world-model errors can make generated rollouts unreliable and slow inference limits large-scale throughput. We introduce RoboWorld, an automated evaluation pipeline that pairs a fast autoregressive video world model with a task-progress-aware vision-language model scoring. To enable reliable long-horizon autoregressive world-model rollouts, we propose Step Forcing, which combines anchored and one-step self-forwarded contexts to reduce train-test mismatch while preserving action-observation dynamics. Together, these components enable RoboWorld to align strongly with real-world robot evaluation across tasks and environments, achieving Pearson's r = 0.989 and Spearman's

ρ

= 0.970.

Abstract

Robotic Policy Evaluation

Explore similar work

RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation

vla-eval: A Unified Evaluation Harness for Vision-Language-Action Models

RoboColosseum: Evaluating Physical Intelligence at Scale In-The-Wild