Vision-language-action (VLA) models have emerged as a promising paradigm for general-purpose robotic manipulation. In our production setting, hundreds of VLA-powered robots are already operating in real-world environments, creating an immediate need for efficient fleet serving. We present XInfer, a production-deployed VLA serving system designed to reduce the inference infrastructure cost of large-scale robotic deployments. XInfer jointly coordinates heterogeneous VLA computation with the evolving response requirements of robotic execution. By exploiting computational parallelism while managing interference, XInfer improves GPU efficiency while protecting time-critical robotic feedback. We evaluate XInfer across three VLA models using production-derived workloads, achieving 1.26-3.75 times the effective robot capacity (ERC) of state-of-the-art systems under physical-execution constraints. Our implementation is publicly available at https://github.com/X-Square-Robot/xinfer.
Figures & tables
Figure 1: Production parcel handling. A carton (top) and a soft parcel (bottom) are reoriented and pushed onto the conveyor. Each row follows the same parcel from left to right.
Figure 2: Projected parcel throughput on one GPU (VLA-2B), combining measured latency with single-robot trials. Guides mark 13k parcels/h; ERC requires response P50 <150ms and zero request errors. Shading reflects trial variation; dashed curves and hatching mark extrapolation.
Figure 3: VLA computation and real-time chunking. (a) Conditioning keys and values are reused across denoising steps. (b) The frozen prefix aligns the next chunk with committed actions while inference overlaps execution. The illustrated early response satisfies L<dΔ<TE , so the next send occurs at tk+TE .
Figure 4: Five separately recorded request streams aligned on a common 2-s axis. Top: bars mark requests in flight and ticks mark sends. Bottom: the superimposed in-flight count.
Figure 5: XInfer architecture and illustrative execution. (a) Central scheduling controls admission at the ViT, Prefill, and Action entries. (b) Admission permits concurrent work when A’s response is predicted to meet its deadline.
Figure 6: Scheduling decisions (illustrative predictions). (a) Protect A using remaining slack. (b) Refresh shared slack after each launch. (c) Defer w for an earlier-deadline forecast. Panel (b) fixes t and H={A} , with additive costs and other checks passing. Dashed outlines mark forecasts; hatching marks excess cost in (b) and extra delay in (c).
Input
VLA-2B
Wall-OSS-0.5
π0.5
Camera 1 (pixels)
224×352
196×308
196×308
Cameras 2/3 (pixels)
224×320
196×280
196×280
Patch size (pixels)
16×16
14×14
14×14
Visual tokens
217
217
217
Instruction tokens
41
41
40
State tokens
72
74
127
Table 1: Input configuration. Token segments sum to the full prefix; action tokens and sequence padding are excluded.
Figure 7: Serving performance across three models. Latency uses a log scale; dashed lines mark 150 ms . Large markers and guides identify confirmed ERCs for XInfer and Robion.
Serving method
VLA-2B
Wall-OSS-0.5
π0.5
XInfer
18
15
24
SGLang
10
5
14
vLLM-Omni
10
8
15
FlashRT
5
4
9
Robion
11
10
19
Table 2: ERC (robots) under the response-latency criterion.
Figure 8: Four workloads with matched resources: (a–c) show three metrics for alternating phases; (d–f) show attainment for the other workloads. Curves show means with min–max whiskers. Large markers denote ERC in (a) and 33% lower mean latency for XInfer versus Slack-Gate at 15 robots in (b). Shading in (d) marks added robots’ measured active periods.
Figure 9: Cumulative ablation on π0.5 at 24 robots. Bars show per-window means, rows retain prior mechanisms, and arrows show incremental gains. Execution-aware concurrency enables intra-stage concurrency. Cross-stage coordination combines Context–Action overlap and independent progression. Full engine: 2.40 × throughput and 3.06 × P50 speedup over All-Off.
Figure 10: Scheduler ablations. Mean SLO attainment drop relative to full XInfer for each scheduler variant. Appendix C.3 gives workload settings.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 11: Response delay increases physical effort, with different effects across parcel types. Each point averages five tasks of one type; the black line is the mean of the four types and the band their range. Zero denotes no added delay. Full counts are in Table 4 .
Added delay (ms)
Handled parcels/h
Completed parcels/h
Handled SD
0
1968
1920
86.5
25
1944
1912
43.3
50
2020
1968
73.3
100
1892
1812
66.1
150
1552
1516
96.2
200
1376
1352
129.4
Appendix
Table 3: Hourly parcel rates under added response delay. Handled and completed are measured aggregate rates; handled SD is estimated from three trial markers digitized from the source plot. Figure 2 uses handled rates.
Added delay (ms)
Carton A
Gray bag
Carton B
Long carton C
Mean
0
6/5
5/5
5/5
5/5
1.05/1.00
50
6/5
5/5
6/5
6/5
1.15/1.00
100
5/5
6/8
6/5
7/5
1.20/1.15
150
6/7
5/8
7/6
6/5
1.20/1.30
200
6/6
6/7
8/6
6/5
1.30/1.20
250
8/5
6/8
6/5
5/6
1.25/1.20
Appendix
Table 4: Manipulation attempts (flip/push) by parcel type under added response delay. Type columns give totals over five tasks; Mean gives averages over twenty tasks.
Setting
Configuration
Scope
Single-model serving; distributed placement and routing are excluded.
Execution units
Context and Action each use a sequential CUDA stream with a dedicated CPU submission thread.
Round progression
Context work in round k overlaps Action work from round k−1 ; the next round starts after both finish.
Result publication
Publish completed Action results after the round barrier (a reproduction choice); CPU output processing may overlap the next GPU round.
SM allocation
An offline lookup selects the Context SM quota that minimizes full-round CUDA completion time; Action remains unrestricted.
CUDA Graphs
Capture supported execution-unit and Context SM configurations before measurement.
Appendix
Table 5: Robion configuration used in the serving-system comparison.
N
Configuration
QPS
P50
P95
P99
8
All-Off
44.26
20.00
26.92
32.46
M
44.25
19.51
24.50
30.54
M+C
44.26
19.87
24.82
27.34
M+R
44.25
20.40
31.38
33.66
M+G
44.27
19.67
20.54
21.08
M+C+R+G
44.27
19.85
21.19
21.71
Appendix
Table 6: Incremental ablations on π0.5 using a multi-stream baseline. M denotes execution-aware concurrency, C CPU–GPU overlap, R independent progression, and G GPU-stage overlap. C, R, and G are enabled individually on top of M. All-Off disables all four mechanisms. QPS is measured in requests/s. P50, P95, and P99 are client response latencies in ms.
Model
State
Either
Both
CPU/GPU
VLA-2B
Ind., off
85.49
58.36
0.00
Ind., on
99.72
79.73
56.85
Lock., on
90.79
25.45
11.28
Wall-OSS-0.5
Ind., off
82.05
55.35
0.00
Ind., on
98.78
87.78
26.88
Lock., on
87.55
68.99
14.49
Appendix
Table 7: Wall-time shares (%) at N=32 over 61–69 s. Ind./Lock.: independent/lockstep; off/on: CPU overlap. Either/Both: union/intersection of Context/Action kernel intervals. CPU/GPU: kernel overlap with instrumented CPU preparation/output scopes.
Figure 12: CPU–GPU overlap reduces idle gaps in a separate profiler diagnostic. VLA-2B serves 32 robots with overlap (a) disabled and (b) enabled. The figure shows the 61–61.5 s excerpt; the outlined gap marks an observed GPU idle interval. Table 7 reports statistics for the full profiling window.
Figure 13: Remaining-time prediction errors for running stages. Positive error is underprediction. Each represented request has equal total weight within its robot population. Curves include both tails; the horizontal scale is logarithmic outside [−1,1]ms .
Workload
FS
No interference model
Strict gating
No arrival- aware adm.
Entry-only admission
Zero-slack promotion
Phases, N=15
96.49
+24.29 (3)
+7.73 (3)
−0.40 (2)
+4.87 (3)
+0.27 (1)
Phases, N=16
95.57
+45.74 (3)
+11.16 (3)
+5.02 (3)
+15.97 (3)
+0.19 (1)
Phases, N=18
95.96
+45.47 (3)
+21.34 (3)
+11.31 (3)
+12.13 (3)
+1.04 (3)
Phases, N=20
94.23
+45.68 (3)
+30.49 (3)
+13.80 (3)
+11.07 (3)
−0.29 (1)
Bursty arrivals
98.28
+9.92 (3)
+7.54 (3)
+0.58 (2)
+0.71 (2)
+2.26 (3)
Multiple budgets
96.01
+14.24 (3)
+22.29 (3)
+1.47 (2)
+5.64 (3)
−0.25 (2)
Appendix
Table 8: GPU component ablations. Strict gating jointly replaces slack stealing and arrival-aware admission. FS: SLO attainment of full FleetServe (mean of the nine rounds in the three campaigns, %). Other columns: gain of full FleetServe over each variant (percentage points, mean of three rounds paired with full FleetServe in the same campaign), with the number of rounds in which full FleetServe is higher. Bursty robot arrivals use the full measurement window.
Workload
Overall attainment (%)
Tight attainment (%)
Wide mean latency (ms)
Goodput (responses/s)
Phases, N=15
97.3 / 97.8
94.5 / 95.6
114 / 148
80.4 / 80.4
Phases, N=16
89.2 / 96.6
78.3 / 93.2
200 / 208
74.0 / 79.0
Phases, N=18
84.3 / 96.5
68.7 / 93.0
260 / 262
68.7 / 78.4
Bursty arrivals
97.7 / 97.8
95.7 / 96.0
94 / 95
66.6 / 66.5
Multiple budgets
94.6 / 96.1
75.0 / 81.9
139 / 236
77.7 / 74.7
Appendix
Table 9: Arrival-aware GPU ablation, six-round means. Each cell gives U / T: FleetServe without / with arrival-aware admission under matched execution resources. Tight budgets are 90 ms for alternating phases and bursty arrivals, and 80 ms for multiple budgets. Wide-budget latency is for the 400 ms class. Bursty results use the full measurement window.
Workload
FIFO
EDF
Slack-Gate
FleetServe
Δ goodput (responses/s)
N=8
74.86 / 99.96 / 99.81
99.86 / 99.92 / 99.99
99.97 / 100.00 / 99.90
99.94 / 99.91 / 99.94
−0.03
N=10
98.94 / 97.98 / 96.17
99.94 / 99.92 / 99.98
99.98 / 99.97 / 99.96
99.90 / 98.95 / 99.93
−0.14
N=12
99.77 / 99.75 / 99.58
99.66 / 98.45 / 99.70
99.92 / 99.86 / 99.27
99.89 / 97.99 / 99.79
+1.86
N=14
96.30 / 96.01 / 97.49
91.05 / 88.72 / 96.71
98.86 / 99.26 / 97.93
95.82 / 98.22 / 99.48
+5.34
N=15
81.19 / 81.36 / 82.35
87.00 / 90.61 / 91.35
95.88 / 97.28 / 95.88
96.85 / 97.14 / 99.19
+10.84
N=16
50.14 / 49.98 / 52.38
92.20 / 91.23 / 88.00
95.69 / 95.68 / 94.77
96.12 / 96.89 / 97.59
+10.62
Appendix
Table 10: Scheduling policy comparison under matched execution resources. SLO attainment (%) is reported as R1 / R2 / R3. The N rows use alternating task phases; bursty arrivals use the full [30,210) s send window. Δ goodput is the difference between three-round means (FleetServe − Slack-Gate; on-time responses/s).