Physical AI requires models that understand spatial relationships and dynamic environments and translate observations into actions under limited onboard compute. We present TuringPAI, a family of models for on-device Physical AI comprising TuringPAI-VL for shared vision-language understanding, TuringPAI-Drive for autonomous driving, and TuringPAI-Embodied for robotic manipulation. TuringPAI-VL is a 0.65B model built on HybridViT-MLQ, which combines convolutional local aggregation with Transformer-based global modeling. Multi-scale Latent Queries integrate spatial features and observation history within the same visual Transformer, providing a compact, structured interface to the language model. Training combines general vision-language data with geometric, cross-view, and temporal supervision. TuringPAI-VL supports general multimodal understanding and improves quantitative geometric perception over Qwen2-VL-2B, while providing a shared initialization for the two downstream VLA models. For autonomous driving, TuringPAI-Drive maintains all evaluated ADE/FDE metrics within 1.2% of a full-token HybridViT baseline using the MLQ interface and achieves comparable aggregate scores in an in-house simulation evaluation. Using XPeng Native Runtime (XNRT) on a single NVIDIA Orin-X SoC, TuringPAI-Drive generates trajectories at 12,Hz in on-vehicle operation. For robotic manipulation, TuringPAI-Embodied achieves a 96.8% average success rate across the four LIBERO suites. These results demonstrate a shared vision-language foundation for driving and robotic manipulation under practical on-device constraints.
Figures & tables
Figure 2: Overview of TuringPAI-VL. (a) HybridViT-MLQ combines convolutional local aggregation with a visual Transformer augmented by latent queries, supplying multi-scale visual features to the language model. (b) General multimodal data are complemented by Physical-AI-oriented tasks, illustrated through spatial reasoning, metric depth, cross-view grounding, and cross-view synchronization examples. Solid arrows denote inference flow; dashed arrows denote training or downstream adaptation. The visual-to-language projector is omitted for clarity.
Figure 3: Multi-scale Latent Queries (MLQ). Current-frame features initialize coarse-to-fine queries, processed with observation-window patches using unified 3D RoPE. Masking preserves patch-to-patch attention while queries aggregate patch and earlier-query features; only updated queries pass to the language model. Rows and columns of the attention mask correspond to query and key positions, respectively. The linear projection used to initialize latent queries is omitted for clarity. Frame labels show temporal order; token counts are schematic.
Figure 4: Composition of the TuringPAI-VL training pool across general understanding, spatial perception and reasoning, and multi-image and temporal understanding.
Configuration
Stage 1 (alignment)
Stage 2 (joint)
Data type
general image–text
image/video–text: grounding,
text-rich, spatial, temporal
Resolution
native dynamic
native dynamic
Objective
image–text alignment
joint + logit distillation
Global batch
2,048
4,096
Peak LR
2.0×10−4
5.0×10−5
Table 1: VLM training setup. Stage 1 trains the encoder and adapter with the language model frozen; Stage 2 trains all parameters. ∗ The H800 GPU-hours indicate total training cost, measured as the number of H800 GPUs multiplied by wall-clock training hours.
Group
Benchmark
Qwen2-VL-2B Base ( Wang et al., 2024a )
InternVL3-1B Base ( Zhu et al., 2025 )
TuringPAI-VL
Grounding & Counting
RefCOCO avg ( Kazemzadeh et al., 2014 ; Yu et al., 2016 )
64.16
77.88
80.91
CountBenchQA ( Paiss et al., 2023 ; Beyer et al., 2024 )
83.57
56.67
69.61
Text-rich Perception
OCRBench V1 ( Liu et al., 2024 )
724
716
727
OCRBench V2 en ( Fu et al., 2025 )
32.59
25.29
32.10
OCRBench V2 cn ( Fu et al., 2025 )
12.79
21.36
29.33
ChartQA test ( Masry et al., 2022 )
54.52
62.92
76.96
Table 2: VLM benchmark results for TuringPAI-VL and public models. Bold and underlined mark the best and second-best among the three compared models. Results for Qwen2-VL-2B Base and InternVL3-1B Base are from our own evaluations.
Category
Benchmark / Metric
Qwen2-VL-2B Base ( Wang et al., 2024a )
InternVL3-1B Base ( Zhu et al., 2025 )
TuringPAI-VL
Geometric Perception
NYU Depth V2 ( Silberman et al., 2012 ) ( AbsRel ↓ )
0.70
0.38
0.146
ETH3D ( Schöps et al., 2017 ) (EPE in px ↓ )
235.57
199.05
193.49
7-Scenes ( Shotton et al., 2013 ) (AUC@30 ∘ ↑ )
2.03
0.35
19.99
3D Object Localization
ARKitScenes ( Baruch et al., 2021 ) ( n=1,090 )
N/V
N/V
33.76
ScanNet ( Dai et al., 2017 ) ( n=910 )
N/V
N/V
39.55
SUN RGB-D ( Song et al., 2015 ) ( n=1,859 )
N/V
N/V
30.12
Table 3: Spatial perception and reasoning results for TuringPAI-VL and two public reference VLMs. Bold and underlined mark the best and second-best valid scores among the displayed models.
Evaluation axis
Metric / suite
Definition
Open loop
ADE / FDE
Lateral and longitudinal displacement errors (m), over the 6 s and 2 s horizons
Yaw error
Mean per-point heading error (rad)
Miss rate
Frames whose lateral endpoint error at 6 s exceeds 0.5 m or whose longitudinal endpoint error exceeds 4.0 m
Driving quality
Compliance
Traffic-rule adherence, e.g., off-center driving and oncoming-lane entry
Comfort
e.g., acceleration jerk, swaying, and abnormal deceleration
Efficiency
e.g., speed-limit shortfall and slow acceleration
Table 4: Driving-VLA evaluation protocol. Open-loop metrics are evaluated frame by frame on the 436k-frame validation split; driving-quality metrics are evaluated on an in-house simulation set. Lower values indicate better performance for all metrics.
6 s horizon
2 s horizon
ADE (m) ↓
FDE (m) ↓
ADE (m) ↓
FDE (m) ↓
Model
lat.
long.
lat.
long.
Yaw ↓
Miss (%) ↓
lat.
long.
lat.
long.
HybridViT
.1591
.9858
.4076
2.7249
.0129
37.58
.0245
.1457
.0686
.3573
HybridViT-MLQ
.1572
.9910
.4031
2.7345
.0128
37.48
.0243
.1469
.0678
.3605
Table 5: Open-loop trajectory prediction performance on the held-out validation set. Results are reported at 2 s and 6 s prediction horizons. ADE and FDE are measured in meters, yaw error in radians, and miss rate in percent. ↓ indicates that lower values are better, and bold denotes the best result in each column.
Compliance ↓
Comfort ↓
Navi. ↓
Off-Ctr.
Onc. Ln.
Tra. Lig.
Line X
Avg.
Jerk
Sway
Decel.
Avg.
Avg.
1.009
1.010
0.997
1.004
1.002
1.061
0.977
0.976
1.004
0.991
Efficiency ↓
Safety ↓
Sl. Acc.
Spd. Sh.
Late LC
Long F.
Avg.
Ins. Brk.
Roadside
Risky LC
Rear Risk
Avg.
0.992
0.973
0.996
1.003
0.992
0.998
1.015
0.997
0.962
0.995
Table 6: Per-metric ratios of unnormalized driving-quality costs, with HybridViT as the reference (1.00); lower ratios indicate lower costs for HybridViT-MLQ. Bold indicates ratios below 1.0. Each Avg. is the geometric mean over all subclasses in its category, not only those shown. Navigation reports the aggregate route-following cost.
Figure 5: On-device latency and numerical fidelity of the TuringPAI-Drive VLM backbone on a single NVIDIA Orin-X SoC. (a) GPU-side VLM prefill inference latency across four deployment configurations. The dashed line marks the 83.3 ms period corresponding to 12 Hz. (b) Cosine errors derived from mean and minimum cosine similarities, shown on a logarithmic scale. The dashed line corresponds to cosine similarity 0.99. Lower values are better in both panels. TRT denotes TensorRT 8.6.
Latency
Cosine
Cosine
MAE
RMSE
Max. abs.
Relative L2
Configuration
(ms) ↓
mean ↑
min ↑
mean ↓
mean ↓
worst ↓
mean ↓
TensorRT FP32
343.863
0.999999
0.999999
0.000631
0.001150
0.102601
0.001068
TensorRT FP32+INT8 (QAT)
257.270
0.967537
0.964966
0.155759
0.276348
24.122121
0.253196
TensorRT FP16
180.489
0.678869
0.657518
0.568197
0.852783
125.000000
–
XNRT BF16+INT8 (QAT)
69.773
0.998369
0.998106
0.035446
0.062417
6.312500
0.057103
Table 7: VLM inference latency and numerical fidelity on a single NVIDIA Orin-X SoC. Latency sums GPU-side ViT and LLM times, excluding preprocessing, transfers, DLA execution, and the action head. Outputs are compared with the original model’s PyTorch FP32 reference. QAT: quantization-aware training; –: unreported. Best values are bold.
Figure 6: Integrated Gradients attribution maps for the planned trajectory across seven camera views in two driving scenes: (a) a daytime left turn and (b) a nighttime left turn. For each scene, rows from top to bottom show the original camera images, HybridViT attributions, and HybridViT-MLQ attributions; camera views are labeled above. The visualizations correspond to the final frame of the four-frame input sequence.
Model
Spatial
Object
Goal
LIBERO-10
Overall
OpenPI π0.5 (published)
98.8
98.2
98.0
92.4
96.85
TuringPAI-Embodied (ours)
98.4
99.6
96.0
93.2
96.8
Table 8: LIBERO success rates (%) by suite and overall. TuringPAI-Embodied was evaluated on 500 trials per suite. The π0.5 scores are the published results for the OpenPI pi05_libero checkpoint ( Physical Intelligence, n.d. ) .
Policy
Vision
LLM
Action expert
Sum
OpenPI π0.5 (published)
84.1
220.6
117.5
422.2
TuringPAI-Embodied (ours)
27.3
60.8
80.2
168.3
Table 9: Reference inference-stage timings (ms) on a single NVIDIA Orin-X SoC. TuringPAI-Embodied can run CNN patch embedding on the DLA. To compare the subsequent stages, vision timing starts after patch embedding for both policies. These timings are provided for reference only.
Appendix figures & tables11 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 7: Box-grounded spatial-relation example.
Figure 8: Metric-depth estimation example.
Figure 9: Camera-conditioned 3D object-localization example.
Figure 10: Relative camera-pose example with rotation and translation questions.
Figure 11: Cross-view pixel-correspondence example. The query pixel and its labeled match are shown in separate views.
Figure 12: Cross-view synchronization example for a robotic-manipulation scene. Numbered images identify camera views; the highlighted view indicates the answer.
Figure 13: Tabletop embodied spatial-reasoning example involving object identification, relative position, and proximity.
Figure 14: Example from XPeng’s internal alignment data for traffic-light-state understanding. The left and right columns show front-telephoto and front-camera views, respectively, over four frames sampled at 0.25 s intervals. The prompt asks for each signal’s color and direction and any countdown of 10 s or less.
Figure 15: Example from XPeng’s internal alignment data for cross-view box grounding. Given an object’s 2D bounding box in the front-right view, the task predicts its corresponding box in the rear-right view.
Category
Subclass
Score
Δ Score (%)
Compliance
Off-center driving
1.009
+0.9
Oncoming-lane entry
1.010
+1.0
Traffic-light issues
0.997
−0.3
Waiting-area adherence
0.992
−0.8
Restricted-lane entry
0.999
−0.1
Line crossing
1.004
+0.4
Appendix
Table 10: Per-metric CCES breakdown of HybridViT-MLQ against HybridViT (1.00). Each Score is the ratio of an unnormalized cost to that of HybridViT; values below 1.00 indicate lower costs. Δ Score (%) is computed from displayed Score values as 100(Score−1) . Class averages are in italics.
Figure 16: Integrated Gradients maps for the planned trajectory across seven camera views in five driving scenes (top to bottom: split merge, parked-vehicle bypass, cut-in, car following, and dense urban-bridge traffic). Rows show input images, HybridViT attributions, and HybridViT-MLQ attributions for the final frame of each four-frame input sequence.
Moving beyond the traditional paradigm of adapting internet-pretrained models to physical tasks, we present DM0, an Embodied-Native Vision-Language-Action (VLA) framework designed for Physical AI. Unlike approaches that treat physical grounding as a fine-tuning afterthought, DM0 unifies embodied manipulation and navigation by learning from heterogeneous data sources from the onset. Our methodology follows a comprehensive three-stage pipeline: Pretraining, Mid-Training, and Post-Training. First, we conduct large-scale unified pretraining on the Vision-Language Model (VLM) using diverse corpora--seamlessly integrating web text, autonomous driving scenarios, and embodied interaction logs-to jointly acquire semantic knowledge and physical priors. Subsequently, we build a flow-matching action expert atop the VLM. To reconcile high-level reasoning with low-level control, DM0 employs a hybrid training strategy: for embodied data, gradients from the action expert are not backpropagated to the VLM to preserve generalized representations, while the VLM remains trainable on non-embodied data. Furthermore, we introduce an Embodied Spatial Scaffolding strategy to construct spatial Chain-of-Thought (CoT) reasoning, effectively constraining the action solution space. Experiments on the RoboChallenge benchmark demonstrate that DM0 achieves state-of-the-art performance in both Specialist and Generalist settings on Table30.
Physical AI policies require inference throughout their lifecycle, including model evaluation, cloud reinforcement learning rollout, edge GPU serving, and onboard deployment. Although these settings share the same checkpoint and action semantics, they often rely on separate inference programs. To unify them, we build PhyAI, a Physical AI inference engine with a single runtime that keeps architecture-specific conditioning, solver, cache, and output logic in model adapters while sharing graph execution, kernels, memory management, and parallel services. The same codebase runs vision-language-action (VLA) models and world-action models (WAMs) on single or multiple GPUs across onboard, edge, and cloud deployments. We used the adapter interface to add MiniCPM-Robot on the day of its release. PhyAI achieves 1.40x-4.65x speedups over the official implementations of pi0, pi0.5, GR00T N1.7, and MiniCPM-Robot. On Cosmos3-Nano-Policy-DROID it reduces latency from 2.46 to 1.18 s on eight H20 GPUs (CFG=2, TP=4), a 2.08x speedup. Specialized runtimes remain faster in several configurations, so our goal is one runtime with competitive latency rather than the fastest result in every case. Detailed profiles reveal why different models need different execution policies: on a Hopper-series GPU at batch size one, the pi0.5 action expert accounts for 8.8% of FLOPs but 57.2% of latency; at batch size 32 its share drops to 13.5% and throughput reaches about 100 samples/s. Cosmos3 remains generation-dominated and gains only 14.3% throughput as batch size increases from 1 to 16. We further introduce the control-time Roofline, which distinguishes inference-bound from environment-bound control; the measured pi0.5 points on four LIBERO suites are environment-bound while Cosmos3 stays inference-bound. Code and benchmarks: https://github.com/mingti-org/phyai.
Chenghua Wang, Daliang Xu, Dongqi Cai +21
1Beijing University of Posts and Telecommunications · 2Nanjing University · 6ModelBest +3
Physical AI aims to build embodied agents that perceive the world, understand and reason about it, and decide how to act. Yet the field has focused primarily on the last component: the action model that maps observations to low-level controls. The prevailing training recipe can erode the perceptual and reasoning capabilities needed for robust behavior, leaving even strong action models vulnerable to scene perturbations and long-horizon tasks. We introduce HarnessPAI, a model- and embodiment-agnostic Harness framework for Physical AI that treats code as the executable and evolvable interface that organizes the underlying action primitive. The framework separates two timescales: within a rollout, it executes open-loop at the program level, with a fixed program guiding and checking execution; across rollouts, it evolves closed-loop, using execution feedback to revise the program and distill failures into reusable skills. Across desktop robot arms, household robots, a robot vacuum, and a legged walking agent, HarnessPAI demonstrates consistent improvements against baselines. Beyond the harness, the converged program also acts as a low-cost, reliable source of expert data: it generates training data for fine-tuning VLA and WAM models on tasks where it is hard for these models to obtain successful trajectories on their own. Our results suggest that the frontier of Physical AI depends not only on stronger action models, but also on executable harnesses that integrate perception, task understanding and reasoning, and action execution into a unified, verifiable, and feedback-driven system. Website: https://darwin-agent.github.io/HarnessPAI