NPU Offloading of a Frozen Visual Encoder for Robot Policy Training
Authors: Hyojun Yun, Seungjae Won, Hyungpil Moon
Organizations: Department of Mechanical Engineering, Sungkyunkwan University, Suwon, Korea · Department of Intelligent Robotics, Sungkyunkwan University, Suwon, Korea
When a robot policy is trained for a new task or dataset, its visual encoder can be frozen and only its action generation module trained, reducing training cost. Freezing removes the encoder's backward pass, but its forward pass must still run at every training step because the input images change, so it keeps consuming GPU compute. We therefore ask whether moving this computation to a low power AI accelerator such as an NPU can reduce total energy despite the added data transfer and longer training time, and how it affects policy performance. We built an asynchronous training pipeline that uses both a GPU and an NPU for the AR-Actor specialist. The frozen visual encoder runs in A8W8 INT8 on a Mobilint Aries2 NPU, while the FP32 action expert is trained on an NVIDIA GeForce RTX 5060 Ti GPU. We compared a GPU-only baseline with four conditions, L1 to L4, which gradually extend NPU offloading from one to four Transformer encoder layers. Each condition was trained for 30,000 steps with three random seeds. We measured GPU board power for the GPU-only condition and combined GPU and NPU board power for the NPU conditions. Energy per sample decreased by 17.1% in L1, which offloaded ResNet18 and the first encoder layer, and by 27.9% in L4, which offloaded ResNet18 and all four encoder layers. In contrast, training time per sample increased by 15.2% in L1 and 37.7% in L4, and peak allocated GPU memory decreased by 19.8 to 20.7%. The 15 resulting policies were each evaluated with the same 300 environment seeds, for a total of 4,500 simulator rollouts. The combined success rate was 93.33% for GPU-only and 91.44 to 92.89% for the NPU conditions. These results show that NPU offloading of a frozen visual encoder can reduce training energy, but it increases training time and lowers policy success rate by 0.44 to 1.89 percentage points compared with GPU-only training.
Figures & tables
Part
Components
Number of parameters
State
Visual encoder
ResNet18, image projection, four Transformer encoder layers
28,771,520
Frozen
Action expert
State projection, four autoregressive Transformer decoder layers, one parallel decoder layer, and action head
22,743,708
Newly initialized and trained
TABLE I : PARAMETER COUNT AND TRAINING STATE OF EACH POLICY MODULE
Condition
Runs on NPU
Frozen computation remaining on GPU
Trained on GPU
GPU-only
None
Entire visual encoder
Action expert
L1
ResNet18 + projection + encoder 0
Encoders 1–3 + K/V reconstruction
Action expert
L2
L1 range + encoder 1
Encoders 2–3 + K/V reconstruction
Action expert
L3
L2 range + encoder 2
Encoder 3 + K/V reconstruction
Action expert
L4
L3 range + encoder 3
K/V reconstruction + final normalization
Action expert
TABLE II : OFFLOADING BOUNDARIES AND THE COMPUTATION THAT REMAINS ON THE GPU
Fig. 1 : Time breakdown of asynchronous GPU and NPU overlap by condition. While the GPU processes batch N, the NPU runs the frozen visual encoder forward pass for batch N+1 in advance.
Item
Setting
CPU
Intel Core i5-10400 @ 2.90 GHz
GPU
NVIDIA GeForce RTX 5060 Ti 16 GB, driver 580.173.02
TABLE III : HARDWARE, SOFTWARE, AND PRECISION SETTINGS
Mean GPU
Mean NPU
Mean combined
Total training
Energy per
vs.
Time per
Throughput
Condition
board power
board power
board power
time
sample
GPU-only
sample
(W)
(W)
(W)
(min)
(J)
(%)
(ms)
(sample/s)
GPU-only
138.59
—
138.59
26.43
0.91512
—
6.607
151.4
L1
80.85
18.81
99.66
30.46
0.75831
−17.1
7.613
131.3
L2
70.67
18.88
89.55
32.94
0.73671
−19.5
8.224
121.6
L3
62.84
19.08
81.92
34.31
0.70244
−23.2
8.578
116.6
TABLE IV : MEAN BOARD POWER, ENERGY, AND TRAINING TIME
Fig. 2 : Accelerator board energy per sample by offloading boundary. Gray lines connect the conditions measured with the same training seed. Colored points show the mean across the three training seeds.
Fig. 3 : Peak allocated GPU memory by condition. Each bar shows the mean across three training seeds.
Fig. 4 : Simulator policy success rate by condition. Large colored points and the dark line show the success rate over 900 evaluations for each condition. Small open points and gray lines connect checkpoint results from the same training seed. The training seed is shown at the right end of each gray line. Values in parentheses show the difference from GPU-only.
We study how to improve training efficiency on Ascend NPUs. In the NPU training programs we study, framework-level compilation optimizations such as operator fusion are only partially applied, leaving fragmented computation and avoidable execution overheads. Profiling-guided tuning can address these inefficiencies, but it often requires costly trial and error. To make training on NPUs more efficient, we propose TrainBridge, an agentic approach for transferring GPU optimization guidance to NPU training. TrainBridge uses the highly optimized execution produced by mature GPU compilation stacks as a task-specific reference for NPU tuning. It extracts optimization intents from the GPU training execution to guide NPU program rewrites and further refines the rewritten program through iterative profiling. Our GPU-to-NPU migration experiments show that TrainBridge achieves an average 1.7x speedup across four workloads covering LLM pretraining, image classification, and vision-language training, and an average speedup of nearly 2.0x across 27 Modded-NanoGPT speedrun records with highly optimized GPU implementations.
Yupeng Chen, Kunxiang Zhao, Xingjian Wang +6
Shenzhen Loop Area Institute · The Chinese University of Hong Kong, Shenzhen, China · Autokernel
Vision-language-action (VLA) policies repeatedly invoke inference to control robots, making graphics processing unit (GPU) energy a recurring cost of task execution. Reducing energy per inference call, however, may not reduce energy per successful task if numerical errors increase failures or slower inference prolongs execution. We therefore target GPU energy per successful task while preserving task success and keeping the inference-latency increase within 10%. Our approach builds on two observations: quantization sensitivity varies across action classes, model layers, and weights versus activations; and numerical precision changes the workload, shifting favorable GPU operating points. We introduce ActTune, an action-aware framework that connects layer-wise precision allocation with workload-dependent GPU operating-point selection over requested frequency--power-cap pairs. A lightweight decision tree learns its splits and leaf precision configurations directly from configuration action errors, then selects precision before each policy call. The controller forecasts the next workload and applies the selected GPU operating point asynchronously using a lookup table calibrated under a latency budget. A shared resident quantized weight bank enables configuration switching without weight reconstruction or additional policy evaluations. On LIBERO, a benchmark for lifelong robot learning, ActTune improves mean task success by up to 2.3% relative to state of the art. Relative to the original BF16 implementations, it delivers up to 2.02× faster inference and, with GPU operating-point adaptation, reduces energy per successful task by up to 76.8%.
Zou Qingyun, Bin Gao, Wenju Zhao +3
National University of Singapore · Huazhong University of Science and Technology
Vision-Language-Action (VLA) policies commonly run Vision-Language Model (VLM) backbones with billions of parameters at every policy inference, which costs latency and energy. We revisit a decoupled alternative for multi-task manipulation: separate vision and language encoders whose representations condition a compact action head. We run a standardized comparison that varies the vision encoder, the language encoder, and the action head while holding the demonstrations, the training-step budget, the tasks, the evaluation protocol, and the measurement platform fixed, against seven VLA baselines. The resulting Decoupled Embodiment Model (DEM) combines a fine-tuned DINOv3 vision encoder, a frozen NeoBERT language encoder, and a MeanFlow head that generates an action chunk in one forward pass. On 18 RoboCasa tasks evaluated with held-out instruction paraphrases and randomized scenes, DEM reaches 55.6% mean success against 56.9% for GR00T N1.7 and 54.6% for π0.5, and on three real-robot tasks it reaches 66.0% against 68.0% for GR00T N1.7. On the same workstation, DEM needs 6.1,ms per policy forward pass, a maximum throughput of 162.7 policy calls per second, and draws an estimated 2.07,J of GPU energy per call, eight to seventeen times the throughput and six to fifteen times less energy than these VLM-backbone policies. Within this trained-task regime, DEM sits on the observed success--latency--energy frontier and provides a strong, efficient baseline for language-conditioned robot skills.
Xiatao Sun, Chen Liang, Ziyao Zeng +5
Department of Computer Science, Yale University, New Haven, CT, USA. · Peking University, Beijing, China. · Digients, Singapore.