LEAP-NBV: Lightweight Edge Active-Perception for Foundation-Model Next-Best-View Planning

Boxun Hu; Jiawei Ge; Axel Krieger; Peng Wang; Tinoosh Mohsenin

cs.AISep 21, 2026

LEAP-NBV: Lightweight Edge Active-Perception for Foundation-Model Next-Best-View Planning

Authors: Boxun Hu, Jiawei Ge, Axel Krieger, Peng Wang, Tinoosh Mohsenin

Organizations: Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD 21218, USA · US Army Research Laboratory, Aberdeen Proving Ground, MD 21005, USA

Abstract

Foundation models are endowing autonomous systems with greater intelligence, enabling a more comprehensive understanding of the environment through visual perception. A representative example is Human Mesh Recovery (HMR), which provides useful estimates of a target's 3D pose and shape that can benefit tactical missions. However, the size and power demands of such models make them difficult to run on edge platforms and limit their real-time performance, undermining the requirements of tactical edge deployment - especially for active perception, where a mobile robot must plan its next-best view on-board and cannot offload computation under contested communications. We present LEAP-NBV, a lightweight active-perception framework that runs foundation-model-driven Next-Best-View (NBV) planning on-board an edge device. To this end, we distill a family of large HMR teachers, each into a compact 32M student, with an offline mesh objective, then quantize the vision encoder to FP16 and characterize its on-device accuracy and latency. Within an occlusion-aware active perception loop, we evaluate all configurations on the same held-out benchmark and deploy the end-to-end pipeline on an NVIDIA Jetson Xavier NX, reporting measured on-device latency and energy. Distillation recovers 6-7 mm of Procrustes-aligned mean per-vertex position error (PA-MPVPE) over the undistilled student on the test set. Selecting the edge-optimal compression model brings the HMR engine to ~12 ms at a small accuracy cost and runs the full closed loop at 3.6 FPS and 2.6 J per frame, achieving a 2.0x speedup and 3.0x lower energy than the uncompressed model while nearly matching downstream task quality.

Figures & tables

Fig. 1 : LEAP-NBV overview. (A) Compression. The ViT backbone dominates the SMPLer-X HMR network ( $304$ M, $93\%$ ); LEAP-NBV distills it into a ViT-S student ( $327{\rightarrow}32$ M) and quantizes the backbone and heads to FP16, keeping the SMPL-X tail in FP32. (B) On-board pipeline. The compressed model drives the active-perception loop (detection, HMR, geometry, NBV planning) on a Jetson Xavier NX at $276$ ms/frame and $2.58$ J/frame, $2.0\times$ faster and $3.0\times$ lower energy than the FP32 model.

Fig. 2 : Cross-scale mesh distillation. A frozen teacher (right) is run once to precompute a target mesh $V_{T}$ for every training crop; the trainable student (left) predicts $V_{S}$ from the same crop and is optimized to match it by the centroid-relative per-vertex loss (Eq. 4 ), which back-propagates to the student only. Distillation collapses each $103$ – $687$ M teacher into the same $32$ M student ( $3$ – $21\times$ smaller), shrinking both the ViT backbone and the neck/heads while leaving the parameter-free SMPL-X tail unchanged.

Fig. 3 : On-device precision map. Precision of the deployed FP16 engine over the $66$ weighted layers: the $6$ patch-embed convolutions, the $49$ encoder linears, the upsample, and the $10$ regressor GEMMs (body/hand pose and the eight parameter heads) all run in FP16, while the unweighted normalization, softmax, activation ops and the parameter-free SMPL-X tail are kept in FP32. This uniform half-precision assignment halves the engine’s memory footprint.

Fig. 4 : Compression ladder: accuracy and model size. For each model family, bars report single-view PA-MPVPE (left axis; lower is better) and dots report model size (right axis, log scale, with dot area also scaled to size) across three variants: the original FP32 teacher, the distilled s32 student, and that student quantized to FP16. Distillation collapses every teacher into a $32$ M student at a small accuracy cost, and FP16 quantization shrinks it further. The dashed and dotted lines mark the baseline s32’s accuracy and size; s32 has no teacher, so only its original is shown.

Fig. 5 : Accuracy–latency Pareto on the Jetson Xavier NX. The full compression ladder on a log latency axis: FP32 teachers (squares), distilled s32 students (diamonds), and FP16-quantized students (circles), with the Pareto frontier; the two insets zoom the distilled ( $\sim$ 53 ms) and quantized-FP16 ( $\sim$ 12 ms) clusters. Distillation and FP16 quantization move the model from the slow FP32 teachers to real-time latency at a small accuracy cost. The l32 model is ringed through both compression steps and we deploy the l32-distilled student in FP16 (circled).

model	$D$	naive FP16	original FP32
s32	384	62.9	62.9
b32	768	56.6	56.6
l32	1024	48.5	48.5
h32	1280	52.4	47.6
SX-h	1280	128.8	47.3

TABLE I : The FP16 overflow is width-dependent. Naive FP16 casts the original model directly to half precision, uniformly across all layers. Under this cast, single-view PA-MPVPE (mm) diverges from the FP32 reference only at ViT-H width (

D{=}1280

); every model with

D{\leq}1024

matches FP32, so the deployed ViT-S student (

D{=}384

) stays well clear of the overflow.

Fig. 6 : End-to-end power over one frame. Board power (VDD_IN) and the compute rail across the stages of a single frame on the Jetson Xavier NX, for the uncompressed l32 pipeline (top) and the deployed FP16 student (bottom). The entire difference is the highlighted HMR stage, which the deployed model runs in $12$ ms rather than $257$ ms; every other stage is model-independent and matches. The deployed pipeline finishes a frame in $276$ ,ms vs. $550$ ,ms ( $2.0\times$ faster) at $2.58$ ,J vs. $7.83$ ,J ( $3.0\times$ lower energy)

	model	Coverage	Area	PA-MPVPE
Original	s32	61.9 $\to$ 87.9	11.5 $\to$ 27.5	62.1 $\to$ 55.0
	b32	61.9 $\to$ 87.7	11.5 $\to$ 28.1	54.8 $\to$ 48.0
	l32	61.8 $\to$ 90.2	11.5 $\to$ 30.2	49.0 $\to$ 44.5
	h32	62.2 $\to$ 88.7	11.4 $\to$ 29.4	46.5 $\to$ 42.4
	SX-h	62.0 $\to$ 88.1	11.5 $\to$ 28.7	48.2 $\to$ 42.6
Deployed	s32 $\leftarrow$ b32	62.0 $\to$ 88.3	11.5 $\to$ 28.1	61.7 $\to$ 54.0

TABLE II : Task-level performance of the NBV loop (

441

frames, real detector); each cell reports the metric as initial view

\to

after the NBV move . Columns: body coverage (%), reconstructed area (

\times 10^{3}

px), and closed-loop PA-MPVPE (mm); “Deployed” rows are distilled students in FP16. After the move, coverage and area closely match between the deployed students and their FP32 teachers, so the 2D task metrics are preserved under

30

–

60\times

compression, while PA-MPVPE keeps a small, constant mesh-quality offset with a similar gain. The deployed configuration ( s32

\leftarrow

l32 ) and its FP32 teacher ( l32 ) are shaded for direct comparison.

Active Perception

Active perception studies how agents choose actions and viewpoints to gather useful sensory evidence while acting under uncertainty.

See all 105 papers

EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition
Sep 13, 2026He Zhang, Siyu Yuan, Siyu Liu +2Human Activity Recognition Wearable Sensing
Northwestern Polytechnical University Xi’an, Shaanxi, China · RPTU University Kaiserslautern-Landau Kaiserslautern, Rheinland Pfalz, Germany · Xi’an University of Technology Xi’an, Shaanxi, China
EdgeVLN: Runtime-Aware Deployment Ready Quantized Vision Language Navigation Model
Sep 28, 2026Rithvik Jonna, Man Namgung, Aakash Gurram +1Visual Navigation VLM Quantization
Department of Electrical and Computer Engineering Johns Hopkins University · Laboratory for Computational Sensing and Robotics Johns Hopkins University
Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies
Mar 16, 2026Utkarsh Grover, Ravi Ranjan, Mingyang Mao +9Robot Foundation Models Edge Computing
University of South Florida, USA · Florida International University, USA · Johns Hopkins University, USA +2

Sep 13, 2026J/K move · Enter open · S save

EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition

He Zhang, Siyu Yuan, Siyu Liu, Sizhen Bian+1

Northwestern Polytechnical University Xi’an, Shaanxi, China · RPTU University Kaiserslautern-Landau Kaiserslautern, Rheinland Pfalz, Germany · Xi’an University of Technology Xi’an, Shaanxi, China

Human Activity Recognition Wearable Sensing Edge ML Domain Generalization Disentangled Representation Learning

Open paper AskPDF arXiv

Sensor-based human activity recognition (HAR) is fundamental to ubiquitous and wearable computing, yet existing foundation models are largely designed for cloud-scale deployment and struggle with real-world sensing shifts, including unseen users, devices, sampling rates, and sensor placements. We present EdgeHAR, an edge-native compact sensor foundation model designed for wearable intelligence. Unlike conventional models that entangle activity knowledge with acquisition variations, EdgeHAR learns transferable representations by factorizing sensor signals into three latent codes: an (i)Activity-Semantic Code capturing reusable activity knowledge, a (ii)Motion-Dynamics Code modeling temporal patterns, and an (iii)Acquisition-Context Code representing sensor-specific variations. This disentangled design enables efficient adaptation to new users, devices, placements, and activity classes with limited target-domain data. By incorporating lightweight adaptation modules, EdgeHAR achieves foundation-model-level transferability while satisfying edge constraints in computation, memory, latency, and privacy. Experiments across heterogeneous HAR datasets demonstrate that EdgeHAR maintains competitive recognition performance under distribution shifts with substantially reduced deployment cost. EdgeHAR establishes a practical paradigm for compact, edge-first foundation models for ubiquitous sensing systems.

Abstract

Active Perception

Explore similar work

EdgeHAR: An Edge-Native Compact Sensor Foundation Model for Human Activity Recognition

EdgeVLN: Runtime-Aware Deployment Ready Quantized Vision Language Navigation Model

Embodied Foundation Models at the Edge: A Survey of Deployment Constraints and Mitigation Strategies