Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin

Zerui Kang; Yishen Lim; Zhouyou Gu; Seungnyun Kim; Seung-Woo Ko; Tony Q. S. Quek; Jihong Park

cs.ROOct 5, 2026

Demo: Vision-Language Model-Guided Online Calibration of an Electromagnetic Digital Twin

Authors: Zerui Kang, Yishen Lim, Zhouyou Gu, Seungnyun Kim, Seung-Woo Ko, Tony Q. S. Quek, Jihong Park

Organizations: Singapore University of Technology and Design Singapore, Singapore · Inha University Incheon, Republic of Korea

Abstract

An electromagnetic (EM) digital twin gives mobile robots wireless situational awareness but depends on material conductivities that change with the environment. Online calibration faces initialization sensitivity and measurement travel costs. We demonstrate a vision-language model (VLM)-guided framework using a Unitree G1 robot and NVIDIA Sionna, with two VLM calls: material classification maps visible materials through ITU-R P.2040 to conductivity priors for Sionna's gradient descent on accumulated received signal strength (RSS) measurements; waypoint planning selects the next measurement location online using residual RSS calibration error and image coverage. In a real indoor scenario, the framework achieves a normalized mean absolute conductivity error of $1.74\times10^{-4}$ within 20 m of travel; random initialization never converges, while random waypoints require over twice the travel.

Figures & tables

Figure 1 . The proposed online calibration framework. The proposed system consists of a VLM, a mobile robot, and a differentiable ray tracing engine. A waypoint-planning call selects the next waypoint from the calibration error and past image coverage, and a material-classification call maps the camera image to initial conductivity values for inverse ray tracing.

Panel of Figure 2 — Figure 2 . Conductivity error and radio maps for the planning and initialization combinations. Two subfigures. The left one plots NMAE against traveled distance for the four combinations of VLM or random waypoint planning and VLM or random initialization; only the VLM-initialized runs converge, and the VLM-planned run converges with the least travel. The right one compares the radio maps calibrated by the VLM-planned system and the random baseline.

Explore similar work

TRACE: Learning to Self-Calibrate Wireless Digital Twins from ISAC Measurements
Sep 26, 2026Saad Masrur, Saeed R. Khosravirad, Ismail GuvencDigital Twins Integrated Sensing And Communication
Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC · Radio Systems Research, Nokia Bell Labs, Murray Hill, NJ, USA
Trajectory Design and Budgeted Querying for Digital Twin Calibration
Aug 9, 2026Vladyslava Spitkovska, Dmytro KuzmenkoTrajectory Generation Twins
Department of Mathematics, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine · Department of Multimedia Systems, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine · Department of Computer Science, University of Turin, Turin, Italy
Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models
Apr 22, 2026Shelly Francis-Meretzki, Mirco Mutti, Yaniv Romano +1Temporal Difference Photometric Supervision
1Technion - Israel Institute of Technology.

Sep 26, 2026, cs.AIJ/K move · Enter open · S save

TRACE: Learning to Self-Calibrate Wireless Digital Twins from ISAC Measurements

Saad Masrur, Saeed R. Khosravirad, Ismail Guvenc

Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC · Radio Systems Research, Nokia Bell Labs, Murray Hill, NJ, USA

Digital Twins Integrated Sensing And Communication Propagation Mean Squared Error Measures

Open paper AskPDF arXiv

Wireless digital twins (DTs) rely on 3D environment models to predict radio propagation and support wireless-network decisions, yet these models are often initialized from imperfect 3D maps. Errors in building position, height, footprint, and orientation can therefore cause a high-fidelity propagation engine to simulate the wrong physical environment. In this paper, we study how a deployed wireless network can repair an existing DT using its own radio frequency (RF) measurements. In particular, we introduce Twin Residual Alignment and Calibration Engine (TRACE), a physics-grounded learning-based self-calibration framework that treats twin maintenance as residual alignment between the physical world and the current DT. Using the same sensing configuration as the physical measurements, TRACE ray-traces the current DT, coherently backprojects the measured and simulated RF onto a common world grid, and extracts the same local region around each building's current DT position. A multi-view corrector then fuses evidence across sensing nodes and neighboring buildings to predict a gated six-parameter correction per building, without relying on absolute layout or sensor ordering, and supports iterative correction through re-rendering. On 5,400 held-out samples from unseen simulated scenes at 28 GHz, TRACE reduces 3D position RMSE from 2.202 m to 0.302 m and yaw RMSE from 4.978° to 0.894°, outperforming ViT and U-Net baselines under changes in layout, building count, sensing-node count, and SNR. On measured 28 GHz RF data from the NIST outdoor courtyard, a model trained only on synthetic RF reduces mean planar wall-position error from 1.00 m to 7.8 cm, without measured-data fine-tuning or geometric labels. These results show that the discrepancy between measured and twin-rendered RF can serve as a learning signal for repairing a wireless DT.

Abstract

Explore similar work

TRACE: Learning to Self-Calibrate Wireless Digital Twins from ISAC Measurements

Trajectory Design and Budgeted Querying for Digital Twin Calibration

Temporal Difference Calibration in Sequential Tasks: Application to Vision-Language-Action Models