Fluoroscopy-Constrained Magnetic Robot Control via Zernike-Based Field Modeling and Nonlinear MPC
Authors: Xinhao Chen, Hongkun Yao, Anuruddha Bhattacharjee, Suraj Raval, Lamar O. Mair, Yancy Diaz-Mercado, Axel Krieger
Organizations: Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218, USA. · Department of Mechanical Engineering, University of Maryland, College Park, MD 20742, USA. · Division of Magnetic Manipulation and Particle Research, Weinberg Medical Physics, MD 20852, USA.
Magnetic actuation enables surgical robots to navigate complex anatomical pathways while reducing tissue trauma and improving surgical precision. However, clinical deployment is limited by the challenges of controlling such systems under fluoroscopic imaging, which provides low frame rate and noisy pose feedback. This paper presents a control framework that remains accurate and stable under such conditions by combining a nonlinear model predictive control (NMPC) framework that directly outputs coil currents, an analytically differentiable magnetic field model based on Zernike polynomials, and a Kalman filter to estimate the robot state. Experimental validation is conducted with two magnetic robots in a 3D-printed fluid workspace and a spine phantom replicating drug delivery in the epidural space. Results show the proposed control method remains highly accurate when feedback is downsampled to 3 Hz with added Gaussian noise (sigma = 2 mm), mimicking clinical fluoroscopy. In the spine phantom experiments, the proposed method successfully executed a drug delivery trajectory with a root mean square (RMS) position error of 1.18 mm while maintaining safe clearance from critical anatomical boundaries.
Figures & tables
Fig. 1: (a) Electromagnetic coil system used in this study. The experimental workspace (yellow square) is centered between the coil stacks. (b) Spine phantom workspace imaged with an optical camera (left) and fluoroscopy X-rays (right). The robot agent is visible in both. (c) Robotic agents evaluated, including a drug-delivery capsule that can be filled with liquid medication.
Fig. 2: (a) The spine structure in sagittal (longitudinal) plane. (b) The spine structure in transverse (horizontal) plane. [ 29 ]
Fig. 3: (a) CAD model of a human vertebral segment sectioned in the sagittal plane. (b) Spine phantom fabricated from ABS, agarose gel, and a glycerin–water solution. The epidural space is indicated by purple lines.
Fig. 4: (a) Mean absolute error (MAE) and R2 score of Zernike model with different polynomial order. (b) FEA-computed magnetic-field magnitude for a small coil at unit current (coil located at the bottom of the figure). (c) Absolute fitting-error distribution for the Zernike model under the same condition.
Fig. 5: Flow chart of the proposed control method
Fig. 6: (a) The workspace for the magnet robot. The purple line shows the desired trajectory. (b) Robot actual and NMPC-predicted trajectories.
Fig. 7: (a) RMS position errors under different tracking rates. (b) RMS orientation errors under different tracking rates.
Fig. 8: (a) RMS Position errors under different tracking noise levels. (b) RMS Orientation errors under different tracking noise levels.
Fig. 9: (a) RMS position errors under different tracking rates with fixed noise level. (b) RMS orientation errors under different tracking rates with fixed noise level. (c) Violin plot at 3Hz with fixed noise level. All tests has p<10−3 , denoted by ***.
Fig. 10: Results of spine-phantom experiment. The green curve denotes the desired trajectory; the red curve shows the actual robot trajectory; purple lines delineate the epidural space’s boundaries.
Autonomous robots can increase uptime and reduce human exposure in Big Science facilities, but strong magnetic fields needed for their operation corrupt sensors and induce pose-dependent mechanical wrenches that destabilize robots and challenge conventional reactive controllers. This paper presents a control framework for modeling, estimating, and dynamically compensating for spatially varying magnetic wrenches acting on legged robots to improve robustness in these fields. We introduce a custom physics plugin for the MuJoCo simulator to model magnetic forces on rigid-body elements, alongside an inverse field-estimation framework to infer the latent magnetic field directly from quadruped dynamic responses and any number of sensor readings. Furthermore, we develop a Magnet-Aware Model Predictive Control (MPC) and Whole-Body Control (WBC) architecture that predicts and counteracts magnetic perturbations during locomotion to increase the range of magnetic fields in which the robot can operate. The effectiveness of the framework is validated through both simulation and physical hardware experiments. We show our method increases the the maximum rejectable disturbances from magnetic field in the force space by a factor of 2.5 and and between 1.6-2 times in torque space compared to a non-compensated system. Within the proposed magnetic field, this constitutes an increase in the area in which the robot can operate by 29,34% of which 10,84% would have previously caused an immediate collapse to non-compensated controllers.
J. Playan Garai, S. B. Djuve, C. McGreavy +1
Technical University of Munich (TUM), Munich, Germany · Conseil Europ´een pour la Recherche Nucl´eaire (CERN), BE/CEM/MRO, Geneva, Switzerland
Magnetic navigation provides contact-free and line-of-sight-independent feedback for robotic systems, yet existing approaches typically rely on explicit pose estimation or direct use of raw magnetic measurements, making accurate control susceptible to modeling errors, measurement noise, and disturbances. This work presents MagServo, a hierarchical learning-based framework for robust 6-DoF magnetic servoing directly using the learned latent magnetic feature. MagServo learns uncertainty-resilient magnetic representations through masked reconstruction and captures state-dependent interaction dynamics between robot motion and latent magnetic transitions without analytical magnetic models or explicit Jacobian supervision. Based on the learned dynamics, a hierarchical controller combines nonlinear model predictive control for coarse approach with local Jacobian inversion for precise fine regulation. Extensive physical experiments demonstrate submillimeter and subdegree accuracy, achieving mean terminal errors of 0.386 mm and 0.479 degree for 6-DoF pose reaching. MagServo further outperforms a localization-based control baseline in complex trajectory tracking and maintains robust performance under unseen magnetic-source configurations without retraining. A supplementary video of the real-robot experiments is available at https://youtu.be/rZt1NUP1Mr0.
Yuhan Tan, Yameng Zhang, Pei Liu +2
Department of Bioengineering, Imperial College London, UK. · Department of Mechanical Engineering, The University of Hong Kong, Hong Kong SAR. · School of Computation, Information and Technology, Technical University of Munich, Germany.
Electromagnetic navigation systems (eMNS) enable a number of magnetically guided surgical procedures. A challenge in magnetically manipulating surgical tools is that the effective workspace of an eMNS is often severely constrained by power and thermal limits. We show that system-level control design significantly expands this workspace by reducing the currents needed to achieve a desired motion. We identified five key system approaches that enable this expansion: (i) motion-centric torque/force objectives, (ii) energy-optimal current allocation, (iii) real-time pose estimation, (iv) dynamic feedback, and (v) high-bandwidth eMNS components. As a result, we stabilize a 3D inverted pendulum on an eight-coil OctoMag eMNS with significantly lower currents (0.1-0.2 A vs. 8-14 A), by replacing a field-centric field-alignment strategy with a motion-centric torque/force-based approach. We generalize to multi-agent control by simultaneously stabilizing two inverted pendulums within a shared workspace, exploiting magnetic-field nonlinearity and coil redundancy for independent actuation. A structured analysis compares the electromagnetic workspaces of both paradigms and examines current-allocation strategies that map motion objectives to coil currents. Cross-platform evaluation of the clinically oriented Navion eMNS further demonstrates substantial workspace expansion by maintaining stable balancing at distances up to 50 cm from the coils. The results demonstrate that feedback is a practical path to scalable, efficient, and clinically relevant magnetic manipulation.
Jasan Zughaibi, Denis von Arx, Maurus Derungs +5
Multi-Scale Robotics Lab, ETH Zurich, 8092 Zurich, Switzerland · Learning and Dynamical Systems Group, Max Planck Institute for Intelligent Systems, 72076 Tübingen, Germany