FloVMos: Optical Flow-based Medical Video Mosaicking
Authors: Jinyang Liu, Sandesh Ghimire, Chaman Singh, Jennifer Dy, Milind Rajadhyaksha, Dana H. Brooks, Octavia Camps, Kivanc Kose
Organizations: Electrical and Computer Engineering, Northeastern University, 360 Huntington Ave, Boston, 02115, MA, USA. · Dermatology Service, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, 10065, NY, USA.
Biomedical imaging modalities often require a trade-off among resolution, field of view (FOV), and acquisition speed. Video mosaicking offers a strategy to overcome this limitation by computationally stitching sequential high-resolution frames into a wide-FOV composite. However, existing methods struggle with non-rigid deformations, and modality-specific artifacts arising in clinical and research imaging. Here, we present FloVMos, a generalizable, optical-flow-based deep learning framework for real-time video mosaicking across diverse biomedical imaging modalities. FloVMos achieves robust, pixel-level registration by fine-tuning an optical flow model on synthetic training data with ground-truth deformation fields. We introduce a pipeline for generating this training data, simulating realistic tissue motion and imaging distortions from existing mosaics or raw videos. Our automated synthetic data generation and optical flow model training based on this data allow users to adapt FloVMos to different imaging modalities. To demonstrate this, we applied FloVMos to seven diverse imaging modalities: reflection confocal microscopy, open-top light-sheet microscopy, fetoscopy, laparoscopy, dermoscopy, sparse spectral microscopy, and endoscopy. FloVMos outperforms conventional baselines in accuracy, robustness, and speed for all the tested modalities. This adaptable and training-efficient framework enables large-area visualization with real-time performance and may support broader use of video-based biomedical imaging in research and clinical workflows.
Figures & tables
Figure 1: An overview of the FloVMos videomosaicking pipeline highlighting its four key steps: (i) data acquisition, (ii) image registration using a fine-tuned optical flow estimator (a deep neural network), (iii) real-time image warping, and (iv) image alignment and blending.
Figure 2: Overview of FloVMos : a The first step in training involves the acquisition of training data. We have developed two approaches for easy, rapid, and automated acquisition of the required training data. In the first, when sample mosaics are easily available, we simply simulate videos by following a randomized but known path through the mosaic and then adding noise and distortion, as illustrated in the figure. In the second, when we only have videos available, we randomly choose or simulate pairs of images from the video, again with known relationships between the elements of each pair, and add noise and distortion. In both cases, we can thus assemble a collection of labeled pairs of images to use as training data. b We fine-tune the optical flow neural network using the annotated training data. c After training, we apply the DNN to newly acquired (or “test”) videos. The video frames are registered using DNN-predicted optical flow and then assembled into a mosaic by our graph-cuts based stitching algorithm. appending each registered image to the growing mosaic.
Figure 3: Results of applying FloVMos to different imaging modalities: Top: FloVMos qualitative results are presented for eight cases from seven different imaging modalities: (a) Fetoscopy; (b) Dermoscopy; (c) real RCM; (d) SSM; (e) simulated RCM; (f) Endoscopy; (g) Laparoscopy) and (h) videos simulated from an OTLS mosaic. The red circle / box in each image shows the location of the first frame in the video. (We use a red circle for the fetoscopy video to match the actual FOV of the device; when we use FloVMos to create a mosaic, the frames are stitched onto a black background.) Bottom: Histograms of distances (in pixels) between image feature point pairs in the source mosaics or video frames and their matches in the resulting FloVMos mosaic.
Figure 4: Quantitative assessment of accuracy and speed of FloVMos in comparison to selected existing videomosaicking algorithms on the simulated RCM videos: (a) Comparison of accuracy of FloVMos and three comparison algorithms: we compare accuracy using both the Structural Similarity Index Metric (SSIM, left panel) and Learned Perceptual Image Patch Similarity (LPIPS, middle panel) of FloVMos to the Parallax, As-Projective-As-Possible (APAP) [ 17 ] , and Automated Video-Mosaicking (AVM) algorithms. Note that for SSIM, higher is better, while for LPIPS, lower is better, as indicated by the up and down arrows in the image titles. The right panel reports on mosaicking speed in frames per second (FPS) for the same four algorithms. We show box plots for the full results with 800 simulated RCM videos. The horizontal line inside the box represents the median (50th percentile) of the data. The edges of the box denote the first quartile ( 25th percentile) and the third quartile (75th percentile). (b) Comparison among the same four algorithms in terms of registration error as a function of increased degree of non-rigid deformation. Colors represent the different algorithms, as indicated in the legend. Each simulated image pair is shown as a dot on the plot. The degree of (artificially induced) non-rigid registration is quantified as explained in the text. (c) Comparison among the same four algorithms in terms of mosaicking speed in frames per second. Results are shown in frames per second for videos consisting of 5, 10, 20, and 50 frames, as shown on the horizontal axis. The different algorithms are shown by the color of the box, as indicated in the legend. Again results are shown as box plots for all the synthesized RCM videos, using the same box plot format as in the top panel.
Figure 5: Stitching results obtained using the base optical flow model on the video sequences presented in Fig. 3 . The left two columns illustrate modalities characterized by large featureless black or white regions within frames (fetoscopy, OTLS, SSM, and RCM), whereas the right column depicts modalities visually more akin to conventional RGB imagery. Modalities featuring substantial black or white regions often exhibit failures in registration and subsequent stitching. Modalities resembling standard RGB images generally demonstrate improved registration performance even without fine-tuning; however, registration failures still occur. For example, incorrect alignment of the last several frames is evident in (e) endoscopy and (f) dermoscopy, and less accurate optical flow predictions result in noticeable artifacts in the final mosaic, as exemplified in (e) laparoscopy.
Medical image registration aligns scans across time, patients, or imaging modalities to track anatomy and support diagnosis, treatment, and image-guided procedures.
Deep learning has shown significant value in medical image registration for motion correction; however, current techniques are either limited by the type and range of motion they can handle or require iterative inference and/or retraining for new imaging data. To address these limitations, we introduce UniMo, a Unified Motion Correction framework that uses deep neural networks to correct various types of motion in medical imaging. UniMo uses an alternating optimization scheme with a unified loss function to train an integrated model of 1) an equivariant neural network for global rigid motion correction and 2) an encoder-decoder network for local deformations. It features a geometric deformation augmenter that 1) enhances the robustness of global motion correction by addressing local deformations, whether caused by non-rigid motion or geometric distortions, and 2) generates augmented data to improve training. As a hybrid model that uses both image intensities and shapes, UniMo is robust to appearance variations and generalizes to various imaging modalities without retraining. We trained and tested UniMo for motion tracking in fetal magnetic resonance imaging, which is challenging due to 1) both large rigid and non-rigid motion and 2) large variations in image appearance. We then tested the trained model, without retraining, on three public datasets: MedMNIST, lung CT, and BraTS. UniMo surpassed existing motion correction methods in accuracy and, notably, enabled one-time training on a single modality while maintaining high stability and adaptability across multiple unseen imaging datasets. By offering a unified solution to motion correction, UniMo marks a significant advance in challenging applications with a mixture of bulk motion and local deformations. Code is available at https://github.com/IntelligentImaging/UNIMO
Jian Wang, Razieh Faghihpirayesh, Danny Joca +2
Boston Children’s Hospital and Harvard Medical School, Boston, MA, USA · Department of Electrical and Computer Engineering at Northeastern University and also with the Department of Radiology at Boston Children’s Hospital, Boston, MA, USA · Department of Radiology at Boston Children’s Hospital and Harvard Medical School, Boston, MA, USA +2
Video virtual try-on aims to transfer a target garment onto a moving person across video frames. Current methods rely on human parsing masks or pose keypoints that frequently fail under large motions and occlusions, causing boundary artifacts and temporal inconsistency. A further limitation is that most approaches rely solely on attention mechanisms for temporal modeling, providing no explicit motion supervision. We propose FlowVVTON, a mask-free framework that eliminates parsing mask dependency entirely. Optical flow is used solely as a training-time supervision signal: a flow-warped latent loss, applied across all layers of the generation model, enforces multi-scale temporal consistency by aligning adjacent-frame features under explicit physical motion constraints. A two-stage training strategy establishes mask-free spatial alignment before introducing flow-guided temporal supervision. Experiments on TikTokDress show that FlowVVTON outperforms baselines by substantial margins, particularly in temporal consistency (5.7× VFID-R improvement over SwiftTry), while requiring no segmentation masks, pose keypoints, or region annotations at any stage.
Motion perception is fundamental to computer vision and physical AI, and plays a central role in robotics, autonomous vehicles, human-computer interaction, and video compression. Traditional approaches such as background subtraction, frame-differencing and optical flow algorithms often struggle in complex or dynamic scenes, typically sacrificing spatial resolution for computational tractability. Here we introduce MoViD (Motion estimation via Virtual dispersion and phase Detection), a physics-inspired algorithm grounded in the principles that govern photonic time stretch - a highly successful ultrafast realtime data acquisition platform. Drawing inspiration from the optical physics therein, MoViD treats video as a collection of optical time series waveforms associated with pixels. Motion of an object through a pixel broadens the time series spectrum and, after group delay dispersion, induces a temporal phase that exists only if there is motion. Spatiotemporal group delay dispersion templates, corresponding to dielectric functions, classify specific motion trajectories. Unlike conventional empirical phase-based motion-analysis pipelines that require narrow-band spatial channelization and pyramid reconstruction, the physics-based MoViD operates directly on the broadband video signal, vastly increasing its computational speed. The resulting algorithm achieves unprecedented performance in both motion characterization and motion magnification, providing full-resolution motion mapping in real time at nearly 200 frames per second at 4K resolution and 70 frames per second for 8K video - orders of magnitude faster than classical optical flow approaches or learning-based methods. Notably, such extreme performance enables real time motion analysis at ultrahigh resolution for motion detection and compensation in robotics and other autonomous devices.
Callen MacPhee, Yiming Zhou, Bahram Jalali
ECE Department, UCLA, 405 Hilgard Ave., Los Angeles, 90095, California, USA.