eess.IVOct 3, 2026

FloVMos: Optical Flow-based Medical Video Mosaicking

Authors: Jinyang Liu, Sandesh Ghimire, Chaman Singh, Jennifer Dy, Milind Rajadhyaksha, Dana H. Brooks, Octavia Camps, Kivanc Kose

Organizations: Electrical and Computer Engineering, Northeastern University, 360 Huntington Ave, Boston, 02115, MA, USA. · Dermatology Service, Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, 10065, NY, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Biomedical imaging modalities often require a trade-off among resolution, field of view (FOV), and acquisition speed. Video mosaicking offers a strategy to overcome this limitation by computationally stitching sequential high-resolution frames into a wide-FOV composite. However, existing methods struggle with non-rigid deformations, and modality-specific artifacts arising in clinical and research imaging. Here, we present FloVMos, a generalizable, optical-flow-based deep learning framework for real-time video mosaicking across diverse biomedical imaging modalities. FloVMos achieves robust, pixel-level registration by fine-tuning an optical flow model on synthetic training data with ground-truth deformation fields. We introduce a pipeline for generating this training data, simulating realistic tissue motion and imaging distortions from existing mosaics or raw videos. Our automated synthetic data generation and optical flow model training based on this data allow users to adapt FloVMos to different imaging modalities. To demonstrate this, we applied FloVMos to seven diverse imaging modalities: reflection confocal microscopy, open-top light-sheet microscopy, fetoscopy, laparoscopy, dermoscopy, sparse spectral microscopy, and endoscopy. FloVMos outperforms conventional baselines in accuracy, robustness, and speed for all the tested modalities. This adaptable and training-efficient framework enables large-area visualization with real-time performance and may support broader use of video-based biomedical imaging in research and clinical workflows.

Figures & tables

Figure 1: An overview of the FloVMos videomosaicking pipeline highlighting its four key steps: (i) data acquisition, (ii) image registration using a fine-tuned optical flow estimator (a deep neural network), (iii) real-time image warping, and (iv) image alignment and blending.
Figure 2: Overview of FloVMos : a The first step in training involves the acquisition of training data. We have developed two approaches for easy, rapid, and automated acquisition of the required training data. In the first, when sample mosaics are easily available, we simply simulate videos by following a randomized but known path through the mosaic and then adding noise and distortion, as illustrated in the figure. In the second, when we only have videos available, we randomly choose or simulate pairs of images from the video, again with known relationships between the elements of each pair, and add noise and distortion. In both cases, we can thus assemble a collection of labeled pairs of images to use as training data. b We fine-tune the optical flow neural network using the annotated training data. c After training, we apply the DNN to newly acquired (or “test”) videos. The video frames are registered using DNN-predicted optical flow and then assembled into a mosaic by our graph-cuts based stitching algorithm. appending each registered image to the growing mosaic.
Figure 3: Results of applying FloVMos to different imaging modalities: Top: FloVMos qualitative results are presented for eight cases from seven different imaging modalities: (a) Fetoscopy; (b) Dermoscopy; (c) real RCM; (d) SSM; (e) simulated RCM; (f) Endoscopy; (g) Laparoscopy) and (h) videos simulated from an OTLS mosaic. The red circle / box in each image shows the location of the first frame in the video. (We use a red circle for the fetoscopy video to match the actual FOV of the device; when we use FloVMos to create a mosaic, the frames are stitched onto a black background.) Bottom: Histograms of distances (in pixels) between image feature point pairs in the source mosaics or video frames and their matches in the resulting FloVMos mosaic.
Figure 4: Quantitative assessment of accuracy and speed of FloVMos in comparison to selected existing videomosaicking algorithms on the simulated RCM videos: (a) Comparison of accuracy of FloVMos and three comparison algorithms: we compare accuracy using both the Structural Similarity Index Metric (SSIM, left panel) and Learned Perceptual Image Patch Similarity (LPIPS, middle panel) of FloVMos to the Parallax, As-Projective-As-Possible (APAP) [ 17 ] , and Automated Video-Mosaicking (AVM) algorithms. Note that for SSIM, higher is better, while for LPIPS, lower is better, as indicated by the up and down arrows in the image titles. The right panel reports on mosaicking speed in frames per second (FPS) for the same four algorithms. We show box plots for the full results with 800 simulated RCM videos. The horizontal line inside the box represents the median (50th percentile) of the data. The edges of the box denote the first quartile ( 25th percentile) and the third quartile (75th percentile). (b) Comparison among the same four algorithms in terms of registration error as a function of increased degree of non-rigid deformation. Colors represent the different algorithms, as indicated in the legend. Each simulated image pair is shown as a dot on the plot. The degree of (artificially induced) non-rigid registration is quantified as explained in the text. (c) Comparison among the same four algorithms in terms of mosaicking speed in frames per second. Results are shown in frames per second for videos consisting of 5, 10, 20, and 50 frames, as shown on the horizontal axis. The different algorithms are shown by the color of the box, as indicated in the legend. Again results are shown as box plots for all the synthesized RCM videos, using the same box plot format as in the top panel.
Figure 5: Stitching results obtained using the base optical flow model on the video sequences presented in Fig. 3 . The left two columns illustrate modalities characterized by large featureless black or white regions within frames (fetoscopy, OTLS, SSM, and RCM), whereas the right column depicts modalities visually more akin to conventional RGB imagery. Modalities featuring substantial black or white regions often exhibit failures in registration and subsequent stitching. Modalities resembling standard RGB images generally demonstrate improved registration performance even without fine-tuning; however, registration failures still occur. For example, incorrect alignment of the last several frames is evident in (e) endoscopy and (f) dermoscopy, and less accurate optical flow predictions result in noticeable artifacts in the final mosaic, as exemplified in (e) laparoscopy.
More in this topic

Medical Image Registration

Medical image registration aligns scans across time, patients, or imaging modalities to track anatomy and support diagnosis, treatment, and image-guided procedures.

See all 95 papers

Explore similar work

CardsList

  1. A Unified Deep Learning Framework for Motion Correction in Medical Imaging

    Sep 21, 2024Jian Wang, Razieh Faghihpirayesh, Danny Joca +2Fetal MRIMedical Image Registration

    Boston Children’s Hospital and Harvard Medical School, Boston, MA, USA · Department of Electrical and Computer Engineering at Northeastern University and also with the Department of Radiology at Boston Children’s Hospital, Boston, MA, USA · Department of Radiology at Boston Children’s Hospital and Harvard Medical School, Boston, MA, USA +2

  2. FlowVVTON: Flow-Guided Mask-Free Video Virtual Try-On

    Aug 31, 2026Shengyao Chen, Xianbing Sun, Liqing Zhang +1Virtual Try-OnTemporal Consistency in Video Generation

    Shanghai Jiao Tong University, Shanghai, China

  3. Real Time Motion Detection in Video Inspired by Photonic Time Stretch

    Oct 23, 2026Callen MacPhee, Yiming Zhou, Bahram JalaliOptical Flow

    ECE Department, UCLA, 405 Hilgard Ave., Los Angeles, 90095, California, USA.