cs.CVOct 23, 2026

Real Time Motion Detection in Video Inspired by Photonic Time Stretch

Authors: Callen MacPhee, Yiming Zhou, Bahram Jalali

Organizations: ECE Department, UCLA, 405 Hilgard Ave., Los Angeles, 90095, California, USA.

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Motion perception is fundamental to computer vision and physical AI, and plays a central role in robotics, autonomous vehicles, human-computer interaction, and video compression. Traditional approaches such as background subtraction, frame-differencing and optical flow algorithms often struggle in complex or dynamic scenes, typically sacrificing spatial resolution for computational tractability. Here we introduce MoViD (Motion estimation via Virtual dispersion and phase Detection), a physics-inspired algorithm grounded in the principles that govern photonic time stretch - a highly successful ultrafast realtime data acquisition platform. Drawing inspiration from the optical physics therein, MoViD treats video as a collection of optical time series waveforms associated with pixels. Motion of an object through a pixel broadens the time series spectrum and, after group delay dispersion, induces a temporal phase that exists only if there is motion. Spatiotemporal group delay dispersion templates, corresponding to dielectric functions, classify specific motion trajectories. Unlike conventional empirical phase-based motion-analysis pipelines that require narrow-band spatial channelization and pyramid reconstruction, the physics-based MoViD operates directly on the broadband video signal, vastly increasing its computational speed. The resulting algorithm achieves unprecedented performance in both motion characterization and motion magnification, providing full-resolution motion mapping in real time at nearly 200 frames per second at 4K resolution and 70 frames per second for 8K video - orders of magnitude faster than classical optical flow approaches or learning-based methods. Notably, such extreme performance enables real time motion analysis at ultrahigh resolution for motion detection and compensation in robotics and other autonomous devices.

Figures & tables

Figure 1: Real-Time Full-Resolution Motion Analysis using the MoViD Algorithm. Three video frames of a quadruped robot (top row) and the corresponding MoViD outputs (bottom row). In the MoViD visualizations, motion directionality is encoded according to the color wheel shown in the bottom-left inset. Notably, visualization of translational and rotational motion is shown as the quadruped rotates and climbs the stairs in the original video. The MoViD algorithm characterizes motion at the pixel-level in full resolution. The resulting motion templates serve as a compact, label-free representation for Physical AI and robotic perception tasks, including anomaly detection, object association, throughput estimation, and long-term system monitoring.
Figure 2: Operating and physical principles of MoViD. Together, panels (a–c) illustrate how MoViD converts intensity variations to spectral phase kinematics, enabling pixel-level motion detection. (a) Conceptual illustration: two objects that move at different velocities (blue = fast, red = slow) modulate the blue and red pixels time series via their motion. The gray pixel remains static and therefore produces a constant pixel time series. (b) Each time series is modulated via the moving object. Faster motion generates a narrower intensity pulse and broader spectrum. Virtual chromatic dispersion encodes motion in spectral phase. The broader spectrum is stretched more strongly in time, producing a larger temporal-phase modulation Δψ(t)\Delta\psi(t) ; conversely, slower motion yields a narrower bandwidth and a weaker phase response. The constant gray pixel time series has essentially zero temporal bandwidth and therefore produces zero phixel response. Spectral power remains unchanged. (c) Without dispersion, the intensity pulse width, spectral power, and temporal phase are unaltered, so motion cannot be discriminated.
Figure 3: Algorithmic Flow of MoViD. Pseudocode of MoViD and block diagram of the physical pipeline. Each single-pixel time series of the input video is passed through virtual chromatic dispersion, followed by coherent detection to extract the phixel phase map that forms the output motion video. Inputs to the algorithm are a single-channel input video as data and parameters phase strength S and normalized spectral phase kernel ϕ(ω)\phi(\omega) . Performed independently for every pixel: a 1-D FFT applies a user-defined spectral phase H[ω]=e−iϕ[ω]H[\omega]=e^{-i\phi[\omega]} ; an inverse FFT returns the dispersed signal; coherent phase detection yields the phixel field ψ[n,m,t]\psi[n,m,t] .
Figure 4: Algorithmic Flow of MoViD Motion Magnification. Pseudocode of MoViD motion magnification. Each single-pixel time series of the input video is passed through virtual chromatic dispersion, followed by magnification of quadrature information. After summation with the in-phase component, the magnitude of the signal serves as the output. Inputs to the algorithm are an input video as data and parameters phase strength S which should be greater than 2π2\pi , gain term GG , and normalized spectral phase kernel ϕ(ω)\phi(\omega) .
Figure 5: Algorithmic Block Diagram of MoViD vs. Existing Motion‑Magnification Pipelines. (a) Current state-of-the-art (steerable-pyramid) workflow. Video frames are first decomposed into a multi-scale spatial pyramid; every spatial bin then undergoes phase-specific temporal filtering, spatial Fourier shifting, and a final pyramid reconstruction, resulting in multiple serial passes over the data cube and high memory traffic. (b) MoViD processing chain. A single broadband chromatic-dispersion operator generates in-phase (I) and quadrature (Q) channels; coherent gain is applied to the Q-channel and the two streams are summed to produce the output video. Eliminating pyramid decomposition/reconstruction collapses the pipeline to two lightweight stages, greatly reducing computational complexity.
Figure 6: Motion Segmentation Capabilities of MoViD in a Real-World Industrial Sorting Scenario. Three frames from an overhead camera of a manufacturing conveyor sorting system (top row) and the corresponding MoViD outputs (middle row), demonstrating single-pixel-level motion segmentation. MoViD isolates only dynamically changing regions while suppressing any static background, yielding high-contrast representations of moving packages. The highlighted zoom-in (bottom) shows that fine object details are preserved by MoViD, including package edges and printed barcodes, which are clearly visible in the motion domain. This capability enables precise object-level segmentation, sorting, and tracking for industrial inspection, sorting, and automation tasks.
MethodF1 scoreIoU
KNN0.7720.649
MOG20.6880.557
MoViD0.8210.707
Table 1: Motion Localization on the CDnet2014 Baseline Category. Mean F1 score and IoU are unweighted averages across the four sequence-level values. Higher values are better. Bold values mark the best result.
Figure 7: Directional and magnitude-based motion classification demonstrated on wind turbine footage . (a) Original video frames showing turbine rotation. (b) Directional classification of motion components where color represents directionality. (c) Speed-based classification where colors represent different velocity ranges, revealing how blade tips move faster than regions closer to the hub.
Figure 8: Comparison with Traditional and State-of-the-Art Optical Flow . The left result is generated via the classical Farnebäck Method [ 23 ] , the center is generated with the learning-based approach RAFT [ 11 ] , and the right is MoViD. Whereas the classical approach is too coarse and low-resolution, the learning-based approach introduces artifacts in the background. MoViD shows the highest resolution result. Frame-per-second metrics are shown for each approach, with MoViD running at double the traditional approach and an order of magnitude faster than the deep learning approach.
Figure 9: Runtime Advantage of MoViD over Existing Motion Magnification Pipelines. (Top) Runtime comparison (FPS) across various resolution scales up to 8K resolution. (Bottom) Visualization of runtime comparison (FPS) at 1080p resolution. Throughput benchmarking on an Intel i9-12900K CPU and NVIDIA TITAN RTX GPU. Across seven resolutions (240p →\rightarrow 8K), MoViD vastly outperforms both the steerable-pyramid CPU baseline [ 6 ] and a learning-based GPU baseline [ 13 ] . At 1080p, MoViD reaches 21 FPS on CPU (vs. 0.09 FPS) and 874 FPS on GPU (vs. 6.2 FPS); at 8K it still sustains 68 FPS on GPU, whereas competing methods fail to achieve real-time performance. These gains exceed two orders of magnitude at every tested resolution and stem directly from the streamlined architecture. Results represented by a ×\times correspond to the situation in which the algorithm is not able to complete the task on the data given memory or timing constraints of the machine.
More in this topic

Optical Flow

Optical flow estimates the apparent motion of pixels between images, supporting video understanding, tracking, robotics, and image synthesis.

See all 58 papers

Explore similar work

CardsList

  1. Real-time Motion Segmentation with Event-based Normal Flow

    Feb 24, 2026Sheng Zhong, Zhongyang Ren, Xiya Zhu +3Motion SegmentationEvent-Based Vision

    Neuromorphic Automation and Intelligence Lab (NAIL) at School of Artificial Intelligence and Robotics, Hunan University, Changsha, China. · Capital One. · University of Maryland, College Park.

  2. SMF-VO: Direct Ego-Motion Estimation via Sparse Motion Fields

    Nov 12, 2025Sangheon Yang, Yeongin Yoon, Hong Mo Jung +1Visual Odometry

    Department of Artificial Intelligence, Hanyang University, Seoul, Korea · Department of Mechanical Engineering, Seoul National University, Seoul, Korea · All authors are with the Robot Vision Lab, Seoul National University.

  3. CodecMotion-OAD: Efficient Online Action Detection with Compressed-Domain Motion

    Oct 17, 2026Zepeng Sun, Naichuan Zheng, Hailun Xia +1Temporal Action DetectionHuman Activity Recognition

    School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China