Motion perception is fundamental to computer vision and physical AI, and plays a central role in robotics, autonomous vehicles, human-computer interaction, and video compression. Traditional approaches such as background subtraction, frame-differencing and optical flow algorithms often struggle in complex or dynamic scenes, typically sacrificing spatial resolution for computational tractability. Here we introduce MoViD (Motion estimation via Virtual dispersion and phase Detection), a physics-inspired algorithm grounded in the principles that govern photonic time stretch - a highly successful ultrafast realtime data acquisition platform. Drawing inspiration from the optical physics therein, MoViD treats video as a collection of optical time series waveforms associated with pixels. Motion of an object through a pixel broadens the time series spectrum and, after group delay dispersion, induces a temporal phase that exists only if there is motion. Spatiotemporal group delay dispersion templates, corresponding to dielectric functions, classify specific motion trajectories. Unlike conventional empirical phase-based motion-analysis pipelines that require narrow-band spatial channelization and pyramid reconstruction, the physics-based MoViD operates directly on the broadband video signal, vastly increasing its computational speed. The resulting algorithm achieves unprecedented performance in both motion characterization and motion magnification, providing full-resolution motion mapping in real time at nearly 200 frames per second at 4K resolution and 70 frames per second for 8K video - orders of magnitude faster than classical optical flow approaches or learning-based methods. Notably, such extreme performance enables real time motion analysis at ultrahigh resolution for motion detection and compensation in robotics and other autonomous devices.
Figures & tables
Figure 1: Real-Time Full-Resolution Motion Analysis using the MoViD Algorithm. Three video frames of a quadruped robot (top row) and the corresponding MoViD outputs (bottom row). In the MoViD visualizations, motion directionality is encoded according to the color wheel shown in the bottom-left inset. Notably, visualization of translational and rotational motion is shown as the quadruped rotates and climbs the stairs in the original video. The MoViD algorithm characterizes motion at the pixel-level in full resolution. The resulting motion templates serve as a compact, label-free representation for Physical AI and robotic perception tasks, including anomaly detection, object association, throughput estimation, and long-term system monitoring.
Figure 2: Operating and physical principles of MoViD. Together, panels (a–c) illustrate how MoViD converts intensity variations to spectral phase kinematics, enabling pixel-level motion detection. (a) Conceptual illustration: two objects that move at different velocities (blue = fast, red = slow) modulate the blue and red pixels time series via their motion. The gray pixel remains static and therefore produces a constant pixel time series. (b) Each time series is modulated via the moving object. Faster motion generates a narrower intensity pulse and broader spectrum. Virtual chromatic dispersion encodes motion in spectral phase. The broader spectrum is stretched more strongly in time, producing a larger temporal-phase modulation Δψ(t) ; conversely, slower motion yields a narrower bandwidth and a weaker phase response. The constant gray pixel time series has essentially zero temporal bandwidth and therefore produces zero phixel response. Spectral power remains unchanged. (c) Without dispersion, the intensity pulse width, spectral power, and temporal phase are unaltered, so motion cannot be discriminated.
Figure 3: Algorithmic Flow of MoViD. Pseudocode of MoViD and block diagram of the physical pipeline. Each single-pixel time series of the input video is passed through virtual chromatic dispersion, followed by coherent detection to extract the phixel phase map that forms the output motion video. Inputs to the algorithm are a single-channel input video as data and parameters phase strength S and normalized spectral phase kernel ϕ(ω) . Performed independently for every pixel: a 1-D FFT applies a user-defined spectral phase H[ω]=e−iϕ[ω] ; an inverse FFT returns the dispersed signal; coherent phase detection yields the phixel field ψ[n,m,t] .
Figure 4: Algorithmic Flow of MoViD Motion Magnification. Pseudocode of MoViD motion magnification. Each single-pixel time series of the input video is passed through virtual chromatic dispersion, followed by magnification of quadrature information. After summation with the in-phase component, the magnitude of the signal serves as the output. Inputs to the algorithm are an input video as data and parameters phase strength S which should be greater than 2π , gain term G , and normalized spectral phase kernel ϕ(ω) .
Figure 5: Algorithmic Block Diagram of MoViD vs. Existing Motion‑Magnification Pipelines. (a) Current state-of-the-art (steerable-pyramid) workflow. Video frames are first decomposed into a multi-scale spatial pyramid; every spatial bin then undergoes phase-specific temporal filtering, spatial Fourier shifting, and a final pyramid reconstruction, resulting in multiple serial passes over the data cube and high memory traffic. (b) MoViD processing chain. A single broadband chromatic-dispersion operator generates in-phase (I) and quadrature (Q) channels; coherent gain is applied to the Q-channel and the two streams are summed to produce the output video. Eliminating pyramid decomposition/reconstruction collapses the pipeline to two lightweight stages, greatly reducing computational complexity.
Figure 6: Motion Segmentation Capabilities of MoViD in a Real-World Industrial Sorting Scenario. Three frames from an overhead camera of a manufacturing conveyor sorting system (top row) and the corresponding MoViD outputs (middle row), demonstrating single-pixel-level motion segmentation. MoViD isolates only dynamically changing regions while suppressing any static background, yielding high-contrast representations of moving packages. The highlighted zoom-in (bottom) shows that fine object details are preserved by MoViD, including package edges and printed barcodes, which are clearly visible in the motion domain. This capability enables precise object-level segmentation, sorting, and tracking for industrial inspection, sorting, and automation tasks.
Method
F1 score
IoU
KNN
0.772
0.649
MOG2
0.688
0.557
MoViD
0.821
0.707
Table 1: Motion Localization on the CDnet2014 Baseline Category. Mean F1 score and IoU are unweighted averages across the four sequence-level values. Higher values are better. Bold values mark the best result.
Figure 7: Directional and magnitude-based motion classification demonstrated on wind turbine footage . (a) Original video frames showing turbine rotation. (b) Directional classification of motion components where color represents directionality. (c) Speed-based classification where colors represent different velocity ranges, revealing how blade tips move faster than regions closer to the hub.
Figure 8: Comparison with Traditional and State-of-the-Art Optical Flow . The left result is generated via the classical Farnebäck Method [ 23 ] , the center is generated with the learning-based approach RAFT [ 11 ] , and the right is MoViD. Whereas the classical approach is too coarse and low-resolution, the learning-based approach introduces artifacts in the background. MoViD shows the highest resolution result. Frame-per-second metrics are shown for each approach, with MoViD running at double the traditional approach and an order of magnitude faster than the deep learning approach.
Figure 9: Runtime Advantage of MoViD over Existing Motion Magnification Pipelines. (Top) Runtime comparison (FPS) across various resolution scales up to 8K resolution. (Bottom) Visualization of runtime comparison (FPS) at 1080p resolution. Throughput benchmarking on an Intel i9-12900K CPU and NVIDIA TITAN RTX GPU. Across seven resolutions (240p → 8K), MoViD vastly outperforms both the steerable-pyramid CPU baseline [ 6 ] and a learning-based GPU baseline [ 13 ] . At 1080p, MoViD reaches 21 FPS on CPU (vs. 0.09 FPS) and 874 FPS on GPU (vs. 6.2 FPS); at 8K it still sustains 68 FPS on GPU, whereas competing methods fail to achieve real-time performance. These gains exceed two orders of magnitude at every tested resolution and stem directly from the streamlined architecture. Results represented by a × correspond to the situation in which the algorithm is not able to complete the task on the data given memory or timing constraints of the machine.
Event-based cameras are bio-inspired sensors with pixels that independently and asynchronously respond to brightness changes at microsecond resolution, offering the potential to handle visual tasks in challenging scenarios. However, due to the sparse information content in individual events, directly processing the raw event data to solve vision tasks is highly inefficient, which severely limits the applicability of state-of-the-art methods in real-time tasks, such as motion segmentation, a fundamental task for dynamic scene understanding. Incorporating normal flow as an intermediate representation to compress motion information from event clusters within a localized region provides a more effective solution. In this work, we propose a normal flow-based motion segmentation framework for event-based vision. Leveraging the dense normal flow directly learned from event neighborhoods as input, we formulate the motion segmentation task as an energy minimization problem solved via graph cuts, and optimize it iteratively with normal flow clustering and motion model fitting. By using a normal flow-based motion model initialization and fitting method, the proposed system is able to efficiently estimate the motion models of independently moving objects with only a limited number of candidate models, which significantly reduces the computational complexity and ensures real-time performance, achieving nearly a 800x speedup in comparison to the open-source state-of-the-art method. Extensive evaluations on multiple public datasets fully demonstrate the accuracy and efficiency of our framework.
Sheng Zhong, Zhongyang Ren, Xiya Zhu +3
Neuromorphic Automation and Intelligence Lab (NAIL) at School of Artificial Intelligence and Robotics, Hunan University, Changsha, China. · Capital One. · University of Maryland, College Park.
Traditional Visual Odometry (VO) and Visual Inertial Odometry (VIO) methods rely on a 'pose-centric' paradigm, which computes absolute camera poses from the local map thus requires large-scale landmark maintenance and continuous map optimization. This approach is computationally expensive, limiting their real-time performance on resource-constrained devices. To overcome these limitations, we introduce Sparse Motion Field Visual Odometry (SMF-VO), a lightweight, 'motion-centric' framework. Our approach directly estimates instantaneous linear and angular velocity from sparse optical flow, bypassing the need for explicit pose estimation or expensive landmark tracking. We also employed a generalized 3D ray-based motion field formulation that works accurately with various camera models, including wide-field-of-view lenses. SMF-VO demonstrates superior efficiency and competitive accuracy on benchmark datasets, achieving over 100 FPS on a Raspberry Pi 5 using only a CPU. Our work establishes a scalable and efficient alternative to conventional methods, making it highly suitable for mobile robotics and wearable devices.
Sangheon Yang, Yeongin Yoon, Hong Mo Jung +1
Department of Artificial Intelligence, Hanyang University, Seoul, Korea · Department of Mechanical Engineering, Seoul National University, Seoul, Korea · All authors are with the Robot Vision Lab, Seoul National University.
Online action detection (OAD) identifies ongoing actions in streaming videos. Many existing OAD systems combine RGB appearance with optical flow, but dense-flow extraction remains a major computational bottleneck. Codec motion vectors (MVs) are available during decoding, avoiding a separate optical-flow computation. On THUMOS'14, optical flow exceeds MVs by 11.2 mAP points with motion-only inputs, whereas the gap narrows to 3.7 points after RGB fusion, with the largest reduction in action-category discrimination. We propose CodecMotion-OAD, which replaces optical flow with decoder-provided MVs while retaining existing temporal detector architectures. Its two-stage training first aligns MV features with a pretrained flow-feature interface without action labels and then trains RGB+MV detectors using ground-truth labels and soft predictions from a frozen RGB+Flow teacher. To the best of our knowledge, this is the first application of codec MVs to frame-level OAD. On THUMOS'14 with MiniROAD, CodecMotion-OAD achieves 68.3 mAP versus 70.9 mAP for matched-distillation RGB+Flow, while reducing decoder-to-prediction processing time by 11.48-fold on CPU and 2.04-fold on GPU.
Zepeng Sun, Naichuan Zheng, Hailun Xia +1
School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, Beijing, China