cs.CVOct 27, 2026
SaveMvT: Event-based Multi-view Projection for Multiple Object Tracking
Organizations: College of Aerospace Science and Engineering, National University of Defense Technology, Changsha, 410073, Hunan, China · Hunan Provincial Key Laboratory of Image Measurement and Vision Navigation, Changsha, 410073, Hunan, China · College of Science, National University of Defense Technology, Changsha 410073, China · Mobile Perception Laboratory, School of Information Science and Technology, ShanghaiTech University, Shanghai 201210, China
Abstract
Event cameras are increasingly used for Multiple Object Tracking (MOT), but their asynchronous event output often requires specialized methods. Existing processing methods primarily follow two paradigms, pseudo-frames and event-by-event. The former is the prevailing approach since its data format aligns with images, making image-based techniques applicable. However, it suffers from tracking failures when trajectories overlap or are spatially close on pseudo-frames. Facing this challenge, we propose a multi-view pipeline, Multi-view Tracking (MvT), which preserves the 2D data format to leverage image-based techniques directly while introducing additional spatio-temporal views to resolve tracking ambiguities in a single view. MvT comprises a Multi-view Projection (MvP) module and a Multi-view Fusion (MvF) stage. MvP encodes events into three complementary spatio-temporal views while mitigating the pattern discretization. Within MvF, multi-view results are unified into a 3D coordinate system, and tracklets are associated through an optimization model subject to specific criteria combination. Evaluations on four datasets, including our self-collected Small Objects Dataset (SOD), show that MvT seamlessly integrates image-based methods and outperforms existing non-learning and learning trackers in generalized scenarios, and effectively resolves the single-view tracking ambiguities. Being training-free, MvT is applicable when ground-truth annotation is infeasible, thereby highlighting its practical, data-efficient potential. Code is available at https://github.com/zhazhabiu/MvTracking.
Figures & tables
Fig. 1: Comparison of different event processing approaches. (a) Asynchronous event streams of the Raindrops sequence in SOD. (b)-(d) The first row shows data representations (0.3 period) of pseudo-frames (event images [ 8 ] here), event-by-event processing, and our multi-view representation. The second row shows the corresponding tracking results from: our single-view framework variant with region growing, E-MS [ 7 ] , and the original MvT+RG. Different colors indicate distinct trajectories.
Fig. 2: Overview of our MvT pipeline and visualization of main stages, taking an image-based segmentation method as an example. (a) An event stream is divided into blocks using a sliding window and then fed into the processing stages one by one. (b) and (d) detail the key components of MvT, MvP and MvF, respectively. MvP generates multiple views through time-realigned events. In MvF, complete trajectories are recovered from the segmentation results of multiple views. (c) denotes the image-based segmentation stage.
Fig. 3: Illustrations of challenges in current event-based tracking paradigms. (a) In pseudo-frame representations, overlapping trajectories (green and blue) may lead to erroneous tracking. Arrows indicate the direction of motion. (b) In the TOX plane, each trajectory consists of a series of discrete events. Objects moving at different velocities (red, blue, and dark gray) generate trajectories with distinct temporal distributions, which necessitate different processing parameters.
Fig. 4: Pipeline of the proposed MvP. The raw event data first undergoes time realignment to compress temporal gaps. The realigned events are then projected onto three orthogonal planes (YOT, XOY, and TOX). Finally, each view is refined through a Gaussian pyramid to enhance structural continuity, producing the final multi-view representation. The projections are grayscale, and we binarize them only for readability.
Fig. 5: Pipeline of the MvF stage. It consists of two core phases: (a) Isolation, which resolves trajectory ambiguities by integrating segmentation results from multiple views to extract distinct 3D trajectories, and (b) Association, which associates continuous and discontinuous tracklets into complete trajectories through optimal assignment. A simple filtering rule based on event count is applied to the final output to exclude isolated noise.
Fig. 6: Illustration of iterative region post-processing in the YOT view, where tracklets evolve along the t-axis. (a) Initial segmentation results, with colors denoting different detection or segmentation results. The tracklets are highly fragmented. (b) During post-processing, a direction vector (red lines) is calculated from its two centroids and (green dots) for each tracklet fragment. (c) Merged tracklets after post-processing. (d) The final direction vectors and centroids recalculated for the tracklets are shown.
Fig. 7: Illustration of occlusion handling via multi-view trajectory isolation. The extracted trajectories from the XOY and YOT views are shown, where trajectory in the XOY corresponds to separated trajectories and in YOT, while trajectory corresponds to .
Fig. 8: Illustration of tracklet association criteria. and denote the angular deviations between the tracklet directions ( , ) and the normalized centroid connection vector , respectively. (a) Criterion 2 : tracklets with small directional variation are associated through motion continuity constraints. (b) Criterion 3 : tracklets exhibiting big directional changes are linked via spatio-temporal intersection verification.
| Category | Source | Seq. | Segments | Objects | Type | Size | Scene | Motion Type | Speed (px/s) | Camera | GT | Period (s) |
| SOD-r | Real-world | Sphere-20 | 3 | 20 | Balls | 5-10px | Mixed | Non-linear | 2500 | Fixed | No | 0.5 |
| Sphere-30 | 3 | 30 | Balls | Mixed | Non-linear | 0.5 | ||||||
| Raindrops | 3 | 100 | Raindrops | Rain | Linear | 4.0 | ||||||
| SOD-b | Blender | Tree | 1 | 19 | Balls | Tree | Free | 100 | 6DoF | Yes | 3.0 | |
| Sky | 1 | 28 | Balls | Cloudy sky | 3.0 | |||||||
| Smog | 1 | 44 | Balls | Full of smog | 3.0 |
TABLE I: The configuration of SOD. “Seq.” means sequences, and “Segments” denotes the number of segments in each sequence.
Fig. 9: The recording equipment and smog scenario setup in the XOY view used in our SOD-b dataset.
Fig. 10: Partial tracking results on ECD Shapes [ 19 ] are presented in 2D and 3D. In 2D images (the XOY view), red boxes denote GT, while green boxes represent the tracking results. For 3D visualizations, the displayed period is 200 ms around the timestamp of the left image. Distinct colors represent individual trajectories. From left to right, the sequences are translation (frame 532), 6DoF (frame 236), and 6DoF (frame 1014).
| Seq. | GTs | Method | P | R | MOTA | Track | Track | IDF1 | FPS |
| (%) | (%) | (%) | IoU(%) | Acc(%) | (%) | ||||
| trans. | Yes | ETD | 48.72 | 40.61 | -6.31 | – | – | – | 7.44 10 -7 |
| RMRNet | 95.88 | 91.69 | – | – | – | – | 6.67 | ||
| AEBT | 81.87 | 81.73 | 54.11 | – | – | – | 2.51 | ||
| No | E-MS | 74.74 | 97.24 | 62.56 | 30.51 | 40.57 | 40.36 | 2.81 10 4 | |
| MvT+RG | 75.85 | 99.36 | 64.67 | 43.73 | 55.69 | 72.84 | 2.58 10 1 |
TABLE II: Results of MOT on ECD Shapes. Here “GTs” indicates whether the method uses GTs initialization. “trans.” and “rotat.” are translation and rotation sequences. “FPS” refers to the processing batches/frames per second. The best and second best values are highlighted.
| Method | Seg. IoU(%) | Seg. Acc(%) | Number of Tracks |
| AEMOT | 36.73 | 37.83 | 297 |
| E-MS | 29.52 | 98.08 | 17686 |
| PROPHESEE | 28.39 | 33.99 | 954 |
| MvT+RG | 54.81 | 57.76 | 351 |
| MvT+WS | 29.93 | 30.42 | 276 |
| MvT+MS | 53.44 | 71.66 | 630 |
TABLE III: Tracking performance on BEES.
| Method | Task | Source | Train | Event Rep. | Seg. IoU(%) | Seg. Acc(%) | Track IoU(%) | Track Acc(%) | IDF1(%) | FPS |
| HDETrack | SOT | 2024CVPR | Yes | Event Count | 74.06 | 86.87 | 72.86 | 86.02 | 99.00 | 8.66 |
| +MvT | MOT | – | No | MvP | 65.29 | 72.71 | 60.91 | 67.17 | 73.15 | 147.68 |
| TSETrack | SOT | 2026IJCV | Yes | Event Count | 78.03 | 88.55 | 76.00 | 88.22 | 99.50 | 13.98 |
| +MvT | MOT | – | No | MvP | 68.88 | 74.42 | 60.17 | 67.49 | 75.14 | 146.01 |
| YOLOX* | Detect. | 2021ArXiv | Yes | Event Count | 69.58 | 70.86 | – | – | – | – |
| +OCSORT* | MOT | 2023CVPR | No | Event Count | 61.25 | 62.02 | 35.70 | 29.44 | 39.72 | 615.75 |
TABLE IV: Results on Ev-UAV. “*” means the image-based method. “Train” column indicates whether the method needs to be trained. “Rep.” means the representation. In “Event Rep.” column, “Event Count” is event count images, one of the pseudo-frame representation, while “Point” is the original event data format. Except for two SOT models, the FPS for the remaining models is computed excluding the detection/segmentation time (or XOY processing time for MvT).
Fig. 11: Box plots of Feature Age on BEES.
Fig. 12: Box plots of Feature Age on SOD-r Raindrops.
Fig. 13: Tracking result examples on the BEES dataset for GT, AEMOT, Prophesee SDK, and MvT+RG, visualized across two periods (2-2.5 and 3.5-4 ). 2D images (XOY views) are synthesized by accumulating all tracked event trajectories, where black pixels indicate no events and other colors correspond to distinct trajectories. Note that official GTs do not distinguish between individual bee trajectories, so they are shown in a uniform color.
Fig. 14: Visualization of tracking results on three scenes in SOD-r. Rows show the 2D and 3D tracking results for three sequences. In the 2D visualizations (XOY views), the background is an event accumulation, where black and white denote the absence and presence of events, respectively.
| Method | Task Type | Event Rep. | Seg. IoU(%) | Seg. Acc(%) | Track IoU(%) | Track Acc.(%) | IDF1(%) |
| HDETrack | SOT | Event Count | 0.70 | 24.90 | 0.60 | 0.73 | 13.33 |
| +MvT | MOT | MvP | 0.70 | 24.90 | 1.69 | 9.68 | 38.38 |
| TSETrack | SOT | Event Count | 0.82 | 29.14 | 1.23 | 1.55 | 13.64 |
| +MvT | MOT | MvP | 0.82 | 29.13 | 2.78 | 9.83 | 33.16 |
| YOLOX* | Detection | Event Count | 0.41 | 0.42 | – | – | – |
| +OCSORT* | MOT | MvP | 0.42 | 0.42 | 2.87 | 0.40 | 6.52 |
TABLE V: Results on SOD-b. Hereafter, only the unique best values are highlighted in bold.
Fig. 15: Tracking results of OCSORT and our framework on Ev-UAV based on YOLOX. 2D visualizations are XOY views. The sequences shown in the left and right two columns are test_017 and val_019, respectively.
Fig. 16: Results on the sequence (test_013) in Ev-UAV where MvT exhibits a decrease in segmentation performance. The first row presents the GT, TSETrack, and TSETrack+MvT results. In the second row, (a) shows the detected trajectories in the 11th processing window of TSETrack. (b)-(d) present the detections in (a) encoded by our multi-view representation, different colors indicate different tracklets MvT detects.
Fig. 17: Case study comparing existing pseudo-frames and multi-view representation under occlusion (first row, Ev-UAV test_007 sequence) and closely distributed trajectories (second row, SOD-r Raindrops). (a) shows the results of multi-view variants, MvT+TSETrack and MvT+RG. (b) shows the results of voxel-grid variant MvF+TSETrack and single-view variant MvF+RG.
Fig. 18: The results of different combinations of and in Gaussian kernel design using RG on ECD.
Fig. 19: The results of different combinations of and in Gaussian kernel design using WS on ECD.
Fig. 20: Results of different numbers of layers on Gaussian pyramid.
| Seq. | Rep. | P(%) | R(%) | MOTA(%) | IDF1(%) |
| translation | MvP | 76.49 | 98.91 | 64.49 | 75.87 |
| SingV | 74.22 | 98.83 | 61.42 | 70.65 | |
| VG-① | 30.21 | 97.98 | -173.50 | 12.15 | |
| VG-② | 10.62 | 64.96 | -529.21 | 2.05 | |
| rotation | MvP | 69.89 | 98.45 | 49.56 | 70.98 |
| SingV | 65.42 | 98.61 | 41.14 | 65.16 |
TABLE VI: Ablation results of the multi-view representation on ECD using MS.
| Method | Rep. | Track IoU(%) | Track Acc(%) | IDF1(%) |
| YOLOX | MvP | 50.79 | 54.44 | 71.74 |
| SingV | 34.20 | 45.27 | 61.73 | |
| RVT | MvP | 33.04 | 13.99 | 39.22 |
| VG-③ | 18.31 | 11.67 | 44.37 |
TABLE VII: Ablation results of the multi-view representation on Ev-UAV.
| Seq. | Model | P(%) | R(%) | MOTA(%) | IDF1(%) |
| translation | MvT+RG | 75.85 | 99.36 | 64.67 | 72.84 |
| w/o TR | 75.80 | 99.33 | 64.47 | 72.23 | |
| rotation | MvT+RG | 65.73 | 99.11 | 42.47 | 65.97 |
| w/o TR | 65.73 | 99.11 | 42.47 | 65.81 | |
| 6dof | MvT+RG | 77.90 | 99.48 | 67.83 | 57.21 |
| w/o TR | 77.90 | 99.48 | 67.81 | 56.52 |
TABLE VIII: Ablation results of time realignment (TR) on ECD.
| Method | Model | Track IoU(%) | Track Acc(%) | IDF1(%) |
| HDETrack | MvT+RG | 60.91 | 67.17 | 73.15 |
| w/o TR | 59.19 | 66.64 | 71.66 | |
| TSETrack | MvT+RG | 60.17 | 67.49 | 75.14 |
| w/o TR | 59.14 | 67.45 | 74.52 | |
| RVT | MvT+RG | 33.04 | 13.99 | 39.22 |
| w/o TR | 33.02 | 13.73 | 38.82 |
TABLE IX: Ablation results of time realignment (TR) on Ev-UAV.
More in this topic
Multi-Object Tracking
Multi-object tracking (MOT) links detections across video frames to maintain object identities through motion, occlusion, and changing viewpoints.
See all 193 papers