TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation

Jini Yang; Eunbeen Hong; Soowon Son; Hyunkoo Lee; Sunghwan Hong; Sunok Kim; Seungryong Kim

cs.CVMar 24, 2026

TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation

Authors: Jini Yang, Eunbeen Hong, Soowon Son, Hyunkoo Lee, Sunghwan Hong, Sunok Kim, Seungryong Kim

Organizations: KAIST AI · ETH Zurich · Korea Aerospace University

Abstract

Event cameras capture per-pixel brightness changes with microsecond resolution, offering continuous motion information lost between RGB frames. However, existing event-based motion estimators depend on large-scale synthetic data that often suffers from a significant sim-to-real gap. We propose TETO (Tracking Events with Teacher Observation), a teacher-student framework that learns event motion estimation from only $\sim$ 25 minutes of unannotated real-world recordings through knowledge distillation from a pretrained RGB tracker. Our motion-aware data curation and query sampling strategy maximizes learning from limited data by disentangling object motion from dominant ego-motion. The resulting estimator jointly predicts point trajectories and dense optical flow, which we leverage as explicit motion priors to condition a pretrained video diffusion transformer for frame interpolation. We achieve state-of-the-art point tracking on EVIMO2 and optical flow on DSEC using orders of magnitude less training data, and demonstrate that accurate motion estimation translates directly to superior frame interpolation quality on BS-ERGB and HQ-EVFI.

Figures & tables

Figure 1: Training data scale of event-based trackers.

Figure 2: Real event vs. synthetic event analysis. (a) Inter-Event Interval (IEI) distributions show that real events concentrate in short intervals with rapid decay, while synthetic events exhibit long tails and periodic artifacts. (b) The appearance of real and synthetic events differs substantially, with synthetic events from V2E [ 29 ] and Voltmeter [ 38 ] exhibiting artifacts not present in real-world recordings.

Dataset	Align	Modality	Motion dyn	Scene	Resolution	Challenge	FPS
E-POSE [ 27 ]	✗	E+R	Cam	Indoor	346 $\times$ 260	-	-
M3ED [ 7 ]	✗	E+R	Cam	Indoor/Outdoor	1280 $\times$ 720	-	20
MTevent [ 2 ]	✗	E+R	Obj+Cam	Indoor	640 $\times$ 480	-	25, 100
HS-ERGB [ 57 ]	✗	E+R	Obj+Cam	Indoor/Outdoor	1280 $\times$ 720	Non-linear Motion	240
RELI11D [ 62 ]	✗	E+R	Obj+Cam	Indoor/Outdoor	1280 $\times$ 800	Complex Motion	60
DSEC [ 16 ]	✗	E+R	Cam	Outdoor	640 $\times$ 480	Low Light	20

Table 1: Public event-RGB paired datasets. (E: Event, R: RGB, G:Grayscale.) Align indicates beamsplitter-based spatial alignment between event and frame sensors. Challenge denotes the dominant difficulty of each dataset. Both BS-ERGB and ERF-X170FPS capture extreme object motion, but the higher frame rate of ERF-X170FPS (170 vs. 28 FPS) allows the RGB teacher to produce stable pseudo-label trajectories. Blue rows denote our training sets.

Figure 3: Object motion query sampling. Given teacher-predicted optical flow, we estimate a global affine model via RANSAC and compute residual flow to identify independently moving regions. Queries are oversampled from these regions to prevent bias toward dominant ego-motion patterns.

Figure 4: Overall architecture. The concentration network aggregates multi-scale events into a 3-channel representation. Pseudo-labels for tracking and optical flow are provided by the teacher tracker.

Figure 5: Video frame interpolation architecture. (a) Motion signals from TETO and event motion mask construction. (b) Overall pipeline with latent-space flow warping and trajectory-guided attention supervision.

Method	Setting	All		interlaken_00_b		interlaken_01_a		thun_01_a		thun_01_b		zurich_city_12_a		zurich_city_14_c		zurich_city_15_a
Method	Setting	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$	EPE $\downarrow$	AE $\downarrow$
RTEF [ 3 ]	In-domain	4.88	–	8.59	–	5.94	–	3.01	–	3.91	–	3.14	–	4.00	–	3.78	–
MultiCM [ 52 ]	In-domain	3.47	13.98	5.74	9.19	3.74	9.77	2.12	11.06	2.48	12.05	3.86	28.61	2.72	12.62	2.35	11.82
BTEB [ 46 ]	In-domain	3.86	–	6.32	–	4.91	–	2.33	–	3.04	–	2.62	–	3.36	–	2.97	–
Paredes et al. [ 47 ]	In-domain	2.33	10.56	3.34	6.22	2.49	6.88	1.73	9.75	1.66	8.41	2.72	23.16	2.64	10.23	1.69	8.88
EV-FlowNet [ 68 ]	In-domain	3.86	–	6.32	–	4.91	–	2.33	–	3.04	–	2.62	–	3.36	–	2.97	–

Table 3: Quantitative comparison on DSEC. Lower EPE, AE is better. Bold and underline denote the best and second-best results within each category.

Figure 7: Qualitative comparison under dynamic motion.

Methods	BS-ERGB [ 56 ]
Methods	$\text{OATS}_{\text{0}}$ ↑	$\text{OATS}_{\text{1}}$ ↑	$\text{OATS}_{\text{2}}$ ↑	$\text{OATS}_{\text{4}}$ ↑	$\text{OATS}_{\text{8}}$ ↑	$\text{OATS}_{\text{16}}$ ↑	$\text{OATS}_{\text{avg}}$ ↑
ETAP [ 22 ]	0.7134	0.7256	0.7388	0.7568	0.7824	0.8149	0.7553
TETO	0.8294	0.8438	0.8587	0.8785	0.9047	0.9324	0.8746

Table 4: OATS on BS-ERGB.

Figure 8: Robust tracking in extreme conditions on BS-ERGB (Top) and EVFI-LL (Bottom) [ 64 ] .

Figure 9: Qualitative comparison under dynamic motion.

Method	$\times$ 1				$\times$ 3				$\times$ 6
Method	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
TimeLens-XL [ 43 ]	15.29	0.0808	28.2734	0.8223	38.11	0.0695	29.7315	0.8367	38.11	0.1314	22.4263	0.7574
CBMNet-Large [ 36 ]	10.64	0.1667	29.2348	0.7737	11.23	0.1730	28.4550	0.7580	13.31	0.1791	27.6728	0.7511
RE-VDM [ 8 ]	16.37	0.1004	28.0351	0.8631	16.66	0.1268	27.0266	0.8119	19.40	0.1258	26.9337	0.8426
TETO-VFI	10.41	0.0684	25.7317	0.8346	7.65	0.0689	26.2135	0.8314	7.88	0.0859	25.1521	0.7955

Table 5: Video frame interpolation comparison on BS-ERGB

Method	$\times$ 7				$\times$ 15
Method	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
TimeLens-XL [ 43 ]	26.50	0.0601	25.5082	0.8856	28.83	0.0559	25.6656	0.8942
CBMNet-Large [ 36 ]	18.92	0.0717	29.0001	0.8849	26.19	0.0827	27.3779	0.8767
RE-VDM [ 8 ]	21.18	0.0695	26.9527	0.8714	33.42	0.0901	25.5576	0.8565
TETO-VFI	17.39	0.0401	26.7525	0.9093	19.89	0.0501	25.3575	0.8923

Table 6: Comparison on HQ-EVFI.

Methods	AJ↑	$\delta^{x}_{avg}$ ↑	OA↑
TETO	67.9	81.4	92.2
w/o $\mathcal{L}_{\text{flow}}$	66.3	79.9	92.1
w/o temporal augmentation	63.3	77.0	92.2
w/o $\mathcal{M}_{\mathrm{obj}}$	60.3	74.6	91.5
w/ feature loss	63.7	76.6	91.9

Table 7: Ablation study on EVIMO2.

Methods	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
Model Component
TETO-VFI	10.11	0.0821	25.4232	0.8022
$-$ $\mathcal{M}_{\text{event}}$	11.06	0.0867	23.5839	0.8076
$-$ $\mathcal{L}_{\text{attn}}$	10.82	0.0869	23.3661	0.8014
$-$ $\mathcal{F}_{\text{warp}}$	13.88	0.1203	21.9624	0.7573
Motion Estimator

Table 8: Ablation study on BS-ERGB.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Methods	AJ↑	$\delta^{x}_{avg}$ ↑	OA↑
TETO	67.9	81.4	92.2
w/o ConcentNet	56.5	69.4	91.6

Appendix

Table 1: Architecture design ablation on EVIMO2.

Figure 1: Layer and head ablation for matching property.

Figure 2: Visualization of Event Motion Mask $\mathcal{M}_{\text{event}}$ .

Figure 3: OATS evaluation pipeline. Motion regions are extracted solely from events, and then query trajectories are evaluated based on whether they remain within the corresponding object masks over time.

Scene Name	Model	BS-ERGB
Scene Name	Model	$\text{OATS}_{\text{0}}$ ↑	$\text{OATS}_{\text{1}}$ ↑	$\text{OATS}_{\text{2}}$ ↑	$\text{OATS}_{\text{4}}$ ↑	$\text{OATS}_{\text{8}}$ ↑	$\text{OATS}_{\text{16}}$ ↑	$\text{OATS}_{\text{avg}}$ ↑
ball_05	ETAP	0.4538	0.4677	0.4840	0.5095	0.5518	0.6234	0.5150
ball_05	TETO	0.7131	0.7321	0.7542	0.7872	0.8357	0.8976	0.7867
ball_06	ETAP	0.6553	0.6684	0.6831	0.7076	0.7456	0.8085	0.7114
ball_06	TETO	0.7602	0.7747	0.7927	0.8209	0.8633	0.9168	0.8214
basket_08	ETAP	0.6785	0.6851	0.6930	0.7068	0.7314	0.7694	0.7107

Appendix

Table 2: Per-scene OATS comparison on BS-ERGB dataset.

Figure 4: Visualization of SAM3 video segmentation masks.

Method	Training data	Total recording	InivTAP			DrivTAP			Mean $\delta^{x}_{avg}$ ↑
Method	Training data	Total recording	AJ↑	$\delta^{x}_{avg}$ ↑	OA↑	AJ↑	$\delta^{x}_{avg}$ ↑	OA↑	Mean $\delta^{x}_{avg}$ ↑
E2Vid-PIPs++ [ 66 ]	-	-	-	14.3	-	-	30.4	-	22.4
E2Vid-Chrono [ 34 ]	-	-	10.4	17.9	68.8	8.1	16.9	68.7	17.4
E2Vid-CoTracker3 [ 32 ]	-	-	12.8	19.4	56.6	18.6	27.0	89.4	23.2
ETAP [ 22 ]	synthetic	5 h	12.8	22.3	86.3	13.5	27.8	68.1	25.1
TAPFormer-E [ 41 ]	synthetic	6 h	20.9	29.8	79.0	28.9	37.6	90.2	33.7

Appendix

Table 3: Quantitative results on real-world TAP benchmark.

Figure 5: Scenes selected for the main quantitative results on HQ-EVFI. Scenes with highly dynamic motion are chosen to evaluate models under complex real-world motion.

Method	$\times$ 7
Method	FID $\downarrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$
TimeLens-XL [ 43 ]	24.09	0.0569	26.9754	0.8696
CBMNet-Large [ 36 ]	10.40	0.0781	31.3721	0.8855
TETO-VFI	5.73	0.0417	27.9414	0.9069

Appendix

Table 4: Video frame interpolation result on full HQ-EVFI dataset.

Methods	Inference time (sec)
VDM-EVFI [ 8 ]	2428.00
TETO	283.60 ( $\times$ 8.56)

Appendix

Table 5: Inference time comparison between diffusion based frame interpolation models. Inference time is measured on a single RTX 3090 GPU. TETO is 8.56 times faster than VDM-EVFI.

Figure 6: Visualization of TETO showing tracking beyond appearance.

Figure 8: Qualitative comparison of video frame interpolation.

Figure 9: Qualitative comparison of video frame interpolation.

Figure 10: Qualitative comparison of video frame interpolation ablations. (A) is the baseline model without any conditional signal, (B) adds $\mathcal{F}_{\text{warp}}$ , and (C) adds $\mathcal{L}_{\text{attn}}$ and adds $\mathcal{M}_{\text{event}}$ for final TETO-VFI. Each result corresponds to the respective entry in the ablation table.

Explore similar work

Motion-aware Event Suppression for Event Cameras
Feb 26, 2026Roberto Pellerito, Nico Messikommer, Giovanni Cioffi +2Event-Based Vision
Robotics and Perception Group, University of Zurich, Switzerland
Static in Frames, Dynamic in Events: Rethinking Features in Event Cameras as Motion Cues
Aug 11, 2026Hesam Araghi, Jan van Gemert, Nergis TomenOptical Flow Event-Based Vision
Computer Vision Lab, Delft University of Technology
Event-based Scene Synthesis via Inter-Frame Residual Alignment
Dec 19, 2025Jiyun Kong, Jun-Hyuk Kim, Jong-Seok LeeEvent-Based Vision Video Prediction
Yonsei University, Korea · Chung-Ang University, Korea

Feb 26, 2026J/K move · Enter open · S save

Motion-aware Event Suppression for Event Cameras

Roberto Pellerito, Nico Messikommer, Giovanni Cioffi, Marco Cannici+1

Robotics and Perception Group, University of Zurich, Switzerland

Event-Based Vision

Open paper AskPDF arXiv

Event cameras report asynchronously per-pixel brightness changes with microsecond latency, encoding dynamic visual information as a sparse stream of events. However, their extreme temporal resolution floods perception systems with entangled events from ego-motion and independently moving objects (IMOs), which existing solutions fail to efficiently decouple, relying instead on prohibitive dense 3D reconstructions or limited hand-tuned filters. In this work, we introduce the first framework for Motion-aware Event Suppression, which learns to filter events triggered by IMOs and ego-motion in real time. Our model jointly segments IMOs in the current event stream while predicting their future motion, enabling anticipatory suppression of dynamic events before they occur. Our lightweight architecture achieves 173 Hz inference on consumer-grade GPUs with less than 1 GB of memory usage, outperforming previous state-of-the-art methods on the challenging EVIMO benchmark by 67% in segmentation accuracy while operating at a 53% higher inference rate. Moreover, we demonstrate significant benefits for downstream applications: our method accelerates Vision Transformer inference by 83% via token pruning and improves event-based visual odometry accuracy, reducing Absolute Trajectory Error (ATE) by 13%.

Abstract

Explore similar work

Motion-aware Event Suppression for Event Cameras

Static in Frames, Dynamic in Events: Rethinking Features in Event Cameras as Motion Cues

Event-based Scene Synthesis via Inter-Frame Residual Alignment