Generative Spatiotemporal Data Augmentation

Jinfan Zhou; Lixin Luo; Sungmin Eum; Heesung Kwon; Jeong Joon Park

cs.CVDec 14, 2025

Generative Spatiotemporal Data Augmentation

Authors: Jinfan Zhou, Lixin Luo, Sungmin Eum, Heesung Kwon, Jeong Joon Park

Organizations: University of Chicago · University of Michigan, Ann Arbor · DEVCOM Army Research Laboratory

Abstract

We explore spatiotemporal data augmentation using video foundation models to diversify both camera viewpoints and scene dynamics. Unlike existing approaches based on simple geometric transforms or appearance perturbations, our method leverages off-the-shelf video diffusion models to generate realistic 3D spatial and temporal variations from a given image dataset. Incorporating these synthesized video clips as supplemental training data yields consistent performance gains in low-data settings, such as UAV-captured imagery where annotations are scarce. Beyond empirical improvements, we provide practical guidelines for (i) choosing an appropriate spatiotemporal generative setup, (ii) transferring annotations to synthetic frames, and (iii) addressing disocclusion - regions newly revealed and unlabeled in generated views. Experiments on COCO subsets and UAV-captured datasets show that, when applied judiciously, spatiotemporal augmentation broadens the data distribution along axes underrepresented by traditional and prior generative methods, offering an effective lever for improving model performance in data-scarce regimes.

Figures & tables

Figure 2: We use video diffusion models to expand spatially and temporally to the current scene. Thanks to the consecutive frames of the generated video, SAM 2 video segmentation model can be used to robustly, accurately and automatically generate segmentation masks and the bounding boxes for the generated frames.

Figure 3: Object recall and KID vs. number of synthetic frames. Recall increases sharply then saturates after

\sim

30 frames, showing diminishing returns in coverage. KID increases with frame count, but 1:1 sampling (red) stabilizes realism compared to unrestricted sampling (blue).

Augmentation	Raw	Instagen	ControlAug	Ours
Image Recall	88.60	97.14	98.62	98.98
Object Recall	72.83	78.42	86.46	88.56
KID $\downarrow$	$5\times 10^{-6}$	$5.5\times 10^{-3}$	$8.7\times 10^{-4}$	$\mathbf{6.5\times 10^{-4}}$

Table 1: Comparison of image and object recall under different augmentations. Results are computed using 8 generated images per original image with a 1:1 mixing ratio for all augmentation methods.

Figure 4: Disocclusion handling strategies. (a) Original image. (b) Spatially augmented image showing newly visible regions (pseudo-label example). (c–e) Illustration of tracking-based masking: (c) original frame with grid points, (d) augmented frame with tracked points, and (e) masked result excluding disoccluded areas. These visualizations highlight that disoccluded regions can be detected and masked, though the overall performance benefit is marginal.

Figure 5: Qualitative comparison of our spatiotemporal augmentation on COCO and VisDrone against baselines. Our method produces video frames conditioned on the input image, yielding coherent and expressive spatial(top row) and temporal(bottom row) variation compared to the conventional geometric augmentation. ControlAug [ 14 ] synthesizes images conditioned on the HED boundary map of the original image, preserving spatial structure but altering appearance. InstaGen [ 15 ] samples new images using a finetuned diffusion model guided by text prompts (e.g., “a photograph of person and bench,” or “a drone photograph containing car, pedestrian, and bicycle”).

Dataset	Method	mAP@50	mAP@50:95
COCO5k	Trad	36.3	24.8
	Trad + ControlAug (8x)	40.1 ( $+3.8$ )	27.6 ( $+2.8$ )
	Trad + InstaGen (8x)	40.3 ( $+4.0$ )	28.1 ( $+3.3$ )
	Trad + Ours (8x)	41.1 ( $+\textbf{4.8}$ )	28.6 ( $+\textbf{3.8}$ )
VisDrone	Trad	45.9	28.5
	Trad + ControlAug (8x)	48.5 ( $+2.6$ )	30.5 ( $+2.0$ )

Table 2: Augmentation Effectiveness Comparison. We compare our method against traditional augmentations, InstaGen [ 15 ] , and ControlAug [ 14 ] on COCO5k, VisDrone, and Semantic Drone, using the same number of generated frames (8x the original dataset size on COCO5k and VisDrone, 80x on Semantic Drone). Overall, our method outperforms the baselines.

	Raw	ControlAug [ 14 ]	InstaGen [ 15 ]
Base	24.8	27.6	28.1
+Ours	28.6	29.7	29.1
$\Delta$	+3.8	+ 2.1	+1.0

Table 3: Combining with Existing Augmentations . mAP on COCO 5k when we apply spatiotemporal augmentations along with existing augmentations. Further performance gains can be obtained by incorporating spatiotemporal augmentations.

Dataset size	5k	10k	15k	20k
Raw	19.2	25.4	28.4	30.2
Augmented	22.7	27.9	30.6	32.2
$\Delta$	+3.5	+2.5	+2.2	+2.0

Table 4: Instance segmentation on COCO. Our generative augmentation consistently improves instance segmentation performance compared to traditional augmentation alone, yielding gains across different dataset sizes.

Dataset	Method	mAP@50	mAP@50:95
COCO5k	Spatial (8x)	41.1 (+4.8)	28.6 (+3.8)
COCO5k	Temporal (8x)	38.7 (+2.4)	26.7 (+1.9)
VisDrone	Spatial (40x)	49.2 (+3.3)	31.1 (+2.6)
VisDrone	Temporal (40x)	49.6 (+3.7)	31.3 (+2.8)

Table 5: Effects of Spatial vs. Temporal augmentation on COCO5k and VisDrone performance. We use DimensionX [ 49 ] for spatial generation and LTX-Video-13B [ 19 ] for temporal generation.

Figure 6: Dataset size vs. mAP on 3 datasets. Detection results of YOLOv11x on COCO under different dataset sizes

	raw	w/o 1:1 mixing	w/ 1:1 mixing
mAP@50	36.3	39.9 (+3.6)	41.1 (+4.8)
mAP@50:95	24.8	27.9 (+3.1)	28.6 (+3.8)

Table 6: Ablation on 1:1 mixing . Comparison of mAP on COCO5k with and without 1:1 mixing between raw and augmented data. Enforcing balanced data mixing further improves performance.

Data	#frames per img	mAP@50	mAP@50:95
Raw	–	36.3	24.8
Sample every 3 frames	8	40.7	28.2
Sample every 4 frames	8	41.1	28.5
Sample every 5 frames	8	41.1	28.6
Use first 36 frames	36	40.5	28.3

Table 7: Effect of different frame selection strategies on COCO5k performance. Frame selection starts from the first frame of the generated video. The 1:1 mixing ratio is employed for all experiments.

Removed samples	w/o filtering	5%	10%	20%
mAP@50	41.1	40.5	41.4	40.4
mAP@50:95	28.6	28.3	29.0	28.2

Table 8: CLIP-based filtering. Performance optimization by filtering augmented data on COCO5k.

	$k$ -NN		Linear
Augmentation	Top-1	Top-5	Top-1	Top-5
Trad Aug	23.13	38.81	27.86	49.28
Gen Aug	25.87	42.20	29.90	52.39
$\Delta$	+2.74	+3.39	+2.04	+3.11

Table 9: DINO results . ImageNet classification performance of a ViT-S pretrained with DINO using generative augmentation. Our augmentation consistently improves both

k

-NN and linear evaluation accuracy.

Appendix figures & tables11 assets

Supplementary material from the paper’s appendix.

Appendix

\text{flows},\;\text{visconf}\leftarrow\text{Model.forward\_sliding}(V)

Appendix

Algorithm 1 Dense Disocclusion Mask Computation

Dataset size	5k	10k	15k	20k
Raw	19.8	30.0	34.1	36.1
Augmented	24.5 (+4.7)	31.7 (+1.7)	35.3 (+1.2)	38.9 (+2.8)

Appendix

Table 10: Comparison of mAP@50:95 for RT-DETR-x trained on raw vs. augmented data as training set size increases.

Figure 7: Detection results of RT-DETR-x on COCO under different dataset sizes

Dataset size	5k	10k	15k	20k	25k	30k	35k
Raw	18.7	24.2	27.5	29.8	31.2	32.7	33.6
Augmented	20.7	26.3	29.3	31.4	32.6	33.4	34.2

Appendix

Table 11: Comparison of mAP@50:95 for YOLO11n trained on raw vs. augmented data as training set size increases.

Figure 8: Performance vs Model Capacity. Detection results of YOLO11x and YOLO11n on COCO under different dataset sizes

Conf thr.	0.5	0.6	0.7	0.8	w/o pseudo-labels
mAP@50	41.1	40.7	41.4	40.7	41.1
mAP@50:95	28.7	28.3	28.9	28.3	28.6

Appendix

Table 12: Effect of confidence threshold on pseudo-label to performance on COCO5k.

Dataset	Method	mAP@50	mAP@50:95
COCO5k	w/o masking	41.1	28.6
	polygon masking	40.4	28.0
	pixel-wise masking	39.6	27.4
VisDrone	w/o masking	49.6	31.3
	polygon masking	49.6	31.3
	pixel-wise masking	49.5	31.1

Appendix

Table 13: Ablation on disocclusion masking. We compare object detection performance with and without polygon and pixel-wise masking on COCO5k and VisDrone datasets.

Training set size	Augmented	mAP@50	mAP@50:95
100		71.1	39.4
100	✓	74.5 (+3.4)	44.1 (+4.7)
200		81.9	51.3
200	✓	86.8 (+4.9)	56.8 (+5.5)
300		89.8	61.3
300	✓	90.7 (+0.9)	63.4 (+2.1)

Appendix

Table 14: Comparison of mAP on subsets of Semantic Drone [ 10 ] . We evaluate YOLO11x detectors trained with and without augmented data using our method on subsets size of 100, 200, and 300, respectively. All models are evaluated on the remaining 100 images.

Data	Pretrained	mAP@50	mAP@50:95
Raw		45.9	28.5
Augmented (40x)		49.6 (+3.7)	31.3 (+2.8)
Raw	✓	48.2	30.1
Augmented (40x)	✓	50.2 (+2.0)	31.7 (+1.6)

Appendix

Table 15: Effectiveness on Pretrained Models . We finetune a COCO-pretrained YOLO11x model on VisDrone with and without our augmentation method. Our model can still improve performance significantly in the finetuning setting. Traditional augmentations are applied to all experiments.

Data	mAP@50	mAP@50:95
Raw	45.9	28.5
Resize (8x)	48.2 (+2.3)	30.3 (+1.8)
Random crop (8x)	48.9 (+3.0)	30.7 (+2.2)
Random crop (40x)	49.6 (+3.7)	31.3 (+2.8)

Appendix

Table 16: Ablation on random cropping for VisDrone. We train YOLO11x on augmented VisDrone datasets using frames from videos generated by LTXV-13B from resized full images or from randomly cropped regions, respectively. The table reports mAP@50 and mAP@50:95, showing that random cropping consistently improves detection performance, with larger numbers of crops yielding the strongest gains.

Method	ControlAug	InstaGen	Ours-LTX	Ours-DimX
mAP	30.5	29.8	31.3	31.1
$\Delta$	+2.0	+1.3	+2.8	+2.6
Gen time/src-img	169 s	45 s	110 s	1210 s

Appendix

Table 17: VisDrone mAP vs. wall-clock cost per source image. Our method attains the largest improvement at a competitive cost. For our entries, the reported time covers generating 5 videos per source image and includes SAM-2 mask propagation (4 s per video).

Data Augmentation

Data augmentation expands or transforms training examples to improve learning, data efficiency, and robustness to distribution shifts.

See all 249 papers

AnyView: Synthesizing Any Novel View in Dynamic Scenes
Jan 23, 2026Basile Van Hoorick, Dian Chen, Shun Iwase +7Multi-View Consistency Camera-Conditioned Video Generation
Toyota Research Institute, CA, USA · Amazon Web Services, CA, USA
A Real-Calibrated Synthetic-First Data Engine
May 10, 2026Yukang Shen, Zhiguo Liu, Yingshu Li +1Synthetic Data Augmentation Synthetic Data Generation
Kennesaw State University, Marietta, GA, USA. · Department of Computer Science, Georgia State University, Atlanta, GA, USA.
Exploring the Role of Synthetic Data Augmentation in Controllable Human-Centric Video Generation
Apr 23, 2026Yuanchen Fei, Yude Zou, Zejian Kang +3Synthetic-to-Real Domain Adaptation Synthetic Data Augmentation
Hunan University · Shanghai Jiaotong University · Westlake University +3

Jan 23, 2026J/K move · Enter open · S save

AnyView: Synthesizing Any Novel View in Dynamic Scenes

Basile Van Hoorick, Dian Chen, Shun Iwase, Pavel Tokmakov+6

Toyota Research Institute, CA, USA · Amazon Web Services, CA, USA

Multi-View Consistency Camera-Conditioned Video Generation Video Diffusion Models Multi-View Video Generation Dynamic Novel View Synthesis Temporal Consistency in Video Generation+1

Open paper AskPDF arXiv

Modern generative video models excel at producing convincing, high-quality outputs, but struggle to maintain multi-view and spatiotemporal consistency in highly dynamic real-world environments. In this work, we introduce

\textbf{AnyView}

, a diffusion-based video generation framework for

\textit{dynamic view synthesis}

with minimal inductive biases or geometric assumptions. We leverage multiple data sources with various levels of supervision, including monocular (2D), multi-view static (3D) and multi-view dynamic (4D) datasets, to train a generalist spatiotemporal implicit representation capable of producing zero-shot novel videos from arbitrary camera locations and trajectories. We evaluate AnyView on standard benchmarks, showing competitive results with the current state of the art, and propose

\textbf{AnyViewBench}

, a challenging new benchmark tailored towards

\textit{extreme}

dynamic view synthesis in diverse real-world scenarios. In this more dramatic setting, we find that most baselines drastically degrade in performance, as they require significant overlap between viewpoints, while AnyView maintains the ability to produce realistic, plausible, and spatiotemporally consistent videos when prompted from

\textit{any}

viewpoint. Results, data, code, and models can be viewed at: https://tri-ml.github.io/AnyView/

Abstract

Data Augmentation

Explore similar work

AnyView: Synthesizing Any Novel View in Dynamic Scenes

A Real-Calibrated Synthetic-First Data Engine

Exploring the Role of Synthetic Data Augmentation in Controllable Human-Centric Video Generation