HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

Mauricio Soroco; Yuqiu Liu; Zaid Tasneem; Francesco Pittaluga; Abhishek Aich; Wuyang Chen; Manmohan Chandraker; Ziyu Jiang

cs.CVOct 20, 2026

HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

Authors: Mauricio Soroco, Yuqiu Liu, Zaid Tasneem, Francesco Pittaluga, Abhishek Aich, Wuyang Chen, Manmohan Chandraker, Ziyu Jiang

Organizations: Simon Fraser University · NEC Labs America

Abstract

Instruction-guided video generative models offer a scalable solution for stress-testing autonomous driving systems by simulating diverse environmental conditions. However, driving scenes operate within a dense Operational Design Domain (ODD) governed by strict traffic semantics. When applied to global driving-condition edits, general-purpose instruction-guided video editors frequently corrupt these semantics, leading to structural hallucinations and ``temporal shock.'' This work distills and protects traffic semantics during video editing at three levels. First, we construct a hybrid dataset by combining temporally aligned real-world videos with synthetic edits, balancing foundational scene structure with unambiguous appearance signals. Second, during training, we adapt advantage conditioning from offline reinforcement learning to align the model using dense rewards. We extract dense, pixel-level rewards from the hybrid data to serve as a localized signal and actively steer generation toward structurally reliable pixels. Finally, our model mitigates temporal shock by introducing thinking frames, a transition mechanism that seamlessly reconciles static image conditioning with dynamic video context. Extensive experiments demonstrate that our method prevents the corruption of multi-actor dynamics, establishing state-of-the-art spatiotemporal consistency, yielding consistent user preference (39% relative improvement in content preservation and 12% in temporal consistency), and achieving a significant 5% gain in downstream road line segmentation and 7% gain in downstream multi-object detection.

Figures & tables

Figure 1: HorizonComposer maintains complex traffic semantics during global appearance editing (rows 1-3) using a careful design across data (Sec 3.2 ), model (Sec. 3.4 ), and training (Sec. 3.3 ). Beyond global transformations, our fidelity in traffic semantics allows HorizonComposer to seamlessly compound global and local edits for comprehensive scene simulation (row 4).

Figure 2: Dataset Construction and Pseudo-Labeling Pipeline. (Top) Realistic Dataset Development : Real-world data (Sec. 3.2.1 ) are matched via pose alignment, temporally filtered, and processed to generate editing instructions, forming real-world video pairs. (Bottom) Pseudo Dataset Development (Sec. 3.2.2 ): We synthesize driving scenes by editing the initial frame with an image prior, followed by a video generative model to propagate the edits and produce synthetic video pairs.

Figure 3: Fine-grained Reward Map Generation (Sec. 3.3.1 ): For each video pair, we combine frame-level video-editing metrics, pixel-level scene-change masks and driving-specific semantic checks to construct dense reward maps, which guide training toward reliable appearance edits while protecting safety-critical traffic semantics. Technical details are in App. H .

Panel of Figure 4 — Figure 4: Qualitative Comparison of Driving Scene Editing. We highlight key differences in semantic preservation and visual quality across methods. Target conditions are shown in text. Row order from top to bottom: input video, CCEdit, UniVideo, SyntheticPip, Ours. (a) SyntheticPip hallucinates a yellow traffic light from the initial image condition. (b) SyntheticPip introduces a new vehicle that suddenly disappears, breaking temporal consistency. (c) Only our method and UniVideo accurately retain the “STOP AHEAD” road marking. Overall, CCEdit regularly suffers from hazy artifacts and UniVideo distorts visual realism, whereas our approach maintains high-fidelity instruction alignment and strict content preservation.

	Temporal		Edit Align.			ODD Preservation				Quality
Model	FF- $\alpha\downarrow$	FF- $\beta\downarrow$	Succ. $\uparrow$	CLIP $\uparrow$	GPT $\uparrow$	Subj. $\uparrow$	Bg. $\uparrow$	Lane $\uparrow$	Car $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
In-Domain (ID) Evaluation
CC-Edit	+18.16	-78.85	-6.97	-4.46	-20.82	+5.09	+2.94	-65.95	-34.39	-2.01	-8.08
UniVideo	-54.30	-22.84	-13.02	-5.51	-21.39	+2.82	+0.92	-25.40	+2.73	+0.82	-12.57
SyntheticPip	–	–	–	–	–	–	–	–	–	–	–
Ours	+14.44	+0.58	-1.34	+0.16	-0.70	+1.70	+0.94	+20.15	+7.46	-1.49	-4.57

Table 1: Relative Improvement (%) over SyntheticPip. We report percentage improvements computed from mean scores relative to SyntheticPip. For lower-is-better metrics, improvement is

(b-m)/b\times 100

; for higher-is-better metrics, it is

(m-b)/b\times 100

, where

b

and

m

denote baseline and model means. Positive values indicate improvement. We highlight best and runner-up positive improvements on In-Domain (ID) and Out-of-Distribution (OOD) scenarios. Raw scores with standard deviations are provided in Tab. 5 in the appendix.

Model	Content Preservation	Instruction Following	Photorealism	Temporal Consistency	Overall
Ours	46%	36%	43%	38%	40%
SyntheticPip	33%	33%	33%	34%	33%
UniVideo	7%	7%	7%	7%	7%
CCEdit	5%	5%	6%	3%	5%
Tie	8%	20%	11%	19%	15%

Table 2: User Study. Preference rates across four evaluation criteria and overall preference pooled across criteria.

Figure 5: Combining Local and Global Appearance Edits. We apply HorizonComposer on top of HorizonForge [ 53 ] to combine an ego-motion edit with a global appearance edit. Given an initial scene, HorizonForge modifies the trajectory to synthesize a new vehicle maneuver (e.g., a right-turn lane change) and generates the corresponding base scene. HorizonComposer then applies global appearance edits–shifting the season to autumn while preserving the newly edited BEV trajectory and local semantics.

Method	mIoU	Pixel Acc	mAcc
Original	37.48	98.76	50.04
+ SyntheticPip	38.45	98.81	53.05
+ Ours	40.27	98.88	55.62

Table 3: Lane marking segmentation performance on SeRM.

Training Data	mAP $\uparrow$	mAP@50 $\uparrow$	mAP@75 $\uparrow$
Real-200	0.167	0.316	0.154
Real-200 + Generated-200	0.175	0.324	0.165
Relative improvement	+4.79%	+2.53%	+7.14%

Table 4: Downstream multi-object detection performance on BDD100K.

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Qualitative Comparison of Driving Scene Editing. We highlight key differences in semantic preservation and visual quality across methods. Target conditions are shown in text. Row order from top to bottom: input video, CCEdit, UniVideo, SyntheticPip, Ours. (a) SyntheticPip cannot preserve left turn lane markings. (b) Ours preserves the “BUS ONLY” road marking that SyntheticPip removes. (c) Our method successfully preserves the road turn signal that SyntheticPip erroneously removes. (d) Our method successfully preserves the crosswalk and the red car, which SyntheticPip erroneously removes. (e) Our method preserves the clearance height of the overpass. (f) Only UniVideo and Ours preserve the solid white line and turn signal. Overall, CCEdit regularly suffers from hazy artifacts and UniVideo distorts visual realism, whereas our approach maintains high-fidelity instruction alignment and strict Operational Design Domain (ODD) preservation.

In-Domain Evaluation (Boreas, nuScenes, Waymo)
	Temporal		Edit Align.			ODD Preservation				Quality
Model	FF- $\alpha\downarrow$	FF- $\beta\downarrow$	Succ. $\uparrow$	CLIP $\uparrow$	GPT $\uparrow$	Subj. $\uparrow$	Bg. $\uparrow$	Lane $\uparrow$	Car $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
CC-Edit	3.2811 [-1pt] $\pm$ 0.0238	0.9638 [-1pt] $\pm$ 0.0005	0.8685 [-1pt] $\pm$ 0.0128	0.2376 [-1pt] $\pm$ 0.0005	5.9436 [-1pt] $\pm$ 0.1320	0.9454 [-1pt] $\pm$ 0.0002	0.9710 [-1pt] $\pm$ 0.0000	0.1374 [-1pt] $\pm$ 0.0129	0.5103 [-1pt] $\pm$ 0.0118	0.4928 [-1pt] $\pm$ 0.0075	0.5667 [-1pt] $\pm$ 0.0030
UniVideo	6.186 [-1pt] $\pm$ 0.140	0.662 [-1pt] $\pm$ 0.002	0.812 [-1pt] $\pm$ 0.013	0.235 [-1pt] $\pm$ 0.001	5.901 [-1pt] $\pm$ 0.106	0.925 [-1pt] $\pm$ 0.001	0.952 [-1pt] $\pm$ 0.001	0.301 [-1pt] $\pm$ 0.003	0.799 [-1pt] $\pm$ 0.026	0.507 [-1pt] $\pm$ 0.002	0.539 [-1pt] $\pm$ 0.004
SyntheticPip	4.0092 [-1pt] $\pm$ 0.1992	0.5389 [-1pt] $\pm$ 0.0038	0.9336 [-1pt] $\pm$ 0.0071	0.2487 [-1pt] $\pm$ 0.0006	7.5063 [-1pt] $\pm$ 0.0380	0.8996 [-1pt] $\pm$ 0.0022	0.9433 [-1pt] $\pm$ 0.0006	0.4035 [-1pt] $\pm$ 0.0020	0.7778 [-1pt] $\pm$ 0.0070	0.5029 [-1pt] $\pm$ 0.0050	0.6165 [-1pt] $\pm$ 0.0089
Ours	3.4303 [-1pt] $\pm$ 0.0775	0.5358 [-1pt] $\pm$ 0.0065	0.9211 [-1pt] $\pm$ 0.0052	0.2491 [-1pt] $\pm$ 0.0001	7.4541 [-1pt] $\pm$ 0.0219	0.9149 [-1pt] $\pm$ 0.0008	0.9522 [-1pt] $\pm$ 0.0003	0.4848 [-1pt] $\pm$ 0.0030	0.8358 [-1pt] $\pm$ 0.0037	0.4954 [-1pt] $\pm$ 0.0025	0.5883 [-1pt] $\pm$ 0.0016

Appendix

Table 5: Quantitative Comparison on Driving Scene Editing. We evaluate models across temporal stability, instruction alignment, ODD preservation, and overall video quality. Values indicate means with standard deviations below. We report best and runner-up mean performances on In-Domain (ID) and Out-of-Distribution (OOD) scenarios.

	Temporal		Edit Align.			ODD Preservation				Quality
Model Variant	FF- $\alpha$ ↓	FF- $\beta$ ↓	Succ.↑	CLIP↑	GPT↑	Subj.↑	Bg.↑	Lane↑	Car↑	Aes.↑	Img.↑
SyntheticPip	3.4702	0.5285	0.9289	0.2308	7.4855	0.9018	0.9443	0.5774	0.7880	0.5465	0.6126
Finetune RGB only	3.3009	0.5505	0.9234	0.2353	7.4600	0.9038	0.9457	0.6006	0.8243	0.5456	0.5986
+ Depth Cond.	3.1225	0.5442	0.9227	0.2334	7.4970	0.9013	0.9433	0.5984	0.8308	0.5461	0.5985
+ Dense Rewards	3.3927	0.5225	0.9048	0.2307	7.4206	0.9011	0.9436	0.6266	0.8595	0.5482	0.6066
+ Temp. Aug. (Ours)	3.2185	0.5235	0.8762	0.2299	7.3836	0.9126	0.9525	0.6281	0.8558	0.5472	0.6030

Appendix

Table 6: Ablation Study. Impact of our methods across data (Sec. 3.2 ), training (Sec. 3.3 ) and model (Sec. 3.4 ) on temporal stability and ODD preservation on out-of-distribution data.

	Temporal		Edit Align.			ODD Preservation				Quality
Model Variant	FF- $\alpha$ ↓	FF- $\beta$ ↓	Succ.↑	CLIP↑	GPT↑	Subj.↑	Bg.↑	Lane↑	Car↑	Aes.↑	Img.↑
SyntheticPip	3.8055	0.5427	0.9375	0.2491	7.4934	0.9004	0.9427	0.4029	0.7859	0.4981	0.6067
Finetune RGB only	3.6594	0.5716	0.9569	0.2545	7.5885	0.9073	0.9485	0.4133	0.7917	0.5027	0.5998
+ Depth Cond.	3.4901	0.5585	0.9489	0.2525	7.5848	0.9019	0.9450	0.4276	0.8008	0.5018	0.5932
+ Dense Rewards	3.6197	0.5411	0.9333	0.2500	7.4897	0.8987	0.9426	0.4904	0.8262	0.4975	0.5979
+ Temp. Aug. (Ours)	3.3539	0.5428	0.9155	0.2492	7.4297	0.9156	0.9520	0.4845	0.8351	0.4928	0.5873

Appendix

Table 7: Ablation Study (In-Distribution). Impact of our methods across data (Sec. 3.2 ), training (Sec. 3.3 ) and model (Sec. 3.4 ) on temporal stability and ODD preservation on in-distribution data.

	Temporal		Edit Align.			ODD Preservation				Quality
Data	FF- $\alpha$ ↓	FF- $\beta$ ↓	Succ.↑	CLIP↑	GPT↑	Subj.↑	Bg.↑	Lane↑	Col.↑	Aes.↑	Img.↑
Ours (Pseudo-only)	3.3004	0.5402	0.9048	0.2342	7.4230	0.9119	0.9520	0.5615	0.8296	0.5348	0.5975
Ours (Real-only)	3.4974	0.5215	0.8998	0.2309	7.3903	0.9137	0.9540	0.6246	0.8337	0.5536	0.6135
Ours	3.2185	0.5235	0.8762	0.2299	7.3836	0.9126	0.9525	0.6281	0.8558	0.5472	0.6030

Appendix

Table 8: Ablation on Data Mixture. We evaluate training exclusively on synthetic edits (Pseudo-only) versus real-world data (Real-only), compared to our hybrid approach. Results are computed on the OOD test data. Best results are grey bold ; runner-ups are light grey .

	Temporal		Edit Align.			ODD Preservation				Quality
Data Variant	FF- $\alpha$ ↓	FF- $\beta$ ↓	Succ.↑	CLIP↑	GPT↑	Subj.↑	Bg.↑	Lane↑	Car↑	Aes.↑	Img.↑
Ours (Pseudo-only)	-2.54	-3.20	+3.27	+1.85	+0.53	-0.07	-0.06	-10.61	-3.06	-2.27	-0.91
Ours (Real-only)	-8.67	+0.37	+2.69	+0.43	+0.09	+0.13	+0.16	-0.56	-2.59	+1.16	+1.74
Ours	–	–	–	–	–	–	–	–	–	–	–

Appendix

Table 9: Relative Improvement (%) over Hybrid Data. We report percentage improvement compared to our hybrid dataset for temporal stability, instruction alignment, ODD preservation, and overall video quality showing best and runner-up improvements. Results are computed on the OOD test data.

Dataset	Native unit	Native length	Paired videos	Target diversity	Supervision
Boreas [ 4 ]	Driving logs	Variable-length routes	No	Natural weather/season variation	Raw sensor data, poses
nuScenes [ 5 ]	1,000 scenes	20s per scene	No	Natural urban conditions	3D detection/tracking annotations
Waymo [ 49 ]	1,150 segments	20s / 200 frames per segment	No	Natural driving conditions	2D/3D perception annotations
nuPlan [ 28 ]	Driving logs	Large-scale variable-length logs	No	Multi-city driving scenarios	Planning annotations
Ours	Editing pairs	45 frames per clip	Yes	Weather, time, season	Edited target video, reference image

Appendix

Table 10: Comparison between original autonomous-driving datasets and our generated video-editing dataset. Existing datasets provide raw driving logs or perception/planning annotations, while our dataset provides paired source/target videos with explicit editing supervision.

Split	Boreas	nuScenes	Waymo	nuPlan	Total
Train	9,406	9,496	9,446	–	28,348
ID Test	50	50	50	–	150
OOD Test	–	–	–	150	150

Appendix

Table 11: Video counts in our generated driving-video editing dataset.

Dataset	Weather	Time of Day	Season	Total Videos
Train-Boreas	7510	7529	7106	9406
Train-nuScenes	7559	7643	7139	9496
Train-Waymo	7590	7598	7018	9446
ID Test-Boreas	42	42	40	50
ID Test-nuScenes	42	41	40	50
ID Test-Waymo	42	42	40	50

Appendix

Table 14: Number of videos editing each condition type. Counts are not mutually exclusive because one clip may edit multiple attributes.

Dataset	1-cat.	2-cat.	3-cat.	Total Videos
Train-Boreas	1097	3879	4430	9406
Train-nuScenes	1122	3903	4471	9496
Train-Waymo	1086	3960	4400	9446
ID Test-Boreas	4	18	28	50
ID Test-nuScenes	3	21	26	50
ID Test-Waymo	3	20	27	50

Appendix

Table 15: Number of conditions (weather, season, and time of day) edited simultaneously. 1-cat. , 2-cat. , and 3-cat. indicate whether one, two, or all three attributes are changed.

	Temporal		Edit Align.			ODD Preservation				Quality
Variant	FF- $\alpha$ ↓	FF- $\beta$ ↓	Succ.↑	CLIP↑	GPT↑	Subj.↑	Bg.↑	Lane↑	Car↑	Aes.↑	Img.↑
SyntheticPip (baseline)	–	–	–	–	–	–	–	–	–	–	–
only content	+5.55	+0.15	-3.39	+0.45	-0.20	+1.15	+0.89	+7.70	+7.85	-0.29	-2.01
only instruction	+4.84	+0.53	-4.12	+0.06	-0.77	+1.21	+0.88	+8.14	+7.98	+0.87	-0.59
only temporal	+5.19	+0.13	-3.33	+0.06	-1.41	+1.23	+0.84	+7.79	+8.73	+0.35	-1.18
Ours	+7.25	+0.94	-5.67	-0.37	-1.36	+1.19	+0.87	+8.78	+8.61	+0.14	-1.57

Appendix

Table 17: Rewards: Relative Improvement over Baseline. We evaluate the impact of individual components (content, instruction, temporal) compared to our full model and the SyntheticPip baseline. We report percentage improvement on OOD data compared to the SyntheticPip baseline. Best and runner-up improvements are highlighted.

	Temporal		Edit Align.			ODD Preservation				Quality
Variant	FF- $\alpha$ ↓	FF- $\beta$ ↓	Succ.↑	CLIP↑	GPT↑	Subj.↑	Bg.↑	Lane↑	Col.↑	Aes.↑	Img.↑
SyntheticPip	3.4702	0.5285	0.9289	0.2308	7.4855	0.9018	0.9443	0.5774	0.7880	0.5465	0.6126
only content	3.2777	0.5277	0.8973	0.2318	7.4703	0.9122	0.9527	0.6219	0.8498	0.5449	0.6003
only instruction	3.3022	0.5257	0.8906	0.2309	7.4279	0.9127	0.9527	0.6244	0.8509	0.5512	0.6090
only temporal	3.2901	0.5278	0.8980	0.2309	7.3800	0.9129	0.9522	0.6223	0.8568	0.5483	0.6054
Ours	3.2185	0.5235	0.8762	0.2299	7.3836	0.9126	0.9525	0.6281	0.8558	0.5472	0.6030

Appendix

Table 18: Rewards: Component Ablation. We evaluate the impact of individual components (content, instruction, temporal) compared to our full model and the SyntheticPip baseline on OOD data. Best results are grey bold ; runner-ups are light grey .

Figure 8: Qualitative comparison on SeRM lane marking segmentation. Each column shows one target scene, with rows corresponding to the original image, ground-truth annotation overlay, SyntheticPip-augmented training data, and our augmented data.

Method	FF- $\alpha$ $\downarrow$	FF- $\beta$ $\downarrow$	Succ. $\uparrow$	CLIP $\uparrow$	Lane $\uparrow$	Car $\uparrow$	Subj. $\uparrow$	Bg. $\uparrow$	Aes. $\uparrow$	Img. $\uparrow$
Scalar reward	3.3313	0.5619	0.9205	0.2494	0.4078	0.8068	0.9141	0.9528	0.4795	0.5785
Dense reward (ours)	3.3539	0.5428	0.9155	0.2492	0.4845	0.8351	0.9156	0.9520	0.4928	0.5873
Relative change	+0.68%	-3.40%	-0.54%	-0.08%	+18.81%	+3.51%	+0.16%	-0.08%	+2.77%	+1.52%

Appendix

Table 19: Dense versus scalar reward conditioning on the in-distribution evaluation set. Relative changes are computed with respect to the scalar-reward baseline; negative changes indicate improvements for FF-

\alpha

and FF-

\beta

Temporal Consistency in Video Generation

Temporal consistency in video generation studies how models preserve coherent motion, identities, scenes, and events across frames and long sequences.

See all 375 papers

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes
Apr 6, 2026Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem +5Text-Guided Image Editing Autonomous Driving Scenario Generation
Simon Fraser University · NEC Labs America
RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos
Jun 15, 2026Zhenhua Wu, Yun Pang, Mingkun Chang +43D Gaussian Splatting Sim-to-Real Transfer
1Sun Yat-sen University · 2Shanghai Innovation Institute · 3Shenzhen Loop Area Institute
DriveCtrl: Conditioned Sim-to-Real Driving Video Generation
May 14, 2026Haonan Zhao, Yiting Wang, Jingkun Chen +3Autonomous Driving Datasets Synthetic Data Augmentation
University of Warwick · 2Northwestern Polytechnical University · 3Queen Mary University of London

Apr 6, 2026J/K move · Enter open · S save

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes

Mauricio Soroco, Francesco Pittaluga, Zaid Tasneem, Abhishek Aich+4

Simon Fraser University · NEC Labs America

Text-Guided Image Editing Autonomous Driving Scenario Generation Domain Generalization

Open paper AskPDF arXiv

Ensuring safety in autonomous driving requires scalable generation of realistic, controllable driving scenes beyond what real-world testing provides. Yet existing instruction guided image editors, trained on object-centric or artistic data, struggle with dense, safety-critical driving layouts. We propose HorizonWeaver, which tackles three fundamental challenges in driving scene editing: (1) multi-level granularity, requiring coherent object- and scene-level edits in dense environments; (2) rich high-level semantics, preserving diverse objects while following detailed instructions; and (3) ubiquitous domain shifts, handling changes in climate, layout, and traffic across unseen environments. The core of HorizonWeaver is a set of complementary contributions across data, model, and training: (1) Data: Large-scale dataset generation, where we build a paired real/synthetic dataset from Boreas, nuScenes, and Argoverse2 to improve generalization; (2) Model: Language-Guided Masks for fine-grained editing, where semantics-enriched masks and prompts enable precise, language-guided edits; and (3) Training: Content preservation and instruction alignment, where joint losses enforce scene consistency and instruction fidelity. Together, HorizonWeaver provides a scalable framework for photorealistic, instruction-driven editing of complex driving scenes, collecting 255K images across 13 editing categories and outperforming prior methods in L1, CLIP, and DINO metrics, achieving +46.4% user preference and improving BEV segmentation IoU by +33%. Project page: https://msoroco.github.io/horizonweaver/

Abstract

Temporal Consistency in Video Generation

Explore similar work

HorizonWeaver: Generalizable Multi-Level Semantic Editing for Driving Scenes

RealityBridge: Bridging Editable 3D Gaussian Splatting Driving Simulations and Real-World Videos

DriveCtrl: Conditioned Sim-to-Real Driving Video Generation