Organizations: Fariborz Maseeh Department of Civil, Architectural, and Environmental Engineering, The University of Texas at Austin, Austin, TX 78712, USA · Department of Electrical and Computer Engineering, Duke University, Durham, NC 27708, USA
Existing roadside traffic datasets support perception, forecasting, and visual question answering, but they do not evaluate counterfactual video generation, in which a selected actor is modified and the generated future should remain consistent with road topology and unrelated traffic. We introduce TrafficImag, the first benchmark for counterfactual roadside traffic video generation. TrafficImag combines a large-scale roadside dataset (9,022 annotated images, 7,043 deduplicated video clips, and 31,145 actor-centered history-future samples) with an executable protocol that supports behavior reasoning, intervention-aware image editing, and conditional video generation. Each intervention is represented as an actor-level program describing the target actor, intended behavior, legal route, interaction order, and temporal constraints, enabling a unified evaluation interface across heterogeneous foundation models. TrafficImag evaluates four complementary validity dimensions: initial-state correctness, route and behavior validity, interaction consistency, and non-target preservation, and considers an end-to-end counterfactual successful only when all four are satisfied. Across state-of-the-art foundation models, the strongest reasoner reaches 80.4% macro F1, the complete condition interface raises end-to-end success from 23.3% to 55.0% for the best generator. Oracle studies further show that conditional video execution is the primary remaining bottleneck. TrafficImag provides a reproducible benchmark for evaluating and diagnosing counterfactual traffic video generation beyond perceptual video quality.
Figures & tables
Figure 1: Overview of the TrafficImag benchmark.
Figure 2: Day and night counterfactual anchors for vehicle removal and insertion. Each row contains the marked input followed by four editor outputs. Removal is often completed without disturbing distant traffic. Insertion exposes larger differences in target location, scale, lane contact, and illumination, especially at night.
Figure 3: Matched 8-second rollouts for a programmed left-turn counterfactual. T1 is the controlled vehicle and T2 should remain stationary. Seedance completes the turn while maintaining T2 most consistently in this example. Sora retains the scene but progresses more slowly and does not complete the requested route within the shown horizon. Veo introduces additional traffic and exhibits target/path drift.