Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA

Nguyen Hoai Thuong Bui; Thanh Nguyen Vo; Trinh Tra Giang Nguyen; Ha Duc Bui

cs.CVSep 16, 2026

Sim-to-Real Traffic Scene Understanding by Decoupling Semantics from Caption Generation with V-JEPA

Authors: Nguyen Hoai Thuong Bui, Thanh Nguyen Vo, Trinh Tra Giang Nguyen, Ha Duc Bui

Organizations: Ho Chi Minh City University of Technology and Engineering (HCMUTE), Ho Chi Minh City, Vietnam

Abstract

Track 2 of the AI City Challenge 2026 requires both visual question answering (VQA) and traffic event description generation under a challenging synthetic-to real domain shift. Existing vision-language approaches often entangle semantic understanding with language generation, making them susceptible to hallucination and inconsistent reasoning across event phases. In this work, we propose a decoupled semantic understanding framework that first resolves predefined traffic questions into structured semantic facts and subsequently uses these facts to guide caption generation. A frozen V-JEPA encoder extracts predictive scene representations, while a lightweight Llama-based predictor produces answers for VQA queries. To improve reliability, we introduce a training-free structured refinement mechanism that exploits statistical priors, inter-question relationships, and temporal event consistency to correct prediction errors. The refined semantic facts are then provided to Qwen3-VL-8B to generate pedestrian and vehicle descriptions for each traffic event. Experimental results on the official 2026 AI City Challenge Track 2 benchmark show that the proposed method achieves 87.09% VQA accuracy and an overall S2 score of 60.0853, ranking first among all participating teams. These results demonstrate that predictive world representations combined with structured semantic refinement enable more accurate and reliable traffic understanding, leading to higher-quality lan guage generation.

Figures & tables

Figure 1: Answer prediction. The selected camera video is encoded by a frozen V-JEPA 2.1. The video representation is combined with the question and mapped into the answer embedding space. Cosine scores provide the InfoNCE objective during training and the candidate scores for structured decoding during inference. Only the modules shown in orange are optimized on SynWTS.

No.	Category	Queried attributes
1	Environment	Pedestrian appearance and scene, road, traffic, and obstacle attributes
2	Pedestrian orientation	Pedestrian body orientation and direction of travel
3	Pedestrian position	Relative position between the pedestrian and vehicle
4	Pedestrian to vehicle distance	Relative distance between the pedestrian and vehicle
5	Pedestrian gaze	Line of sight, vehicle awareness, and visual status
6	Pedestrian behavior	Coarse and fine-grained pedestrian actions

Table 1: Semantic question categories. The benchmark questions are assigned to nine categories according to the attributes they query.

Figure 2: Refinement and caption generation. The independent VQA answers pass through the training-free refinement (statistical prior, question relation, and phase transition), and the refined answers become facts that, together with the frames, guide Qwen3-VL-8B to write the pedestrian and vehicle captions.

Figure 3: Structure of the caption generation prompt. The refined VQA facts and the sampled frames are both fed to Qwen3-VL-8B, which returns the pedestrian and vehicle captions as a single JSON object.

Component	Setting	Value
VQA predictor	Training	AdamW, 16 epochs
	Learning rate / schedule	$1.2\times 10^{-4}$ / cosine decay / 300 warmup steps
	Batch size	4
	Dropout (visual / question)	0.12 / 0.08
	InfoNCE temperature	0.07
Caption adapter	Training	LoRA, 3 epochs

Table 2: Implementation settings.

Rank	Team	S2	BLEU-4	METEOR	ROUGE-L	CIDEr	Acc. (%)
1	Latent Painter - UTE	60.0853	0.2798	0.4624	0.4969	0.8396	87.0918
2	UIT-Kitchen	57.3307	0.2658	0.4276	0.4595	0.5833	84.3812
3	KZ6	56.7949	0.2540	0.4241	0.4471	0.7691	83.5370
4	MobilityAI	55.5901	0.2532	0.4233	0.4466	0.7601	81.2042
5	Snow leopard	55.5768	0.2438	0.4247	0.4446	0.7243	81.5152

Table 3: Official public leaderboard. The five highest-ranked teams are shown.

Configuration	Acc. (%)	$\Delta$
Model alone (no refinement)	81.28	–
$+$ statistical prior	81.64	$+0.36$
$+$ question relation	83.27	$+1.63$
$+$ phase transition (temporal)	87.09	$\mathbf{+3.82}$

Table 4: Effect of the refinement. Each row adds one correction on top of the rows above.

\Delta

is the gain over the previous row. All scores come from the official evaluation server.

Configuration	Acc. (%)	BLEU-4	METEOR	ROUGE-L	CIDEr	S2
Qwen3-VL-8B (base)	87.0918	0.2221	0.4131	0.4388	0.5846	57.7025
Qwen3-VL-8B $+$ LoRA	87.0918	0.2798	0.4624	0.4969	0.8396	60.0853

Table 5: Effect of LoRA on caption generation performance on the test set.

VLMs for Autonomous Driving

Vision-language models for autonomous driving connect camera and sensor data with language to interpret scenes, predict hazards, and plan actions.

See all 192 papers

Oct 22, 2026

StateBridge: Phase- and Track-Grounded Reasoning for Synthetic-to-Real Traffic Video Captioning and VQA

Traffic safety videos provide rich evidence for understanding interactions between pedestrians and vehicles, but synthetic-to-real evaluation makes this setting challenging because models must learn from simulated scenes and generalize to real-world traffic. In this work, we present StateBridge, a traffic video-language framework designed to operate under this synthetic-only training protocol. Our approach uses phase-aware clip construction, pedestrian and vehicle track summaries, and structured prompts to ground vision-language models in stable traffic evidence rather than appearance alone. We fine-tune public vision-language models on synthetic captioning and visual question answering examples, then combine their outputs through fact-locked caption fusion and evidence-diverse VQA consensus to improve consistency across actor roles, behavioral phases, and question types. StateBridge achieves an official S2 score of 55.4679 on AI City Challenge 2026 Track 2 using only synthetic training data. These results demonstrate the effectiveness of structured evidence transfer for traffic safety captioning and visual question answering.

Blessing Agyei Kyem, Richard Dzinyela, Joshua Kofi Asamoah +5

North Dakota State University, Fargo ND 58102, USA · Sustainable Mobility and Advanced Research in Transportation (SMART) Lab · Texas A&M University, College Station 77840, TX, USA +1

Video Captioning Visual Question Answering Synthetic-to-Real Domain Adaptation VLMs for Autonomous Driving Temporal Video Understanding

Ask PDF

Aug 13, 2026

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations

Traffic video understanding has become an important problem in intelligent transportation, as road videos provide direct evidence for accidents, violations, and interactions between vehicles and vulnerable road users. A useful system should explain how a traffic event develops, why it happens, and when the relevant interaction occurs, yet this remains difficult for multimodal large language models (MLLMs) because traffic videos contain sparse events and varied viewpoints. We introduce UniTraffic-Agent, the MR-CAS solution for Track~3 of the 10th AI City Challenge, which includes Traffic Anomaly Reasoning (TAR) and two out-of-domain evaluations: FETV for fisheye traffic events and PSI-VQA for pedestrian intention reasoning. UniTraffic-Agent follows an observe--reason--act--verify workflow that samples timestamped visual evidence, reasons over all questions from the same clip in one request, and converts responses through task-specific action adapters. On the official Public leaderboards, MR-CAS ranks 16th on TAR with a score of 0.5780, 2nd on FETV with 0.4884, and 4th on PSI-VQA with 64.4161. The code is available at https://github.com/Roclp/UniTraffic-Agent.

Peng Li, Qianqian Xu, Shilong Bao +2

School of Computer Science and Technology, University of Chinese Academy of Sciences (UCAS), Beijing, China · State Key Laboratory of AI Safety, Institute of Computing Technology (ICT), Chinese Academy of Sciences (CAS), Beijing, China · Beijing Academy of Artificial Intelligence (BAAI), Beijing, China +1

Pedestrian Intention Prediction Intelligent Transportation Systems Video Reasoning VLM Reasoning Multimodal Agents Traffic Anomaly Detection+1

Ask PDF

Oct 26, 2026

Reason or Recite: Complementary Specialists with an LLM Judge for Traffic Anomaly Reasoning

Road anomalies such as collisions, violations, and sudden hazards demand more from video understanding than detection: a practical system must verify what happened, describe the scene, and explain the cause. We find these competences pull in opposite directions for vision-language models: closed-form questions reward precise, evidence-grounded reasoning, while open-ended answers are scored by similarity to reference text, where matching the annotation style matters as much as reasoning correctly. We present Reason-or-Recite (RoR), a two-specialist framework for the Traffic Anomaly Reasoning track of the 2026 AI City Challenge. RoR (1) rewrites the official chain-of-thought annotations into a structured, temporally grounded schema and LoRA-tunes a 9B vision-language model on the enhanced traces; (2) fine-tunes a 4-bit-quantized 27B model on answers alone, absorbing the annotation style; and (3) selects between the two candidates per question with a training-free LLM judge. RoR ranks first on the in-domain leaderboard with mean score 0.6788.

Mayur Kulkarni

Stellarview AI, Pune, India

Video Reasoning VLM Reasoning Traffic Anomaly Detection

Ask PDF

Abstract

VLMs for Autonomous Driving

Explore similar work

StateBridge: Phase- and Track-Grounded Reasoning for Synthetic-to-Real Traffic Video Captioning and VQA

UniTraffic-Agent: Unified Traffic Video Reasoning for AI City Challenge 2026 Track 3 with Two Out-of-Domain Evaluations

Reason or Recite: Complementary Specialists with an LLM Judge for Traffic Anomaly Reasoning