SignReasoner: Compositional Reasoning for Complex Traffic Sign Understanding via Functional Structure Units

Ruibin Wang; Zhenyu Lin; Xinhai Zhao

cs.CVApr 12, 2026

SignReasoner: Compositional Reasoning for Complex Traffic Sign Understanding via Functional Structure Units

Authors: Ruibin Wang, Zhenyu Lin, Xinhai Zhao

Organizations: Huawei Noah’s Ark Lab

Abstract

Accurate semantic understanding of complex traffic signs-including those with intricate layouts, multi-lingual text, and composite symbols-is critical for autonomous driving safety. Current models, both specialized small ones and large Vision Language Models (VLMs), suffer from a significant bottleneck: a lack of compositional generalization, leading to failure when encountering novel sign configurations. To overcome this, we propose SignReasoner, a novel paradigm that transforms general VLMs into expert traffic sign reasoners. Our core innovation is Functional Structure Unit (FSU), which shifts from common instance-based modeling to flexible function-based decomposition. By breaking down complex signs into minimal, core functional blocks (e.g., Direction, Notice, Lane), our model learns the underlying structural grammar, enabling robust generalization to unseen compositions. We define this decomposition as the FSU-Reasoning task and introduce a two-stage VLM post-training pipeline to maximize performance: Iterative Caption-FSU Distillation that enhances the model's accuracy in both FSU-reasoning and caption generation; FSU-GRPO that uses Tree Edit Distance (TED) to compute FSU differences as the rewards in GRPO algorithm, boosting reasoning abilities. Experiments on the newly proposed FSU-Reasoning benchmark, TrafficSignEval, show that SignReasoner achieves new SOTA with remarkable data efficiency and no architectural modification, significantly improving the traffic sign understanding in various VLMs.

Figures & tables

Figure 1: Visualizations of complex traffic sign understanding of VLMs. Existing VLMs, including Qwen2.5-VL-72B [ 2 ] , InternVL3-78B [ 42 ] , and Gemini 2.5-Pro, demonstrate a significant inability to correctly comprehend traffic signs. Their failures primarily stem from misidentifying the number or type of directional arrows, or failing to accurately associate the U-Turn prohibition symbol with the correct lane. In contrast, our proposed SignReasoner bases on models with only 7B parameters, achieving fully accurate understanding

Figure 2: Illustration of Functional Structure Units (FSUs) for modeling traffic signs. Four kinds of FSUs are shown, including Notice, Direction, Construction and Lane. Notably, a traffic sign can either be decomposed into multiple FSUs with same or different functions.

Figure 3: Illustration of the Hierarchical Key-Value Schema . The “Sign-1,2,3” tags correspond to images in Fig. 2 for reference.

Figure 4: Illustration of the SignReasoner overall pipeline. The FSU Modeling (Sect. 3.1 ) decomposes the traffic sign multiple into FSUs, which would be concatenated with captions generated from base VLMs for Iterative Caption-FSU Distillation (Sect. 3.2 ). Finally, the distilled models would go through the FSU-GRPO Training (Sect. 3.3 ) to further boost reasoning capacity.

Zero-Shot Performance
Model	TrafficSignEval-Caption					TrafficSignEval-Structure
Model	Direction	Notice	Lane	Const.	Avg.	Direction	Notice	Lane	Const.	Avg.
InternVL3-8B	55.88	57.14	53.40	50.00	54.07	51.43	63.64	54.84	57.14	55.38
InternVL3-78B	55.88	61.90	74.76	57.14	68.02	68.57	63.64	61.29	71.43	63.59
Qwen2.5-VL-7B	47.06	66.67	42.72	64.29	48.26	28.57	45.45	36.29	64.29	37.95
Qwen2.5-VL-72B	64.71	71.43	63.11	71.43	65.12	40.00	68.18	50.81	71.43	52.31

Table 1: Comparing caption and structure protocols under TrafficSignEval.

FSU Design			SignReasoner
Hierar.	Open	Closed	Qwen2.5-VL-7B	Pangu-MM-7B
	$\checkmark$	$\checkmark$	80.27	79.48
$\checkmark$	$\checkmark$		82.89	82.14
$\checkmark$	$\checkmark$	$\checkmark$	83.59	83.08

Table 2: Ablation on FSU Design.

Caption-FSU	Iterative	SignReasoner
Format	Step	Qwen2.5-VL-7B	Pangu-MM-7B
	0	74.87	78.26
$\checkmark$	1	77.95	81.03
$\checkmark$	2	83.08	83.59
$\checkmark$	3	82.37	83.06

Table 3: Ablation on Caption-FSU Iterative Distillation.

Mixed Rewards			SignReasoner
$R_{\text{C-FSU}}$	$R_{\text{FSU}}$	$R_{\text{TED}}$	Qwen2.5-VL-7B	Pangu-MM-7B
$\checkmark$			83.14	83.67
	$\checkmark$		83.08	83.59
$\checkmark$		$\checkmark$	83.65	83.82
	$\checkmark$	$\checkmark$	83.96	84.02
$\checkmark$	$\checkmark$	$\checkmark$	84.10	84.10

Table 4: Ablation on Mixed Rewards Design.

Figure 5: Visual comparison between SignReasoner and base VLMs.

Functions	Possible Keys
Lane	Turn, Location, Special Lane, Time, Date, Speed, Weight, Height, Other Information
Direction	Direction, Via, Destination, Traffic Status, Distance, Other Information
Construction	Construction Site, Detour Information, Other Information
Notice	Direction, License Plate, Vehicle Type, Time, Date, Road Range, Speed, Weight, Height, Other Information

Table 5: Summary of possible keys in each kind of FSU.

Prompt (

P

) / Model Responses

Captioning

\bm{P}_{\textbf{caption}}

: Please analyze this cropped traffic sign image in detail and generate a detailed caption describing the colors, shapes, and patterns of the signs, and explain their meanings. This part of the answer should be placed between the < caption > </ caption > tags. Responses: < caption >This traffic sign is blue in color and has the standard shape of a traffic direction sign, with a white crossroad pattern in the center. The sign displays the names and directions of three roads: 1. Fulong Road (Fulong Rd) is located at the top of the sign, indicating a straight-ahead direction. 2. Mingle Road (Mingle Rd) is on the left side of the sign, indicating a left turn. 3. Yangtaishan Road (Yangtaishan Rd) is on the right side of the sign, indicating a right turn. The color and shape of the traffic sign are designed to be simple and clear, using the white crossroad pattern to clearly indicate the options for going straight ahead or turning left or right. The road names are written in white font, creating a sharp contrast with the blue background, making the information easy to recognize and read. The primary purpose of the traffic sign is to help drivers choose the correct road direction, ensuring safe and efficient travel.</ caption >

FSU-Reasoning

\bm{P}_{\textbf{Cap-FSU}}

: Please provide the information on the traffic sign in a structured format as follows: 1. Provide the global attribute information of the traffic sign. The output key values include {Traffic Sign, Electronic Sign, Obstruction, Blur, Truncation}, with answers chosen from {Yes, No}. Provide the {Other Global Information} of the traffic sign, with answers selected from the traffic sign itself. 2. Provide the functional category of the traffic sign. The answer should be chosen from {Lane, Direction, Notice, Construction}. 3. Provide the number of functional information items on the sign (for lane information, this refers to the number of lanes). 4. List detailed explanations for each functional information item. For lane information, the output key values include {Electronic Sign, Turn, Location, Special Lane, Time, Date, Speed, Weight, Height, Other Information}. For direction information, the output key values include {Direction, Route, Destination, Traffic Status, Distance, Other Information}. For construction information, the output key values include {Construction Site, Detour Information, Other Information}. For notice information, the output key values include {Direction, License Plate, Vehicle Type, Time, Date, Road Range, Speed, Weight, Height, Other Information}. Only provide answers based on the text and symbols on the traffic sign, without adding any extra explanatory text. Place the structured response within the < FSU > </ FSU > tags. Responses: < FSU >{“Traffic Sign”: “Yes”, “Electronic Sign”: “No”, “Obstruction”: “Yes”, “Truncation”: “No”, “Blurriness”: “No”, “Function Type”: “Direction”, “Number of Direction Information”: “2”, “Direction Information 1”: {“Direction”: “Go Straight”, “Via”: “Li Yang Road”, “Destination”: “ [The Bund, Haining Road] ”}, “Direction Information 2”: {“Direction”: “Turn Right”, “Via”: “Li Yang Road”, “Destination”: “Obstruction”}}</ FSU >

Table 6: Example Prompts and Corresponding Model Responses for SignReasoner.

Prompt (

P

) / Model Responses

Caption-FSU

\bm{P}_{\textbf{FSU-Reason}}

: Please provide the information on the traffic sign in a structured format as follows: 1. Please analyze this cropped image of the traffic sign in detail and generate a detailed caption describing the colors, shapes, and patterns of the signs, and explain their meanings. This part of the answer should be placed between the < caption > </ caption > tags. 2. Based on the caption description, provide the global attribute information of the traffic sign. The output key values include {Traffic Sign, Electronic Sign, Obstruction, Blur, Truncation}, with answers chosen from {Yes, No}. Also, provide the {Other Global Information} of the traffic sign, with the answer selected from the traffic sign options. 3. Based on the caption description, determine the functional category of the traffic sign. The answer should be chosen from {Lane, Direction, Notice, Construction}. 4. Based on the caption description, indicate the number of functional information items on the sign (for lane information, this refers to the number of lanes). 5. Based on the caption description, list detailed explanations for each functional information item. For lane information, the output key values include {Electronic Sign, Turn, Location, Special Lane, Time, Date, Speed, Weight, Height, Other Information}. For direction information, the output key values include {Direction, Route, Destination, Traffic Status, Distance, Other Information}. For construction information, the output key values include {Construction Site, Detour Information, Other Information}. For notice information, the output key values include {Direction, License Plate, Vehicle Type, Time, Date, Road Range, Speed, Weight, Height, Other Information}. Only provide the corresponding answers based on the text and symbols on the traffic sign, without adding any extra explanatory text. The structured answers for sections 2-5 should be placed between the < FSU > </ FSU > tags. Responses: < caption >This traffic sign is blue in color and has a standard traffic sign shape with a white crossroad pattern in the center. The sign displays the names and directions of three roads: 1. Fulong Road (Fulong Rd) is located at the top of the sign, indicating a straight ahead direction. 2. Mingle Road (Mingle Rd) is on the left side of the sign, indicating a left turn. 3. Yangtaishan Road (Yangtaishan Rd) is on the right side of the sign, indicating a right turn. The color and shape of the traffic sign are designed to be simple and clear, using the white crossroad pattern to clearly indicate the options for going straight ahead or turning left or right. The road names are written in white font, creating a sharp contrast with the blue background, making the information easy to recognize and read. The primary purpose of the traffic sign is to help drivers choose the correct road direction, ensuring safe and efficient driving.</ caption >< FSU >“Traffic Sign”: “Yes”, “Electronic Sign”: “No”, “Blocked”: “No”, “Truncated”: “No”, “Blurred”: “No”, “Function Type”: “Direction”, “Number of Direction Information”: “3”, “Direction Information 1”: “Direction”: “Straight Ahead”, “Destination”: “ [Fulong Road, Fulong Rd] ”, “Direction Information 2”: “Direction”: “Left Turn”, “Destination”: “ [Mingle Road, Mingle Rd] ”, “Direction Information 3”: “Direction”: “Right Turn”, “Destination”: “ [Yangtaishan Road, Yangtaishan Rd] ”</ FSU >

Table 7: Example Prompts and Corresponding Model Responses for SignReasoner.

Figure 6: Visualizations of Traffic Sign Understanding in SignReasoner (Pangu-MM-7B).

Figure 7: Visualizations of Traffic Sign Understanding in SignReasoner (Pangu-MM-7B).

Figure 8: Flowchart of automatic evaluation algorithm.

Traffic Sign Detection

See all 10 papers

Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding
Jul 27, 2026Yueru Luo, Xu Yan, Changqing Zhou +5Autonomous Driving VLM Reasoning
Shenzhen-FNii, CUHK-SZ · HKUST(GZ) · Tencent T Lab +1
Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles
Aug 9, 2026Pedram MohajerAnsari, Amir Salarpour, Mert D. PeséAdversarial Robustness VLM Distillation
Clemson University, Clemson, SC, USA
Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding
May 20, 2026Lena Wild, Katie Z Luo, Marco PavoneVisual Reasoning Autonomous Driving
KTH Royal Institute of Technology · TRATON · Stanford University +1

Jul 27, 2026J/K move · Enter open · S save

Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding

Yueru Luo, Xu Yan, Changqing Zhou, Yiming Yang+4

Shenzhen-FNii, CUHK-SZ · HKUST(GZ) · Tencent T Lab+1

Autonomous Driving VLM Reasoning

Open paper AskPDF arXiv

Understanding and complying with traffic regulations is a safety-critical requirement for autonomous driving, yet remains challenging due to the diversity and context dependence of traffic signage. Importantly, regulation understanding is not a simple recognition task, but a reasoning problem: whether a rule applies depends on interpreting the sign in relation to the spatial layout of lanes and scene context. To support such reasoning, MapDR provide fine-grained annotations that link each traffic sign's regulatory rules to the specific lanes they govern. Existing methods, however, largely treat this as direct sequence prediction, ignoring the underlying reasoning that connects sign semantics and map structure. To address this limitation, we explicitly incorporate reasoning into this task and propose a framework that equips vision-language models (VLMs) with chain-of-thought (CoT) capabilities. We first design a scalable CoT curation pipeline that bootstraps rationales from a strong LLM through a two-round strategy and employs a VLM-based verifier to filter out incorrect cases, yielding a high-quality set of (CoT, answer) pairs. Building on this foundation, we adopt a two-stage training scheme: supervised fine-tuning (SFT) to teach rationale-to-answer generation, followed by GRPO reinforcement learning with answer-grounded, fine-grained rewards to further improve final answer accuracy. Extensive experiments on MapDR show that our approach significantly improves both interpretability and accuracy, establishing the first reasoning-based framework for regulation-aware autonomous driving.

Abstract

Traffic Sign Detection

Explore similar work

Reasoning to Regulate: Chain-of-Thought for Traffic Rule Understanding

Distilling Vision-Language Models for Robust Traffic Sign Perception in Autonomous Vehicles

Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understanding