Generalist Meets Specialists: An Evolving Multimodal Agent with Persistent Tool Orchestration for Traffic Scenario Understanding
Organizations: Xingchen AGI Lab, China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd. · Tsinghua University · Karlsruher Institut für Technologie · Fudan University
Abstract
Traffic scenario understanding requires combining fine-grained visual evidence with spatial and temporal reasoning. Tool-augmented vision-language models (VLMs) can use specialized perception tools, but effective collaboration faces two challenges: terminal rewards provide limited guidance for intermediate decisions and error recovery, while repeated planning adds unnecessary cost across similar tasks. Echoing the fast-slow dichotomy of human cognition, we propose StarTV-Agent, which combines policy learning with experience evolution to support adaptive reasoning when needed and efficient reuse of prior experience when possible. During training, Role-Sensitive Process Optimization (RSPO) combines process advantages with terminal advantages, while Weakness-Driven Demonstration Learning (WDL) provides validated teacher demonstrations targeting the student's weak questions. During inference, Lightweight Adaptive Workflow Reuse (LAWR) enables a fast path by selecting suitable workflows and binding task-specific arguments and resource references, while a slow ReAct-based path handles tasks without suitable workflows. We further design label-free feedback based on execution completion, workflow adoption, and cost savings to guide workflow reuse and memory updates. Across three challenging benchmarks, StarTV-Agent achieves up to 8.0% relative performance improvement, while reducing token consumption by up to 52.1%.
Figures & tables
| Positioning | Counting | Motion | Class | Existence | Overall | |||||||
| Method | Params | E | H | E | H | E | H | E | H | E | H | |
| DeepSeek-V4.1-Flash | 552B | 33.3 | 42.4 | 72.7 | 60.6 | 72.7 | 62.6 | 55.6 | 40.4 | 85.9 | 70.7 | 59.7 |
| Kimi K3 | 2.8T | 40.4 | 40.4 | 82.8 | 65.7 | 68.7 | 54.5 | 60.6 | 53.5 | 91.9 | 69.7 | 62.8 |
| Qwen3.8-Max | 2.4T | 55.6 | 38.4 | 80.8 | 66.7 | 71.7 | 75.8 | 73.7 | 74.7 | 90.9 | 72.7 | 70.1 |
| GPT-5.5 | N/D | 49.5 | 41.4 | 76.8 | 63.6 | 69.7 | 71.7 | 65.7 | 50.5 | 86.9 | 69.7 | 64.5 |
| Qwen3-VL-8B-Instruct | 8B | 20.2 | 57.6 | 68.7 | 65.7 | 67.7 | 30.3 | 54.6 | 46.5 | 87.9 | 59.6 | 55.9 |
| Method | Params | OPT | TIR | MUL | Mean |
|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | 552B | 40.1 | 40.1 | 46.2 | 42.1 |
| Kimi K3 | 2.8T | 42.3 | 43.7 | 48.4 | 44.8 |
| Qwen3.8-Max | 2.4T | 49.1 | 50.2 | 52.0 | 50.4 |
| GPT-5.5 | N/D | 48.0 | 46.6 | 48.7 | 47.8 |
| Qwen3-VL-8B-Instruct | 8B | 42.3 | 40.9 | 43.7 | 42.3 |
| STEP3-VL-10B | 10B | 36.2 | 36.9 | 37.6 | 36.9 |
| Method | Params | Exact Match | ROUGE-L | BLEU-4 | METEOR | CIDEr | SPICE | GPT-Score |
|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash | 552B | 0.189 | 0.367 | 0.104 | 0.194 | 1.210 | 0.128 | 0.703 |
| Kimi K3 | 2.8T | 0.155 | 0.348 | 0.068 | 0.202 | 1.016 | 0.111 | 0.706 |
| Qwen3.8-Max | 2.4T | 0.237 | 0.392 | 0.061 | 0.212 | 1.045 | 0.095 | 0.811 |
| GPT-5.5 | N/D | 0.145 | 0.355 | 0.104 | 0.208 | 1.135 | 0.122 | 0.752 |
| Qwen3-VL-8B-Instruct | 8B | 0.221 | 0.359 | 0.077 | 0.168 | 1.105 | 0.097 | 0.643 |
| STEP3-VL-10B | 10B | 0.111 | 0.302 | 0.102 | 0.212 | 1.037 | 0.141 | 0.595 |
| Variant | RL | RSPO | Demo SFT | Weak Sel. | RoadSceneVQA GPT-Score | Traffic-VQA OA |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B-Instruct | ✗ | ✗ | ✗ | – | 0.620 | 41.3 |
| Terminal-only RL | ✓ | ✗ | ✗ | – | 0.682 | 42.0 |
| w/o WDL | ✓ | ✓ | ✗ | – | 0.692 | 44.4 |
| w/o Weak Sel. | ✓ | ✓ | ✓ | ✗ | 0.684 | 41.2 |
| StarTV-Agent | ✓ | ✓ | ✓ | ✓ | 0.694 | 44.8 |
| Variant | ReAct | LAWR | RoadSceneVQA GPT-Score | Traffic-VQA OA |
|---|---|---|---|---|
| VLM-only | ✗ | ✗ | 0.643 | 42.3 |
| w/o LAWR | ✓ | ✗ | 0.694 | 44.8 |
| StarTV-Agent | ✓ | ✓ | 0.694 | 45.7 |
| Stage | Avg. Input Tokens | Avg. Output Tokens | Time (s) |
| Traffic-VQA | |||
| Planner | 4,972 | 166 | 2.393 |
| Tool Selector | 7,008 | 337 | 4.598 |
| Executor | 4,866 | 53 | 0.984 |
| Verifier | 6,617 | 302 | 4.294 |
| Summary | 573 | 55 | 0.720 |
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Method | Seatbelt | Red light | Phone use | Solid line | Lane use | Wrong-way | Overall | |||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | Acc. | F1 | |
| Qwen3.8-Flash-Next | 0.5226 | 0.6631 | 0.5179 | 0.6519 | 0.7337 | 0.7337 | 0.5464 | 0.6573 | 0.5484 | 0.5385 | 0.8108 | 0.7835 | 0.6107 | 0.6670 |
| Qwen3.8-27B | 0.5155 | 0.6667 | 0.5298 | 0.6728 | 0.8191 | 0.8468 | 0.5269 | 0.6271 | 0.5848 | 0.6226 | 0.7818 | 0.7209 | 0.6166 | 0.6793 |
| DeepSeek-V4.1-Flash | 0.4949 | 0.4762 | 0.5235 | 0.6816 | 0.5567 | 0.3857 | 0.5707 | 0.6867 | 0.5368 | 0.5749 | 0.5852 | 0.3348 | 0.5455 | 0.5213 |
| GLM-5.3-Flash | 0.5170 | 0.5824 | 0.6250 | 0.7179 | 0.4796 | 0.5952 | 0.5194 | 0.6601 | 0.4791 | 0.6383 | 0.7009 | 0.5455 | 0.5538 | 0.6166 |
| Qwen3-VL-32B | 0.5025 | 0.6633 | 0.5900 | 0.6985 | 0.5779 | 0.7021 | 0.5500 | 0.6751 | 0.8350 | 0.8514 | 0.8075 | 0.8300 | 0.6558 | 0.7440 |
| Answer Type | Reward and Scoring Rule |
|---|---|
| Binary, choice, and short VQA | Exact match ( if correct, otherwise). Task-aware matching avoids substring checks for binary answers. |
| Grounding | , where is the spatio-temporal grounding error. The original error is also recorded for analysis. |
| Open VQA and caption | Normalized exact matches receive ; otherwise, a semantic judge assigns a score in based on the reference answer. |
| Scope | Prompt |
|---|---|
| Shared | “Judge this one already-made Student decision. Score its decision quality in [0,1]. Return only JSON with one numeric key named score. Do not suggest a new action. Assess action legality, including validation_error, as well as task relevance.” |
| Planner | “Is the plan relevant to the question, grounded in visible input, and executable with available tools?” |
| Tool Selector | “Is the selected next tool and subgoal appropriate for the current evidence and available tools?” |
| Executor | “Are parameters and resource references legal, available now, and suitable for the selected tool?” |
| Verifier | “Does the continue/stop decision faithfully reflect actual tool output and evidence?” |
| Summary | “Is the answer supported by observed evidence, does it answer the user question and follow the public task contract?” |
| Tool | Required Input | Returned Evidence |
| Visual reasoning and attribute extraction | ||
| General_Generator_Tool | Image resources and a prompt | Free-form, image-grounded descriptions or reasoning results. |
| Multi_Image_VQA_Tool | Ordered images and a question | Joint visual reasoning results, including temporal and cross-image comparisons. |
| VLM_Attribute_Extraction_Tool | Images or object crops and target attributes | Structured attribute–value pairs for each image. |
| general_ocr | Image | Text blocks with content, confidence scores, and polygon coordinates. |
| Draw_Tool | Image and geometric primitives (box/point/line/polygon) | Annotated image with drawn overlays. |
Tool-Using Vision-Language Agents
Tool-using vision-language agents combine visual and language models with external tools to gather evidence, act, and solve multi-step tasks.
See all 151 papers