cs.CVOct 22, 2026

SAGE: A hybrid vision-language and geometry framework for work-zone pathway prediction under temporary traffic control

Authors: Jiaxi Liu, Keshu Wu, Weizhe Tang, Bofeng Cao, Sikai Chen, Xiaopeng Li, Steven T. Parker, Bin Ran

Organizations: Department of Civil & Environmental Engineering, University of Wisconsin–Madison, Madison, 53706, Wisconsin, United States

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Temporary traffic control redirects vehicles through work zones, requiring automated vehicles to connect an interpretation of the visible devices to a continuous passing path. Geometric predictors can provide several plausible paths, yet selecting the one consistent with the temporary control remains difficult. We propose SAGE (Semantic-to-Action Grounded Execution), a read-decide-execute-check framework that separates path construction, path preference, and control compliance within a shared candidate space. The central idea is to use a vision-language model's (VLM's) trajectory as a preference over intact geometric candidates. A role-decoupled generator constructs the candidates, and rank fusion combines each candidate's geometric score with its agreement with the VLM trajectory. Supervision structured around the Manual on Uniform Traffic Control Devices (MUTCD) trains SAGE-VLM to state the lane status and hazard side before its passage decision and trajectory. The predicted hazard side enters image-space predicates derived from MUTCD provisions, which support a final intervention bounded in score and displacement. On 2,306 real-world ROADWork validation episodes, the candidates include a path within 10 px of the human demonstration in 87% of episodes, yet selection regret accounts for 53% of the geometric ranker's error. SAGE reduces average and final displacement errors by 8.3% and 12.0% relative to the ranker, mostly in the far field, and hard-constraint violations by 37%. The trajectory opinion improves all four tested generator-selector combinations. SAGE is comparable to state-of-the-art VLM modules in accuracy, with a faster-responding VLM, and outperforms four other adapted driving VLMs in displacement error and safety measures. Project page: https://hhjj233.github.io/SAGE-page/.

Figures & tables

Figure 1: Temporary-control-aware pathway prediction. (a) A geometric predictor selects a smooth path toward the coned-off side. (b) A VLM provides a scene interpretation and an imprecise trajectory. (c) SAGE uses the trajectory as a preference over intact geometric candidates and the structured reading as input to checks derived from MUTCD provisions. The paths are illustrative; green dots mark the observed prefix, orange boxes the detected cones, and the red box the work vehicle.
Figure 2: Overview of SAGE. Middle band: RD-YNet proposes 60 + 60 intact candidates, and a geometric ranker scores them. Upper band: SAGE-VLM states the lane status, hazard side, passage decision, goal, and trajectory; each candidate’s distance to the trajectory gives its opinion rank. Right: rank fusion with w=0.3w=0.3 , with a slopegraph of the choices of SAGE w/o VLM, SAGE w/o Z0, and SAGE. Lower band: MUTCD-derived predicates, which take the hazard side from the reading, drive the bounded intervention Z0; in this example, Z0 outputs the seventh fused candidate.
Figure 3: Candidate generation with RD-YNet on a San Antonio validation frame. The free-space branch receives the drivable and off-road maps, and the TTC-conditioned branch all six role maps; each samples 60 endpoints from its goal heatmap. All 10 candidates within 10 px of the demonstration come from the TTC-conditioned branch. Shading marks roles (teal, drivable; gray, off-road; amber, channel; blue, guidance); white lines are candidates, white dots the observed prefix, and green the human demonstration.
ProvisionRequirement (paraphrased)Predicate on CjC_{j}Evidence
§6B.06 ¶02The work space is closed to road users.RSI(Cj)>0\mathrm{RSI}(C_{j})>0Hard-role regions ∪\cup hard and activity detections
§6B.05 ¶02Redirected road users are guided to a new path by channelizing devices.CBC(Cj)>0\mathrm{CBC}(C_{j})>0Channel-role regions ∪\cup channelizing-device detections
§6B.05 ¶02; §6B.06 ¶02; §6L.06 ¶17, ¶19Traffic passes beside the closed side; arrow boards mark lane closures.Endpoint side =aside=a_{\mathrm{side}}SAGE-VLM’s reading
Table 1: MUTCD provisions instantiated as image-space predicates of the count Γ\Gamma in Eq. ( 10 ). RSI: restricted-space intrusion; CBC: channelization-boundary crossing.
Figure 4: From scene evidence to a checked pathway in a San Francisco work zone. (a) Detections and the observed prefix. (b) Author-constructed summary linking the role regions, detections, and the VLM’s hazard-side reading to the predicates. (c) Checks on the choices of SAGE w/o Z0 and SAGE. (d) Z0 replaces the choice of SAGE w/o Z0, which enters the restricted-space proxy, with an admissible candidate 7.21 px away. Green: human demonstration (evaluation only).
 Displacement (px)Metric frameSafetyControl
MethodADE ↓\downarrowFDE ↓\downarrowLat. err. (m) ↓\downarrowLane miss (%) ↓\downarrowCollision (%) ↓\downarrowHard viol. (%) ↓\downarrowScore (0–100) ↑\uparrowHazard side (%) ↓\downarrow
Constant velocity21.5159.760.1746.682.066.5793.1–
Generic planner14.0635.080.1333.781.012.4095.91.16
ROADWork Y-Net38.5875.990.2206.381.493.4586.94.17
Qwen2.5-VL-7B25.8164.900.1966.992.016.6691.80.54
Senna18.7354.240.1455.541.886.3693.31.24
Table 2: Validation results on 2,306 episodes; collision, hard violations, and the score use the 2,086 human-filtered episodes, and the hazard side the 1,295 described-side episodes (Section 4.2 ). Bold: best; underline: second best; –: not evaluated.
Figure 5: Error along the horizon and safety on safety-critical frames. (a) Mean displacement error at each future waypoint, with the far segment shaded. (b) Error reduction relative to SAGE w/o VLM (lines) and departure onset of the 594 turning episodes among all 2,313 (bars, right axis). (c, d) Hard-violation rate and safety score on the 137 safety-critical frames (135 with a valid SAGE-VLM output).
Figure 6: Candidate coverage and selection in a San Francisco work zone. (a) The 60 + 60 candidates and the human demonstration. (b) The choices of SAGE w/o VLM and SAGE. (c) Geometric and opinion ranks of every candidate; the dotted line marks the fused score of the choice of SAGE w/o Z0, which Z0 retains. (d) Cumulative candidate coverage on a logarithmic ADE axis, with both choices marked. Labels such as free:52 and obj:36 give the branch (free-space or TTC-conditioned) and the candidate’s index.
Device rolesLearned rankerEpoolE^{\mathrm{pool}} (px) ↓\downarrowPool within 10 px (%) ↑\uparrowADE (px) ↓\downarrow aloneΔ\Delta ADE (px) ↓\downarrow with TAOΔ\Delta ADE (px) ↓\downarrow with anchors
✗✗6.6385.514.06−- 0.73 [ −- 1.45, −- 0.06]−- 0.41 [ −- 1.18, ++ 0.37]
✗✓6.6385.512.79−- 0.88 [ −- 1.29, −- 0.52]−- 0.71 [ −- 1.29, −- 0.15]
✓✗6.0487.414.58−- 1.33 [ −- 2.20, −- 0.51]−- 1.24 [ −- 2.13, −- 0.40]
✓✓6.0487.412.85−- 1.11 [ −- 1.70, −- 0.51]−- 0.70 [ −- 1.34, ++ 0.04]
Table 3: Generation strength ×\times selection strength. Bold intervals exclude zero; the shaded row is SAGE w/o VLM.
  SAGE, ADE (px) ↓\downarrow 
Episodesnnw/o VLMw/o Z0Δ\Delta ADE (px) ↓\downarrow
All2,30612.8511.74−- 1.11 [ −- 1.70, −- 0.51]
Laterally departing47037.0533.24−- 3.81 [ −- 6.57, −- 0.94]
Not laterally departing1,8366.666.24−- 0.42 [ −- 0.54, −- 0.29]
Activity area1,29812.4311.33−- 1.10 [ −- 1.59, −- 0.67]
Transition area2379.818.40−- 1.41 [ −- 3.47, −- 0.29]
Table 4: ADE of SAGE w/o VLM and SAGE w/o Z0 by episode type and work-zone condition. Bold intervals exclude zero.
 SAGE-VLMQwen3.5-4B variant
Development subset (339 episodes)
Passage direction (%) ↑\uparrow91.292.6
Hazard side, balanced accuracy (%) ↑\uparrow60.573.0
Validation
ADE, executed directly (px) ↓\downarrow12.1515.35
Hazard side agrees (%) ↑\uparrow66.879.9
Table 5: Reading quality and the bounded intervention (Z0). Passage direction compares the decision’s direction (left, straight, right) with the one derived from the demonstration. Hazard-side agreement uses the 1,474 validation episodes whose description names a side; in the Z0 blocks, hard violations and the score use the human-filtered episodes, and the hazard side the 1,295 described-side episodes.
JudgmentBeforeAfterImproved ↑\uparrowWorsened ↓\downarrowPaired change (pp) ↓\downarrow
Corridor inconsistent24/8813/88165−- 12.50 [ −- 21.11, −- 4.40]
Visible contact25/8813/88164−- 13.64 [ −- 21.79, −- 6.02]
Table 6: Human review of the paths changed by Z0.
Figure 7: Qualitative validation cases, chosen from automatically ranked candidates. (a, b) The SAGE-VLM opinion selects a candidate close to the human demonstration that the geometric ranker missed. (c–f) Z0 replaces a choice of SAGE w/o Z0 that enters the restricted-space or channelization proxy (shaded) with a nearby candidate that does not; these panels zoom on the far half of the paths. (g–i) LightEMMA, Impromptu-VLA, and OneVL, executed directly, violate a hard constraint, while SAGE does not. The titles of (a)–(f) give the ADE before and after, with SAGE-VLM’s reading below each panel; the lines below (g)–(i) give the ADE of both methods.
Figure 8: Cases (d), (i), and (g) of Fig. 7 in the device-anchored ground frame. Left: front view with the paths and the work-zone proxies. Right: the same paths from above, with the ego car at the origin, the ±1.83\pm 1.83 -m lane tolerance around the demonstration, and the ±1.07\pm 1.07 -m vehicle footprint; a dark outline marks the object behind each reported overlap. The lateral axis is stretched about twofold, and shading marks distances beyond the rows that the metric frame evaluates (Section 4.2 ).
ComponentMedian (ms) ↓\downarrowp90 (ms) ↓\downarrow
Geometric layer
Role segmentation2.82.8
RD-YNet branch, 60 candidates258–260260–261
SAGE-VLM
Vision encoder3435
Reading and decision710867
Table 7: Per-frame latency at batch size 1 on one otherwise idle RTX 6000 Ada GPU. SAGE-VLM’s complete output is timed in a separate run and in the back-to-back run with OneVL.
Figure 9: Observable four-stage trace for one Denver work-zone frame: (a) scene and observed prefix, (b) rank fusion (TAO), and (c) bounded replacement (Z0), with cards 1–4 for the four stages. Read and Decide show recorded VLM output; Execute and Check show computed ranks and predicates. Rank fusion changes the choice of SAGE w/o VLM, C10, to that of SAGE w/o Z0, C5, and Z0, triggered by the RSI predicate, replaces C5 with C54. Ranks are one-based; candidate indices are zero-based.
More in this topic

Autonomous Driving Planning

Autonomous driving planning selects safe, feasible vehicle actions and trajectories amid traffic, uncertainty, road rules, and changing goals.

See all 384 papers

Explore similar work

CardsList

  1. WorkDrive: Roadwork Chain of Causation for Autonomous Driving

    Jul 16, 2026Tianyi Jiang, Wen Zhang, Sihan Yang +2VLMs for Autonomous DrivingAutonomous Driving Perception

    Peking University · Beijing, China · Xiaomi EV

  2. Vision-Language Work Zone Intelligence for Safety-Critical Speed Regulation of Mixed-Autonomy Vehicles in Dynamic Environments

    Jun 7, 2026Angel Martinez-Sanchez, Kianna Ng, Wesley Maia +7Autonomous Driving Perception

    UC Merced · Johns Hopkins · UC San Diego

  3. S-squared-VLA: Decoupling Semantic and Spatial Streams in Vision-Language-Action Models for Autonomous Driving

    Jul 15, 2026Jianguo Yu, Rukang Wang, Duanfeng Chu +3VLMs for Autonomous DrivingVision-Language-Action Models

    School of Mechanical and Electronic Engineering, Wuhan University of Technology, Wuhan, Hubei, China · Intelligent Transportation Systems Research Center, Wuhan University of Technology, Wuhan, Hubei, China · School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan, Hubei, China