FSDAM: Few-Shot Driving Attention Modeling via Vision-Language Coupling
Organizations: Texas Tech University, Lubbock, TX, USA · Purdue University, West Lafayette, IN, USA · Towson University, Towson, MD, USA
Abstract
Understanding not only where drivers look but also why their attention shifts is essential for interpretable human-AI collaboration in autonomous driving. Driver attention is not purely perceptual but semantically structured. Thus attention shifts can be learned through minimal semantic supervision rather than dense large-scale annotation. We present FSDAM (Few-Shot Driver Attention Modeling), a framework that achieves joint spatial attention prediction and structured explanation generation using 90 annotated examples. Our key insight is to decompose attention into an explicit reasoning representation, including scene context, current focus, anticipated next focus, and causal explanation, and to learn next-focus anticipation through minimal-pair supervision. To address task conflict and large sample requirements of existing models, and to mitigate task interference under limited data, we introduce a novel dual-pathway architecture in which separate modules handle spatial prediction and caption generation. In addition, we use a training-only vision-language alignment mechanism that injects semantic priors into spatial learning without increasing inference complexity, mitigating task interference under few-shot training. Despite extreme data scarcity, FSDAM achieves competitive performance in gaze prediction, and generates coherent, context-aware structural reasoning for improved interpretability. The model further demonstrates strong zero-shot generalization across multiple driving benchmarks. These results suggest that semantically grounded attention modeling enables data-efficient learning and provides a scalable path toward explainable driver attention systems in data-constrained environments.
Figures & tables
| Method | Training Data | CC | KL | SIM | AUC-J | AUC-B | NSS |
|---|---|---|---|---|---|---|---|
| Fully Supervised (Full BDD-A training set) | |||||||
| U²-Net [ 44 ] | Full BDD-A | 0.56 | 1.50 | 0.47 | 0.94 | 0.88 | 3.95 |
| MINet [ 42 ] | Full BDD-A | 0.46 | 10.30 | 0.16 | 0.89 | 0.82 | 3.67 |
| DBNet [ 52 ] | Full BDD-A | 0.57 | 1.30 | 0.41 | 0.95 | 0.91 | 4.41 |
| DeepLabV3 [ 6 ] | Full BDD-A | 0.47 | 9.62 | 0.21 | 0.97 | 0.75 | 2.56 |
| LLada ∗ [ 62 ] | Full BDD-A | 0.60 | 1.16 | 0.47 | – | – | – |
| Method | Training Data | DADA-2000 | DR(eye)VE | W3D | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CC | KL | SIM | AUC-J | AUC-B | NSS | CC | KL | SIM | AUC-J | AUC-B | NSS | CC | KL | SIM | AUC-J | AUC-B | NSS | ||
| U²-Net [ 44 ] | Full BDD-A | 0.42 | 2.18 | 0.37 | 0.91 | 0.82 | 2.73 | 0.57 | 1.52 | 0.45 | 0.90 | 0.82 | 3.20 | 0.44 | 2.10 | 0.37 | 0.90 | 0.81 | 2.81 |
| MINet [ 42 ] | Full BDD-A | 0.32 | 10.45 | 0.14 | 0.82 | 0.73 | 2.02 | 0.44 | 8.87 | 0.36 | 0.84 | 0.80 | 2.65 | 0.35 | 10.51 | 0.14 | 0.83 | 0.77 | 2.54 |
| DBNet [ 52 ] | Full BDD-A | 0.41 | 1.89 | 0.29 | 0.93 | 0.85 | 3.08 | 0.48 | 1.79 | 0.29 | 0.91 | 0.85 | 3.71 | 0.47 | 1.77 | 0.33 | 0.93 | 0.86 | 3.50 |
| DeepLabV3 [ 6 ] | Full BDD-A | 0.42 | 10.26 | 0.19 | 0.96 | 0.73 | 2.23 | 0.67 | 8.78 | 0.47 | 0.91 | 0.84 | 2.74 | 0.53 | 9.70 | 0.32 | 0.95 | 0.81 | 2.35 |
| FSDAM (Ours) | BDD-A-90 (few-shot) | 0.44 | 1.64 | 0.33 | 0.92 | 0.86 | 2.95 | 0.61 | 0.77 | 0.54 | 0.91 | 0.84 | 4.04 | 0.53 | 1.27 | 0.44 | 0.91 | 0.87 | 3.50 |
| Method | Training Regime | Normal Driving | Safety-Critical Situation | Traffic Accident | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU | METEOR | ROUGE | CIDEr-R | BLEU | METEOR | ROUGE | CIDEr-R | BLEU | METEOR | ROUGE | CIDEr-R | ||
| Fully Trained on W3D | |||||||||||||
| GazeXplain * [ 9 ] | Full-data (W3D) [ 70k samples] | 0.31 | 0.30 | 0.22 | 0.42 | 0.19 | 0.29 | 0.37 | 0.55 | 0.17 | 0.20 | 0.44 | 0.66 |
| LLada * [ 62 ] | Full-data (W3D) [ 70k samples] | 0.44 | 0.36 | 0.58 | 0.96 | 0.44 | 0.38 | 0.59 | 1.23 | 0.38 | 0.32 | 0.52 | 1.00 |
| Zero-shot and In-Context Models | |||||||||||||
| Qwen-VL [ 3 ] | Zero-shot (no training) | 0.10 | 0.19 | 0.28 | 0.34 | 0.19 | 0.21 | 0.29 | 0.13 | 0.08 | 0.21 | 0.29 | 0.12 |
| Scenario Type | Frame Pairs | Percentage |
|---|---|---|
| Signalized Intersection | 14 | 15.6% |
| Unsignalized Intersection | 12 | 13.3% |
| Lane Change | 13 | 14.4% |
| Pedestrian Crossing | 11 | 12.2% |
| Residential Area | 10 | 11.1% |
| Normal Driving | 12 | 13.3% |
| Method | Training Regime | Normal Driving | Safety-Critical Situation | Traffic Accident | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BLEU | METEOR | ROUGE | CIDEr-R | BLEU | METEOR | ROUGE | CIDEr-R | BLEU | METEOR | ROUGE | CIDEr-R | ||
| Fully Trained on W3D | |||||||||||||
| GazeXplain * [ 9 ] | Full-data (W3D) [ 70k samples] | 0.31 | 0.30 | 0.22 | 0.42 | 0.19 | 0.29 | 0.37 | 0.55 | 0.17 | 0.20 | 0.44 | 0.66 |
| LLada * [ 62 ] | Full-data (W3D) [ 70k samples] | 0.44 | 0.36 | 0.58 | 0.96 | 0.44 | 0.38 | 0.59 | 1.23 | 0.38 | 0.32 | 0.52 | 1.00 |
| Zero-shot and In-Context Models | |||||||||||||
| Qwen-VL [ 3 ] | Zero-shot (no training) | 0.10 | 0.19 | 0.28 | 0.34 | 0.19 | 0.21 | 0.29 | 0.13 | 0.08 | 0.21 | 0.29 | 0.12 |
| Method | CC | KL | SIM | AUC-J | AUC-B | NSS |
|---|---|---|---|---|---|---|
| FSDAM (Seed 1) | 0.60 | 1.13 | 0.43 | 0.96 | 0.91 | 4.10 |
| FSDAM (Seed 2) | 0.58 | 1.17 | 0.42 | 0.95 | 0.90 | 3.98 |
| FSDAM (Seed 3) | 0.61 | 1.11 | 0.44 | 0.96 | 0.91 | 4.15 |
| Mean Std | 0.60 0.01 | 1.14 0.03 | 0.43 0.01 | 0.96 0.01 | 0.91 0.01 | 4.08 0.09 |
| KL | CC | SIM | NSS | AUC-B | AUC-J | |
|---|---|---|---|---|---|---|
| 8 | 1.74 | 0.41 | 0.27 | 2.32 | 0.890 | 0.910 |
| 16 | 1.73 | 0.42 | 0.28 | 2.37 | 0.890 | 0.910 |
| 24 | 1.39 | 0.53 | 0.37 | 3.64 | 0.890 | 0.960 |
| 32 | 1.38 | 0.54 | 0.35 | 3.52 | 0.913 | 0.962 |
| 40 | 1.17 | 0.58 | 0.43 | 4.09 | 0.907 | 0.964 |
| 50 | 1.16 | 0.58 | 0.41 | 4.07 | 0.917 | 0.955 |
| Metric | None | Normal Driving | Construction Zone | Imminent Hazard | Lane Change | Pedestrian Crossing | Signalized Intersection | Unsignalized Intersection |
|---|---|---|---|---|---|---|---|---|
| KL | 1.13 | 1.15 | 1.21 | 1.13 | 1.18 | 1.13 | 1.16 | 1.24 |
| CC | 0.60 | 0.59 | 0.58 | 0.60 | 0.58 | 0.60 | 0.59 | 0.57 |
| SIM | 0.43 | 0.42 | 0.40 | 0.45 | 0.42 | 0.44 | 0.43 | 0.39 |
| NSS | 4.10 | 3.96 | 3.85 | 4.16 | 3.91 | 4.09 | 4.06 | 3.77 |
| Metric | Full | ||
|---|---|---|---|
| KL | 1.27 | 1.24 | 1.13 |
| CC | 0.53 | 0.54 | 0.60 |
| SIM | 0.43 | 0.43 | 0.43 |
| NSS | 3.71 | 3.73 | 4.10 |
| Metric | 100 samples | 13K samples |
|---|---|---|
| BLEU | 0.28 | 0.37 |
| METEOR | 0.31 | 0.38 |
| ROUGE | 0.25 | 0.32 |
| CIDEr | 0.29 | 0.49 |
| Criterion | Score |
|---|---|
| Causal correctness | |
| Visual grounding | |
| Driving relevance |
| Category | CC | SIM |
|---|---|---|
| Normal driving | 0.58 | 0.47 |
| Safety-critical situation | 0.59 | 0.48 |
| Traffic accident | 0.44 | 0.36 |
| Selection Strategy | CC | SIM |
|---|---|---|
| Random frames | 0.54 0.03 | 0.39 0.02 |
| High-saliency frames | 0.56 0.02 | 0.40 0.02 |
| KL-mined (Ours) | 0.60 0.01 | 0.43 0.01 |