Organizations: New York University Tandon School of Engineering, Brooklyn, NY 11201, USA · NYUAD Center for Artificial Intelligence and Robotics (CAIR) and Embodied AI and Robotics (AIR) Lab, New York University Abu Dhabi, Abu Dhabi 129188, UAE · NYU Grossman School of Medicine, NYU Langone Health, New York, NY 10016, USA · Harvard AI and Robotics Lab, Harvard University, Boston, MA 02114, USA
Vision-language models (VLMs) are rapidly progressing and offer promising capabilities for assistive technologies supporting persons with blindness or low vision. However, existing VLMs are primarily designed for general-purpose captioning and do not explicitly model human perceptual priorities, thereby limiting their ability to emphasize the most relevant information in a scene. To address this gap, we propose a salience-driven captioning framework that prioritizes scene elements according to their importance for human-centered assistance. We curate three salience-aware datasets, namely, Salience COCO, Salience Flickr, and Salience VizWiz, with object-level salience annotations designed to reflect the visual information most relevant to low vision users across different environments. Building on these datasets, we introduce Salience-LLaVA, a salience-aware VLM that incorporates salience cues to generate captions in which important elements are mentioned in the order of importance. Our work makes four main contributions. We build salience-aware datasets verified by low vision participants, propose Salience-LLaVA to describe objects in the order of importance, introduce SCMI to evaluate ordering accuracy, and deploy the system on assistive glasses to demonstrate real-world practicality. Code and datasets are available at: https://github.com/topo-focus/Topofocus
Figures & tables
Fig. 1: Comparison of image descriptions generated by the original LLaVA and Salient-LLaVA. The first row lists salient objects in order of importance, as determined by a human annotator, representing human commonsense prioritization. While the original LLaVA description does not fully align with human perception, Salient-LLaVA better reflects human-centered relevance by highlighting key elements earlier in the caption.
Fig. 2: An Illustration of the salience-aware captioning ground-truth generation process. The input image is processed to produce a salience map, from which salient objects are detected and reordered. A salience-aware description is then generated, highlighting the most salient elements in the scene.
Fig. 3: Examples of images, salience masks, and salience-aware ground-truth captions of Salience COCO, Salience Flickr, and Salience Vizwiz datasets.
Fig. 4: Statistics of word counts and sentence lengths of Salience COCO, Salience Flickr, and Salience VizWiz datasets.
Salience COCO
Salience Flickr
Salience Vizwiz
#Images
8,295
8,725
1,045
#Total words
728,043
712,516
73,689
#Caption sentences
46,012
46,190
4,815
Avg sentence
5
5
4
Vocab size
6,657
6,882
2,854
Avg words in sentence
16
15
15
TABLE I: Statistics for the three salience image captioning datasets.
Fig. 5: QAGNet produces context-dependent salience rankings across different scenes. Left: in a street-crossing scenario, the traffic light is ranked as the most salient object. Center: in a social setting, the person is prioritized over the cup and laptop. Right: in a dining scenario, the cup is ranked above the person. These examples demonstrate that the same object category (e.g., cup, person) receives different salience rankings depending on the scene context.
Fig. 6: Overview of the Salient LLaVA framework for image captioning. The model processes an input image and an instruction, extracting visual features using CLIP-ViT and ResNet-50. A QAGNet module enhances feature extraction, followed by projection layers to generate salience features. These features are fused and processed through a linear layer, combined with tokenized instruction embeddings, and fed into Vicuna to generate the output caption.
Datasets
Models
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-1
ROUGE-2
ROUGE-L
SCMI
Wu-Palmer SCMI
Salience COCO
DeCap [ 51 ]
0.35
0.18
0.06
0.03
0.07
0.26
0.38
0.09
0.23
0.35
0.44
GRIT [ 52 ]
0.27
0.13
0.05
0.03
0.03
0.19
0.35
0.09
0.23
0.28
0.32
Tag2text [ 53 ]
0.27
0.14
0.06
0.02
0.05
0.21
0.36
0.10
0.24
0.33
0.38
LLaVA [ 45 ]
0.25
0.15
0.09
0.06
0.12
0.21
0.34
0.13
0.22
0.36
0.28
LLaVA-FT [ 45 ]
0.46
0.32
0.22
0.20
0.40
0.40
0.55
0.30
0.35
0.51
0.58
SCOPE [ 54 ]
0.44
0.27
0.17
0.10
0.17
0.33
0.48
0.19
0.30
0.53
0.60
TABLE II: Comparison of model performance on our three saliency datasets using multiple evaluation metrics. We additionally include recent state-of-the-art MLLMs evaluated with explicit salience-aware two-shot prompting. The best results for each dataset and metric are highlighted in bold.
Method
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-1
ROUGE-2
ROUGE-L
SCMI
Wu-Palmer SCMI
Static Merge
0.33
0.21
0.13
0.08
0.16
0.25
0.38
0.15
0.25
0.41
0.44
CNN
0.37
0.23
0.15
0.09
0.17
0.29
0.44
0.17
0.28
0.45
0.49
Ours
0.52
0.37
0.28
0.21
0.45
0.43
0.60
0.33
0.40
0.60
0.63
TABLE III: Ablation Study on Salience Information Integration using salience COCO dataset.
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-1
ROUGE-2
ROUGE-L
SCMI
Wu-Palmer SCMI
LLaVA [ 45 ]
0.14
0.10
0.06
0.04
0.03
0.10
0.16
0.05
0.12
0.07
0.20
Ours
0.40
0.31
0.21
0.14
0.26
0.41
0.53
0.22
0.33
0.14
0.56
TABLE IV: Comparison of model performance on real-world dataset. The best results are highlighted in bold
Fig. 7: Smart glasses used in our real-world experiment with smart sensors, earphone and camera.
Fig. 8: A sequence diagram outlines the flow of how user interacts with our captioning model deployed at the backend through ESP32 S3.
Dataset
Setting
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-1
ROUGE-2
ROUGE-L
SCMI
Wu-Palmer SCMI
Salience COCO
Full dataset
0.52
0.37
0.28
0.21
0.45
0.43
0.60
0.33
0.40
0.60
0.63
Low-Light
0.52
0.36
0.27
0.15
0.35
0.40
0.58
0.30
0.38
0.53
0.53
Salience Flickr
Full dataset
0.50
0.35
0.26
0.20
0.45
0.42
0.57
0.31
0.39
0.79
0.83
Low-Light
0.50
0.34
0.23
0.16
0.40
0.39
0.55
0.24
0.36
0.75
0.78
Salience VizWiz
Full dataset
0.47
0.32
0.23
0.17
0.36
0.38
0.54
0.27
0.35
0.71
0.79
Low-Light
0.43
0.28
0.20
0.14
0.32
0.35
0.51
0.25
0.35
0.63
0.63
TABLE V: Robustness evaluation under low-light conditions. “Full dataset” denotes full test set results; “Low-Light” denotes the low-light subset. The best results for each dataset and metric are highlighted in bold.
Datasets
Models
BLEU-1
BLEU-2
BLEU-3
BLEU-4
CIDEr
METEOR
ROUGE-1
ROUGE-2
ROUGE-L
SCMI
Wu-Palmer SCMI
Salience COCO
Samba [ 58 ]
0.51
0.36
0.27
0.18
0.41
0.41
0.59
0.31
0.38
0.57
0.58
DSGNN [ 59 ]
0.51
0.35
0.25
0.19
0.42
0.41
0.59
0.31
0.38
0.57
0.59
Ours
0.52
0.37
0.28
0.21
0.45
0.43
0.60
0.33
0.40
0.60
0.63
Salience Flickr
Samba [ 58 ]
0.52
0.38
0.29
0.23
0.40
0.43
0.59
0.31
0.41
0.69
0.77
DSGNN [ 59 ]
0.52
0.38
0.29
0.22
0.41
0.42
0.59
0.31
0.41
0.70
0.76
Ours
0.50
0.35
0.26
0.20
0.45
0.42
0.57
0.31
0.39
0.79
0.83
TABLE VI: Salience information ablation study across three saliency datasets. The best results for each dataset and metric are highlighted in bold.