OpenVAM: Open-World Visual Attention Modeling with VLMs

Kiana Hooshanfar; Amirhossein Kazerouni; Alireza Hosseini; Michael Brudno; Babak Taati

cs.CVSep 25, 2026

OpenVAM: Open-World Visual Attention Modeling with VLMs

Authors: Kiana Hooshanfar, Amirhossein Kazerouni, Alireza Hosseini, Michael Brudno, Babak Taati

Organizations: University of Tehran · University of Toronto · Vector Institute · University Health Network

Abstract

Predicting human gaze is a core capability for applications ranging from web/UI design analysis to robotics and human-computer interaction. Yet, most visual attention modeling methods output only a dense saliency map, which is often insufficient for action: practitioners need to connect attention peaks to discrete elements in the scene (what) and understand the drivers of those peaks in context (why), while remaining robust to domain shift across natural images, commercial content, and UI/web layouts. We, therefore, introduce OpenVAM (Open-world Visual Attention Modeling with VLMs), a unified framework that jointly addresses universality and explainability across heterogeneous domains (natural scenes, commercial imagery, and UI/web layouts) and supervision modalities. OpenVAM adopts a decoupled-but-aligned design: a dedicated dense visual pathway provides stable, spatially precise localization, while an instruction-following vision--language semantic head generates grounded what/why explanations conditioned on the same image and data-type prompts. A three-stage training strategy preserves strong localization priors while progressively introducing language grounding and improving explanation alignment via parameter-efficient adaptation without perturbing the saliency branch. We further propose a scalable pipeline to generate multi-domain saliency-reason annotations for training and systematic evaluation. Experiments across diverse datasets show that OpenVAM improves robustness under domain shift while producing image-grounded explanations that make saliency predictions more interpretable.

Figures & tables

Figure 1: (a) OpenVAM architecture. An input image is encoded into multi-level visual features. ConvUpConv projections build a feature pyramid for the saliency decoder to predict dense saliency, while the deepest feature ( $L_{12}$ ) is adapted into an instruction-following VLM to generate grounded explanations of salient regions. (b) Model training. Stage I: the visual encoder and saliency decoder learn dense attention localization from multi-scale features ( $L_{3},L_{6},L_{9},L_{12}$ ). Stage II: the VLM is attached, and the visual encoder, adapter, and saliency decoder are trained while the tokenizer, transformer, and LM head remain frozen. Stage III: the encoder and saliency decoder are frozen, while the visual adapter and the language side are tuned to align explanations with saliency.

Figure 2: Overview of the proposed dataset.

Panel of Figure 3 — Figure 3: Qualitative comparison of saliency predictions across domains. Rows 1 and 4 show natural-scene images, Row 2 shows an e-commerce/commercial image, and Row 3 shows a UI/web layout.

Sample	OpenVAM-3B	OpenVAM-7B	Ground Truth
Sample 1	The girl’s face (center-left): Faces are inherently attention-grabbing, and her direct gaze towards the television makes it a strong focal point. The television screen content (right-center): The image of an animal on the TV display creates high contrast against its dark frame. Its central placement within that part of this visual field also draws attention. The girl’s hands and arms (center-right): Her active pose with raised hand suggests motion or interaction, drawing the eye to where she is engaged in an activity.	The woman (center-left): She is the largest subject, and her bright blue sweater creates strong color contrast against the more muted tones of the room. Her active pose and profile view direct attention toward the television. The television screen (center): The woman’s gaze and hands point directly to the screen, which is a focal point of interaction. The screen emits light and displays a distinct image, adding saliency. The cat image on TV (center): The animal picture on the screen is visually prominent and high-contrast compared to the rest.	Woman’s face (mid-left): It is a human face, which naturally draws significant attention, and her head is angled, indicating focus. Woman’s arms (mid-right): Positioned in an animated way, suggesting action and interaction with the TV. TV screen (mid-right): It’s bright and displays an image of a cat, which attracts attention due to contrast and content.
Sample 2	Pendant (bottom-center): The coin-like pendant is a distinct and detailed feature of the necklace, drawing the eye with its circular shape and shiny texture at the lowest point of the jewelry. Blazer and turtleneck (surrounding center): The combination of a brown blazer over a black turtleneck provides a strong color contrast and frames the necklace, helping it stand out. Necklace chain (upper center): The chain is visible against the dark turtleneck and leads the eye down to the pendant.	Pendant (bottom-center): The coin-like pendant is a distinct and detailed feature of the necklace, drawing the eye with its circular shape and shiny texture at the lowest point of the jewelry. Blazer and turtleneck (surrounding center): The combination of a brown blazer over a black turtleneck provides a stylish backdrop and context, with the dark turtleneck creating high contrast that highlights the necklace.	The large circular pendant (lower center): Its significant size and distinctive round shape make it a focal point. The reflective surface of the pendant further enhances its visual prominence. The black turtleneck shirt (center): The solid, dark color of the shirt provides a clean and contrasting background that effectively highlights the bright, metallic details of the necklace positioned against it.
Sample 3	“Facebook logo” (center): The large, centered Facebook logo is highly recognizable and draws immediate attention due to its size and color contrast. “Bitcoin Santa” image (top-left): The image of coins with a Santa hat is visually striking, with bright colors and a unique concept that captures attention. Article headline (center): The headline text is prominent, with bold lettering that stands out against the background. Article headline: “Zuckerberg Sells $2 Billion Worth Of Facebook Stock To Pay Taxes” (center): The headline is bold and positioned centrally, making it a focal point due to its significant financial information.	Headline text (top-left): This text is large, bold, and positioned at the top of the first content card, making it a primary focal point due to its prominence and contrast against the background image of coins. Headline text (center): Located in the center of the grid, this headline is clearly legible, uses a bold font, and is overlaid on a high-contrast image of a Facebook logo being cut, drawing strong visual attention. Zuckerberg face (center-lower): Positioned centrally in the lower half of the grid, increasing its visual prominence.	Headline text (top-left content card): This headline text stands out prominently due to its large size, clear readability, and interesting keywords. The image of shiny gold coins immediately below it is highly distinctive and detailed. Headline text (top-center content card): The headline text is clearly readable and captures attention with a specific number and well-known brand name. The associated image features a distinct company logo in a contrasting visual context. Image of Mark Zuckerberg’s face (bottom-center content card): The presence of a clear human face in the image is a strong visual cue that naturally draws attention. The headline above it reinforces its prominence with a large financial value.
Sample 4	The rider (center): The human figure’s dynamic pose and brightly colored purple and yellow shirt create strong cues for motion and color contrast, making them a primary focal point. The horse (center): As the largest subject in the frame, its dark color contrasts with the lighter background, and its extended legs strongly imply forward motion.	The rider’s upper body (center): The human figure, especially the head and shoulders of the rider, naturally draws significant attention. Its central placement on a unique animal also contributes to its prominence. The horse’s main frame (center-right foreground): As an unusual subject in this scene with strong contrast against its surroundings, it occupies a large portion of the image and is sharply focused. The distinct shape of the saddle on top further highlights it as part of a unique animal. The horse’s main body (center): This area completes their presence within an unusual context in this scene, drawing attention due to its role in supporting and interacting with the main figure. Small object near horse’s front leg (lower center): A small, light-colored object near the horse’s front hoof is visible, but its identity is unclear due to low resolution and lack of context.	Rider’s upper body (center): This area is prominently positioned and brightly illuminated, creating a strong visual contrast with the darker horse. The rider’s posture suggests an active pose, drawing the eye to the central human element. Horse’s head and neck (upper center): This distinct part of the animal is clearly visible and directly engaged with the rider, making it a focal point due to its prominent features and interaction. Horse’s main body (center): The large, dark form of the horse’s back and saddle area creates a significant visual mass in the middle of the frame, providing a stable foundation for the rider. The small white ball (bottom foreground): The ball’s light color stands out against the darker ground, and its position in the path of the horse suggests it is the focus of the central action.

Table 1: Qualitative text comparison. OpenVAM explanations versus ground-truth text for Samples 1–4, respectively.

Dataset	Size	Methods	JScore $\uparrow$	ROUGE $\uparrow$			BLEU $\uparrow$
				R-1	R-2	R-L	B-1	B-2	B-3	B-4
U-EYE [ 34 ]	PV	Gemini-2.5-Pro	0.739	0.511	0.172	0.253	0.403	0.225	0.129	0.074
	3B	Qwen2.5-VL-3B	0.537	0.391	0.103	0.198	0.300	0.151	0.080	0.045
	3B	OpenVAM-3B	0.603	0.429	0.108	0.196	0.330	0.156	0.077	0.040
	4B	Qwen3-VL-4B	0.679	0.444	0.119	0.207	0.345	0.171	0.090	0.049
	4B	OpenVAM-4B	0.675	0.399	0.118	0.220	0.317	0.163	0.088	0.049

Table 2: Evaluation comparison across VLMs on multiple datasets. JScore measures semantic quality, while ROUGE and BLEU measure lexical overlap. Best results are bolded within each model-size group. PV denotes proprietary models.

Training Stages				Saliency					VLM Judge $\uparrow$	ROUGE $\uparrow$			BLEU $\uparrow$
Dataset	S1	S2	S3	CC $\uparrow$	KLD $\downarrow$	AUC $\uparrow$	SIM $\uparrow$	NSS $\uparrow$	JScore	R-1	R-2	R-L	B-1	B-2	B-3	B-4
U-EYE [ 34 ]	joint			0.705	0.567	0.845	0.621	1.708	0.451	0.409	0.097	0.176	0.204	0.137	0.043	0.030
	✓	✗	✗	0.728	0.550	0.846	0.622	1.704	–	–	–	–	–	–	–	–
	✓	joint		0.730	0.559	0.845	0.620	1.714	0.441	0.388	0.083	0.181	0.271	0.113	0.050	0.026
	✓	✓	✗	0.735	0.543	0.847	0.630	1.721	0.434	0.252	0.034	0.133	0.148	0.050	0.019	0.009
	✓	✓	✓	0.734	0.542	0.847	0.639	1.720	0.603	0.429	0.108	0.196	0.330	0.156	0.077	0.040

Table 3: Stage-wise ablation of OpenVAM-3B. We progressively enable Stage 1 (S1), Stage 2 (S2), and Stage 3 (S3), and also compare against full joint training of S1–S3 and joint training of S2–S3 after S1.

Dataset	Method	CC $\uparrow$	KLD $\downarrow$	AUC $\uparrow$	SIM $\uparrow$	NSS $\uparrow$
U-EYE [ 34 ]	SAM [ 14 ]	0.580	1.490	0.811	0.520	1.640
(Web page)	UMSI [ 17 ]	0.562	1.580	0.805	0.510	1.690
	SAM++ [ 34 ]	0.580	1.190	0.800	0.530	1.660
	Transalnet [ 45 ]	0.696	0.616	0.839	0.598	1.601
	UMSI++ [ 34 ]	0.670	0.860	0.830	0.580	1.610
	SUM [ 24 ]	0.731	0.544	0.846	0.630	1.704

Table 4: Saliency prediction performance across various datasets. Baseline results are taken from their respective papers.

Dataset	Method	Saliency Metrics
Dataset	Method	CC $\uparrow$	KLD $\downarrow$	AUC $\uparrow$	SIM $\uparrow$	NSS $\uparrow$
Toronto [ 7 ]	SUM [ 24 ]	0.767	0.558	0.875	0.642	2.170
	OpenVAM-3B	0.784	0.508	0.882	0.659	2.245
	OpenVAM-7B	0.791	0.499	0.883	0.665	2.254
TUD DB 1 [ 44 ]	SUM [ 24 ]	0.790	0.641	–	0.654	–
	OpenVAM-3B	0.720	0.557	–	0.623	–

Table 5: Saliency performance on unseen datasets. OpenVAM generalizes strongly across diverse out-of-distribution benchmarks. “–” indicates missing fixation annotations.

Figure 4: Annotation richness. Train/test distributions of words per explanation and mentioned salient regions.

Figure 5: Dataset scale across domains. Number of samples contributed by each dataset in the corpus.

Dataset	Top-k Cat. Share (%)		Semantic Complexity / Long-tail			Split Drift
Dataset	Top-1	Top-3	Cat Ent.	Eff. #Cats	Obj. Gini	Cat JSD (bits)
CAT2000	26.06	59.99	0.837	7.44	0.080	0.0050
MIT1003	28.36	59.46	0.849	7.67	0.116	0.0028
OSIE	36.91	72.29	0.778	6.46	0.251	0.0026
SalECI	50.48	87.69	0.691	3.83	0.519	0.0085
Salicon	31.19	61.62	0.845	7.58	0.506	0.0002

Table 6: Semantic distribution complexity, long-tail structure, and split drift. We report (i) category dominance via the Top-1 and Top-3 category share (the percentage of all mentions assigned to the most frequent category, or to the three most frequent categories), (ii) semantic diversity via the category entropy (higher indicates a more even category distribution) and the effective number of categories (the entropy exponentiated, interpretable as the number of equally frequent categories that would yield the same entropy), and (iii) object-level long-tail inequality via the Gini coefficient over object mentions (higher indicates that a small number of objects accounts for a large fraction of mentions). To assess split stability, we compute the Jensen–Shannon divergence between the training and test category distributions (category JSD; lower indicates more similar splits).

Figure 6: Spatial cue usage across datasets and splits. For each dataset, we report the average number of normalized location-token mentions per explanation for each spatial cue ( center , left , right , upper , lower , foreground , background ), shown separately for train and test. The close agreement between train and test profiles indicates split-consistent spatial-language usage, while differences across datasets reflect domain-specific composition and layout.

Panel of Figure 7 — Figure 7: Concept- vs. category-level dataset similarity. (a) Fine-grained similarity computed over canonical object-concept distributions for the natural-image datasets, where eye-tracking benchmarks ( CAT2000 , MIT1003 , OSIE ) are more mutually aligned and SALICON is less similar in concept mass because it is collected via mouse. (b) Coarse similarity computed over broad category-share vectors across all six datasets, where the natural-image datasets cluster more tightly after aggregation, while SalECI and U-EYE remain distinct due to their UI/commercial composition. In both panels, each cell reports weighted Jaccard overlap (distributional agreement) and rank correlation $r$ (agreement in category/concept ordering).

Figure 8: Distribution of explanation length across datasets. Boxplots show words per description for each source (median, interquartile range, and whiskers). Length statistics are broadly comparable across datasets, with small shifts in the upper tail for some datasets.

Model	Stage	Batch	Accum.	LR	Warmup	Epochs	Early Stop	$\lambda_{\text{sal}}$	$\lambda_{\text{text}}$	$\lambda_{\text{MSE}}$	$\lambda_{\text{KLD}}$	$\lambda_{\text{CC}}$	$\lambda_{\text{SIM}}$	$\lambda_{\text{NSS}}$	LoRA
OpenVAM	Stage I	8	–	$7.4\times 10^{-5}$	–	30	4	–	–	2.99	12	2.15	1.69	3.17	–
OpenVAM-3B	Stage II	8	4	$5\times 10^{-5}$	3	50	5	–	–	0.1	1.0	1.2	0.8	0.3	–
OpenVAM-3B	Stage III	4	4	$1\times 10^{-4}$	3	25	3	0.05	3.0	0.1	0.5	1.0	0.5	0.1	$r{=}16,\alpha{=}32,p{=}0.05$
OpenVAM-4B	Stage II	8	4	$5\times 10^{-5}$	1	50	5	–	–	0.1	1.0	1.2	0.8	0.3	–
OpenVAM-4B	Stage III	4	4	$1\times 10^{-4}$	3	25	3	0.05	3.0	0.0	0.5	1.0	0.5	0.1	$r{=}16,\alpha{=}32,p{=}0.05$
OpenVAM-7B	Stage II	4	8	$5\times 10^{-5}$	3	50	5	–	–	0.1	1.0	1.3	0.8	0.3	–

Table 7: Training hyperparameters for OpenVAM variants across stages.

Dataset	Image Domain	Acquisition Type	Image Resolution	# Test Samples
Toronto [ 7 ]	Natural scene	Eye	$681\times 511$	120
TUD Database 1 [ 44 ]	Natural scene	Eye	$768\times 512$	29
TUD Database 2 [ 2 ]	Natural scene	Eye	$600\times 600$	160
FIWI [ 54 ]	Web page	Eye	$1360\times 768$	149

Table 8: Summary of unseen evaluation datasets used for generalization.

Panel of Figure 9 — Figure 9: Visualizations of OpenVAM predictions across different natural images.

Sample	OpenVAM-4B	OpenVAM-8B	Ground Truth
Sample 1	White kitchen cabinets (upper-right): The cabinets are large and occupy a significant portion of the upper-right area, drawing the eye due to their size and placement. Their white color contrasts with the darker elements in the kitchen, making them stand out. White countertop (center): The countertop is centrally located and spans a large area, making it a focal point. Its white color contrasts with the darker elements in the kitchen, such as the sink and appliances, drawing attention to it. White sink (left): The sink is a distinct element on the left side of the countertop, contrasting with the countertop’s color. Its placement on the edge of the countertop makes it noticeable. White refrigerator (right): The refrigerator is a large, white appliance that stands out against the darker background. Its size and placement on the right side of the image make it a prominent feature. White microwave (above the sink): The microwave is positioned above the sink, making it a noticeable element in the kitchen. Its white color contrasts with the darker elements in the kitchen, drawing attention to it. White oven (right): The oven is a large, white appliance that stands out against the darker background. Its size and placement on the right side of the image make it a prominent feature.	Black chair (foreground): Its dark color contrasts with the lighter floor and wall, drawing the eye; its placement in the foreground makes it a prominent element in the composition. Kitchen area (center): The kitchen area is centrally located and contains multiple objects, making it a focal point; the contrast between the light-colored cabinets and the darker appliances also draws attention. White cabinets (kitchen): The cabinets are large and take up a significant portion of the kitchen area, making them visually prominent; their light color contrasts with the darker appliances and countertop. Black appliances (kitchen): The black appliances stand out against the lighter cabinets and countertop, creating a visual contrast; their dark color makes them noticeable in the kitchen setting. Light-colored floor (kitchen): The light color of the floor provides a neutral backdrop that allows the other elements in the kitchen to stand out; its large, open space also draws the eye. Light-colored wall (kitchen): The light color of the wall provides a neutral backdrop that allows the other elements in the kitchen to stand out; its large, open space also draws the eye.	Kitchen (overall composition): The kitchen as a whole is salient due to its position as the main subject of the image and its function as a hub of activity, drawing the viewer’s attention. Breakfast bar (lower edge): The breakfast bar is salient because it is positioned along the lower edge and provides a visual boundary. Cabinets and open shelves (upper-right): These elements draw attention because they fill the upper-right area, organizing items and adding visual interest. Refrigerator (center-right): The refrigerator is salient due to its central placement and its typical role as a key appliance in a kitchen. Angled ceiling with skylight (upper-left): The angled ceiling and skylight are salient because they create an unusual shape and introduce natural light, drawing the eye.
Sample 2	Taxis (foreground): The bright yellow color of the taxis contrasts with the surrounding environment, making them stand out; their placement in the foreground ensures they are one of the first things a viewer notices. People (foreground): The presence of people adds a human element to the scene, drawing attention to the activity and life in the city; their movement and interaction with the taxis further enhance their salience. Buildings (background): The buildings provide context and depth to the scene, framing the clock tower and taxis; their size and architectural style contribute to the overall visual interest.	Large, ornate building (center): Its central placement and size immediately draw the eye; the intricate details and unique architectural style make it stand out against the simpler background. People (foreground): The presence of people in the foreground creates a sense of scale and activity, drawing the viewer’s attention to the immediate surroundings. Cars (foreground): The cars are salient because they are in the foreground and are moving, which creates a sense of motion and activity. Sky (background): The bright blue sky provides a contrasting backdrop that highlights the other elements in the scene.	Food Trucks (center): Their bright colors and unique shapes contrast with the surrounding environment, drawing the eye; their central placement in front of the arch makes them a focal point of the scene. Grand Army Plaza Arch (center): Its large size and ornate stone construction make it a prominent feature; its central placement anchors the composition and attracts visual attention. People (center): Their presence milling about the trucks indicates activity and life, drawing the eye to the area where the action is occurring; their contrast against the background helps them stand out. Bright Blue Sky (background): The expansive blue sky provides a bright and contrasting backdrop; the presence of scattered clouds adds visual interest and depth. Trees (surrounding): They frame the scene, providing a natural border that enhances the overall composition; their green color contrasts with the stone of the arch.
Sample 3	Skiers (center foreground): The skiers are centrally located and in the foreground, immediately drawing the viewer’s attention. Their presence suggests activity and human interest in the scene. Snowy slope (visible): The snow-covered slope is a large, bright area that contrasts with the darker elements in the background, making it visually prominent. The texture and expanse of the snow also contribute to its salience. Ski lift (background): The ski lift is a large, man-made structure that stands out against the natural landscape. Its height and position in the background create a sense of depth and scale. Buildings (background): The buildings are visible in the background, providing context and indicating a developed area. Their presence adds to the overall scene and helps to define the location. Sky (visible): The sky is a large, bright area that contrasts with the darker elements in the foreground and background. Its color and texture contribute to the overall visual appeal of the image.	Skiers (center): The skiers are centrally located and are the main subjects of the image, drawing the viewer’s attention to their activity and attire. Their presence in the foreground also makes them more prominent. Snow-covered slope (visible): The snow-covered slope provides a bright, contrasting background that highlights the skiers and their equipment. The texture and expanse of the snow also contribute to the overall visual interest. Buildings (background): The buildings in the background add depth to the image and provide context for the location. Their size and placement in the distance make them noticeable but not overpowering. Skis and poles (visible): The skis and poles are essential elements of the skiers’ gear and are visually distinct due to their shape and arrangement. They contribute to the overall theme of skiing and are important for understanding the activity. Blue jacket (visible): The blue jacket stands out due to its color contrast against the white snow and the other skiers’ clothing. Its placement on the right side of the image also draws the eye. White jacket (visible): The white jacket is salient due to its bright color, which contrasts with the surrounding environment. Its placement on the left side of the image also helps to balance the composition. Black jacket (visible): The black jacket is salient due to its dark color, which contrasts with the white snow and the other skiers’ clothing. Its placement on the left side of the image also helps to balance the composition.	Skiers (center and right): Skiers stand out because they are centrally located in the image and contribute to the action. Their presence indicates the setting and activity of the scene. Ski lift (upper-center background): The ski lift draws attention because it is relatively large and unique to a ski environment. Its placement in the background helps establish the setting. Snowy slope (foreground and midground): The snowy slope is salient due to its extensive coverage and contrast with the trees and building. Its whiteness makes it a dominant feature of the landscape. Trees (scattered in background): Trees contribute to the depth and natural context of the image. Their darker color contrasts with the snow. Small, light-colored building (upper-left background): The building draws attention due to its distinct shape and placement in the background. It serves as a focal point and helps orient the viewer. Colorful jackets and ski pants (worn by most skiers): The bright colors create contrast against the white snow. The variety of colors on multiple subjects draws the eye to the human activity. Skier in a yellow jacket (center-right foreground): The yellow jacket is highly salient due to its brightness. The skier’s forward-facing position directs the viewer’s gaze and creates a focal point.

Table 9: Qualitative text comparison of Figure 9 . Generated explanations from OpenVAM models versus ground-truth text. Samples 1–3 correspond to rows 1–3, respectively.

Sample	OpenVAM-4B	OpenVAM-8B	Ground Truth
Sample 4	Horse (center): The horse is centrally located, immediately drawing the viewer’s attention. Its size and the fact that it is the main subject of the image further enhance its salience. Rider (on horse): The rider is salient because they are the only human figure in the image, creating a focal point. The contrast of the rider’s clothing against the horse’s coat also makes them stand out.	Horse (center): The horse is centrally located and takes up a significant portion of the frame, immediately drawing the eye. Its size and the contrast of its brown coat against the green grass and the background also contribute to its salience. Person (on horse): The person is salient because they are the only human figure in the image, and their presence on the horse creates a focal point. The contrast of their clothing against the horse’s coat also makes them stand out. Horse’s head (center): The horse’s head is salient due to its proximity to the viewer and its placement in the lower-left corner, which is a common starting point for visual scanning. The head is also a natural focal point for observing animals. Horse’s body (center): The horse’s body is salient because it is the main subject of the image and occupies a large portion of the frame. The contrast of its brown coat against the green grass and the background also makes it stand out. Horse’s tail (right): The horse’s tail is salient because it is a distinct feature of the horse and is positioned on the right side of the frame, balancing the composition. The tail’s movement and the contrast of its color against the background also draw attention.	A rider wearing a red backpack (center-left midground): The bright red backpack stands out against the earthy tones of the surrounding landscape and the horse, drawing immediate attention. The central-left midground placement further ensures it captures the viewer’s focus. Another rider (center-right midground): Being in the midground places them within the viewer’s initial field of attention, contributing to their salience. Brown horse (under the rider): Due to their relatively large size and central placement, the brown horse is a prominent subject in the image. Dusty trail (winding upward): The trail creates a visible line and a pathway for the eye to follow, making it a salient element. Rocky landscape (surrounding the trail): The textured and varied terrain contrasts with the smoother sky and path, making it a visually distinct and salient area. Distant hazy mountains (backdrop): Their significant size and placement at the top of the image, though softened by haze, still make them a noticeable backdrop.
Sample 5	Red and white sign (left edge): The sign’s bright red and white colors contrast sharply with the surrounding green grass and blue sky, making it immediately noticeable. Its placement on the left edge also draws the eye, as it is one of the first elements encountered in the image. Light-colored building (center): The building’s size and central placement make it a dominant feature in the image. Its light color contrasts with the darker trees and green grass, further enhancing its salience. Green grass (foreground): The grass covers a large portion of the foreground, providing a base for the other elements. Its uniform color and texture create a visual foundation that draws the eye. Trees (background): The trees create a backdrop for the other elements in the image. Their height and density add depth to the scene, making them a significant part of the overall composition. Blue sky (background): The blue sky provides a bright and open backdrop for the other elements. Its color contrasts with the green grass and trees, making it a noticeable part of the image.	Red Stop sign (left edge): The bright red color contrasts sharply with the green grass and blue sky, immediately drawing the eye. Its placement on the edge of the frame also makes it a focal point. Grassy field (foreground): The large expanse of green grass occupies a significant portion of the image, providing a base and context for the other elements. Its uniform color and texture create a sense of depth and space. Trees (background): The trees create a backdrop that adds depth and context to the scene. Their varying heights and shapes add visual interest and complexity to the background. Sky (overcast): The overcast sky, with its soft, diffused light, sets the overall mood and tone of the image. The presence of clouds adds texture and visual interest to the upper portion of the frame. Power lines (across the sky): The power lines create a network of lines that cut across the sky, adding a sense of structure and complexity to the scene. They also provide a sense of scale and perspective.	Stop Sign (center-left in the foreground): Its bright red color is a high contrast to the surrounding greenery. As a standard symbol, the octagonal shape immediately grabs the viewer’s attention. Menacing Clouds (backdrop): The large size and dark color of the clouds create a sense of drama and foreboding. Grassy Field (behind the stop sign): The bright color creates a visual buffer between the stop sign and the trees. Tree Line (midground): The texture and color of the tree line is of high contrast to the field in front of it. Roofs of Houses (behind the tree line): They interrupt the regular pattern of the tree line. Light Sky (visible through the clouds): The contrast between dark and light creates visual interest and draws the eye to the sky. Power Lines (upper portion of the frame): They are visually salient due to their linear shape cutting across the sky, and their high contrast against the lighter parts of the sky.
Sample 6	Colorful kites (upper-right): The kites are salient due to their bright colors, which contrast with the blue sky. Their placement in the upper-right also draws the eye, as it is a common area for visual focus. Crowd of people (center): The crowd is salient because it is centrally located, drawing the eye to the middle of the image. The density of people also creates a focal point. Sky (background): The sky is salient because it is a large, uniform area that fills the upper portion of the image. Its blue color provides a backdrop for the kites and other elements.	Kite (center): The kite is centrally located and is the largest object in the image, drawing the eye immediately. Its bright colors and dynamic shape contrast with the sky, making it stand out. Crowd (foreground): The crowd is salient because it occupies a large portion of the foreground, creating a sense of depth and scale. The density of people and the variety of colors from their clothing also contribute to its visual prominence. Tents (background): The tents are salient because they define the event space and provide context. Their size and arrangement suggest a organized event, drawing attention to the purpose of the gathering. Sky (background): The sky is salient because it provides a backdrop that contrasts with the other elements. The blue color and scattered clouds create a sense of openness and depth, enhancing the overall composition.	Kites (top): The kites draw attention due to their bright colors and their large size in relation to other objects in the image. Also, they are unique shapes and high in the visual field. Crowd (below): The crowd draws attention due to its sheer size, filling a significant portion of the lower half of the image. Its density creates a visual mass that stands out. White tents (center): The tents draw attention due to their light color that contrasts with the darker ground and the surrounding people. Also, they are centrally placed in the visual field. Person with a kite (foreground): The person is salient due to their foreground placement, which brings them closer to the viewer. The presence of the kite and its vibrant colors also makes the person a focal point. Clear, bright blue sky (top): The sky is salient due to its large size and uniform color, providing a backdrop for the other objects in the image. Its brightness also draws the eye upward.

Table 10: Qualitative Text Comparison Figure 9 . Generated explanations from OpenVAM models versus ground-truth text. Samples 4–6 correspond to rows 4–6, respectively.

Panel of Figure 10 — Figure 10: Visualizations of OpenVAM predictions across different e-commerce and UI images.

Sample	OpenVAM-4B	OpenVAM-8B	Ground Truth
Sample 7	Price text (lower-right): This large, bold red number is highly prominent due to its size, strong color contrast against the white background, and its position as the primary price information. Promotional text (top-center): This large, bold red text is centrally placed at the top of the image, immediately drawing attention to the main product description and key features. Main product (center): The large, clear image of the product bottle is centrally located, making it the primary visual focus for the item being sold. Promotional text (top-left): This large, bold red text is positioned prominently in the upper-left corner, highlighting a significant discount and drawing attention to the offer. Promotional text (top-right): This large, bold red text is placed in the upper-right corner, emphasizing a special event and drawing attention to the promotional period. Promotional text (top-center): This large, bold red text is centrally located at the top of the image, immediately drawing attention to the main product description and key features.	Price text (lower-right): This large, bold price is highly salient due to its prominent size, strong contrast against the red background, and its placement in a dedicated promotional area. Promotional text (right side): This large, bold text is highly salient due to its significant size, strong contrast against the red background, and its clear message of a substantial discount. Main product (center): The large, centrally placed bottle of the product is a primary focal point, drawing attention with its clear presentation and prominent placement. Promotional text (top-left): This text is salient due to its large, bold font and its position at the top of the image, immediately conveying a key product benefit. Promotional text (top-right): This text is salient due to its large, bold font and its position at the top of the image, immediately conveying a key product benefit. Promotional text (right side): This text is salient due to its large, bold font and its clear message of a significant discount, drawing attention to the value proposition.	Promotional text (right): This large, multi-layered badge prominently features bold, clear discount information like (2nd item half price) and (2nd item only 39.5) using strong contrasting colors of red and blue. Product bottle (center-left): The main product bottle is centrally placed and of significant size, featuring readable brand text and with a vibrant, eye-catching gradient design. Promotional text (top): The banner at the top features prominent, clear text (Ali Health International Self-operated Tmall Double 12), with being a key promotional event. Small gold sphere (lower-left): This small, shiny, metallic sphere stands out due to its high contrast against the red background and its unique, reflective appearance.
Sample 8	Main product (center): The white phone stand with a phone placed on it is centrally positioned and brightly illuminated, making it the primary visual focus. Its clean design and large size clearly showcase the product. Feature headline text (top-center): Large bold text at the top highlights the product’s key functionality. Its size and strong contrast against the background make it one of the most noticeable elements. Description text (below headline): A smaller descriptive text block sits directly beneath the main headline and explains the product’s benefit. Its placement near the headline draws attention while providing additional context. Product lighting element (center-bottom): The illuminated ring and base of the stand emit a warm glow that visually emphasizes the lighting feature. The brightness contrasts with the background, making this area visually striking.	Promotional text (top-center): This text is large, bold, and centrally placed, immediately drawing attention to the product’s key feature. Its high contrast against the background further enhances its visibility. Main product (center): The white phone stand is the primary subject, centrally positioned and sharply in focus. Its clean design and the clear display of a phone on it make it the focal point of the image. Promotional text (top-center): Positioned directly below the main feature, this text provides additional context and reinforces the product’s functionality. Its clear readability and central placement contribute to its salience.	The main product (phone on wireless charging stand) (center): This is the largest and most centrally placed object, illuminated by its own integrated light ring and base, making it the primary focus. The phone screen displays a dynamic, colorful pattern, further enhancing its visual prominence. Promotional text (Wireless Charging Small Desk Lamp) (top-left): This text is large, bold, and positioned prominently at the top-left, clearly stating the product’s name with high contrast against the background. Descriptive text (Convenient wireless charging new experience — Place and charge, no wires) (top-left): Located directly beneath the main product title, this text is clearly readable and highlights key product benefits, drawing attention to its practical features. The illuminated light ring and base of the stand (center-bottom): These parts of the product draw attention due to their bright, warm glow, which emphasizes the lighting functionality and enhances the overall visual appeal of the stand.
Sample 9	Models wearing the product (center): Two human models wearing the featured clothing are centrally positioned and brightly lit, making them the primary visual focus. Human faces and full-body poses naturally draw attention and showcase the product in use. Promotional discount banner (lower-left): This banner uses bold typography and bright yellow coloring to highlight a major discount offer. The strong color contrast and large text make it highly noticeable. Additional promotion/offer text block (lower-left): Supporting promotional text sits above or near the discount area and emphasizes event-related offers. Its bold styling and placement near the discount messaging attract attention.	Model’s face (left-center): The model’s face is clearly visible and directly engages the viewer, naturally drawing attention as a human element. Promotional text (lower-left): This large, bold text is centrally placed in the lower-left, highlighting a significant discount and a free gift, making it highly noticeable. Model wearing the product (right-center): The model’s full body, especially the upper torso, is prominently displayed, showcasing the main product in a natural pose. Promotional text (lower-left): This text, positioned directly above the discount, clearly states the original price, creating a strong contrast with the discounted price and drawing attention to the value. Promotional text (lower-left): This text, positioned above the discount, clearly states the original price, creating a strong contrast with the discounted price and drawing attention to the value.	Model on the right (center-right): The model wears the brightly colored yellow hoodie, which significantly stands out against the background, and their full figure is centrally placed, making them a main subject. Model on the left (center-left): This model also wears the main product, a striking yellow hoodie, and her visible face and pose contribute to the product’s presentation. Promotional text (lower-left): This large, bold text highlights a significant discount ( or 50% off), making it a highly attractive promotional element. Price text and (lower-left): These numbers are clearly displayed and vary in color, indicating different price points or savings, with highlighted in green. Callout badge (lower-right): This text provides important context for the prominent price, drawing attention due to its direct relevance and close proximity to the main offer.

Table 11: Qualitative Text Comparison of Figure 10 . Generated explanations from OpenVAM models versus ground-truth text. Samples 7–9 correspond to rows 1–3, respectively.

Dataset	Vision Encoder	Saliency
Dataset	Vision Encoder	CC $\uparrow$	KLD $\downarrow$	AUC $\uparrow$	SIM $\uparrow$	NSS $\uparrow$
U-EYE	DINOv3-S/16	0.716	0.570	0.842	0.618	1.673
	DINOv3-S+/16	0.715	0.569	0.843	0.617	1.671
	DINOv3-B/16	0.728	0.550	0.846	0.622	1.704
	DINOv3-L/16	0.719	0.561	0.843	0.617	1.699
SalECI	DINOv3-S/16	0.768	0.495	0.894	0.656	1.972

Table 12: Vision-backbone ablation across DINOv3 encoders during Stage I training. We compare four visual encoders (DINOv3-S/16, DINOv3-S+/16, DINOv3-B/16, DINOv3-L/16) while keeping all other components and training settings identical. Overall, DINOv3-B/16 provides the strongest performance and is therefore used as the default vision backbone in our experiments.

Dataset	Backbone	CC $\uparrow$	KLD $\downarrow$	AUC $\uparrow$	SIM $\uparrow$	NSS $\uparrow$
U-EYE [ 34 ]	Qwen ViT	0.656	0.673	0.826	0.577	1.509
U-EYE [ 34 ]	DINOv3-B/16 (ours)	0.728	0.550	0.846	0.622	1.704
SalECI [ 32 ]	Qwen ViT	0.690	0.671	0.869	0.577	1.747
SalECI [ 32 ]	DINOv3-B/16 (ours)	0.788	0.466	0.898	0.675	2.031
OSIE [ 62 ]	Qwen ViT	0.718	0.596	0.891	0.601	2.494
OSIE [ 62 ]	DINOv3-B/16 (ours)	0.912	0.229	0.935	0.777	3.903

Table 13: Backbone ablation under Stage-I saliency training. We replace the visual backbone with the Qwen Vision Encoder or DINOv3-B/16 while keeping the dense saliency training protocol fixed.

Model	CC $\uparrow$	KLD $\downarrow$	SIM $\uparrow$	JScore $\uparrow$
OpenVAM-4B	0.850	0.351	0.718	0.713
OpenVAM w/ Qwen3.5-4B	0.852	0.348	0.722	0.734

Table 14: Effect of a stronger VLM backbone. We compare OpenVAM-4B with a variant using Qwen3.5-4B as the VLM semantic head, averaged over the six benchmarks. The stronger VLM backbone improves both saliency metrics and rationale quality, with the largest gain on JScore.

Comparison	Spearman $\rho$	Kendall $\tau$	$\|\Delta\|$
Prompt sensitivity (different prompts, same run)
Default vs. strict grounding	0.84	0.65	–
Default vs. semantic-equivalence	0.88	0.69	–
Strict grounding vs. semantic-equivalence	0.81	0.61	–
Reasoning consistency (same prompt, repeated runs)
Run 1 vs. Run 2	0.94	0.80	0.018

Table 15: Stability of JScore. Top: agreement between judge-prompt variants on the same prediction–reference pairs (prompt sensitivity). Bottom: agreement between repeated runs of the default prompt on the same pairs (reasoning consistency / test–retest reliability).

|\Delta|

is the mean absolute score difference per item; ICC is computed across all three runs jointly.

Model	Input	Prec.	Active Params	Latency (ms) $\downarrow$	Peak Mem. (MB) $\downarrow$
SUM	$256^{2}$	FP32	57.50 M	$14.64\pm 0.27$	307
TranSalNet-Res	$384\times 288$	FP32	72.51 M	$9.32\pm 0.03$	519
TempSAL	$256^{2}$	FP32	242.52 M	$111.09\pm 27.05$	1,049
DeepGaze IIE	$256^{2}$	FP32	104.05 M	$109.56\pm 2.92$	548
OpenVAM-S1 (saliency-only)	$256^{2}$	FP32	102.78 M	$11.32\pm 0.09$	787
OpenVAM-3B (Full/no decode)	$256^{2}$	BF16	3,257.33 M	$77.89\pm 7.95$	6,585

Table 16: Inference efficiency on a single RTX 3090 with batch size 1, averaged over 500 images after 20 warm-up iterations. OpenVAM-S1 denotes the Stage-I DINOv3+dense-decoder saliency pathway without the VLM. “Full/no decode” activates the complete OpenVAM semantic pathway but excludes autoregressive rationale decoding.

Model	Max New Tokens	Mean Generated	Latency (ms) $\downarrow$	Peak Mem. (MB) $\downarrow$
OpenVAM-3B	256	141.5	$4106.45\pm 1463.91$	6,585
OpenVAM-7B	512	115.0	$3388.75\pm 3344.44$	15,240

Table 17: End-to-end rationale-generation cost on a single RTX 3090. Latency includes prompt processing and complete autoregressive decoding and therefore depends strongly on the generated sequence length.

Metric	Ref. P1	Ref. P2	OpenVAM P1	OpenVAM P2
Rationale items	398	398	460	460
Object presence	69.9%	100%	60.6%	98.0%
Location accuracy	75.7%	90.8%	72.9%	86.0%
SAM success	69.9%	82.6%	60.6%	73.2%
SMC	0.398	0.445	0.363	0.398
mIoU@20	0.219	0.242	0.186	0.200

Table 18: Spatial-grounding audit on 100 stratified images. P1 is the fully automatic SAM-only evaluation; P2 applies object-name fallbacks and visual verification to distinguish segmentation failures from grounding errors.

	SMC $\uparrow$		mIoU@20 $\uparrow$		Peak Hit $\uparrow$
Dataset	Human	Pred.	Human	Pred.	Human	Pred.
CAT2000	0.301	0.276	0.202	0.184	58.8%	23.5%
MIT1003	0.437	0.385	0.207	0.204	52.9%	52.9%
SALICON	0.636	0.626	0.239	0.244	87.5%	75.0%
OSIE	0.504	0.480	0.276	0.257	76.5%	70.6%
SalECI	0.370	0.372	0.180	0.180	35.3%	23.5%

Table 19: Spatial consistency of OpenVAM-generated rationales with human saliency and OpenVAM’s predicted saliency. The same rationale-derived SAM masks and 100 images are used for both targets.

Model / Setting	JScore $\uparrow$	R-1 $\uparrow$	R-2 $\uparrow$	R-L $\uparrow$	B-1 $\uparrow$	B-2 $\uparrow$	B-3 $\uparrow$	B-4 $\uparrow$
Qwen2.5-VL-3B + Image + Prompt	0.605	0.398	0.109	0.199	0.273	0.139	0.062	0.031
Qwen2.5-VL-3B + Image + Saliency + CoT	0.635	0.403	0.114	0.211	0.283	0.146	0.076	0.040
OpenVAM-3B	0.659	0.449	0.118	0.215	0.349	0.170	0.085	0.045
Qwen2.5-VL-7B + Image + Prompt	0.645	0.421	0.124	0.212	0.282	0.146	0.079	0.038
Qwen2.5-VL-7B + Image + Saliency + CoT	0.660	0.435	0.133	0.225	0.291	0.155	0.083	0.045
OpenVAM-7B	0.685	0.475	0.135	0.234	0.369	0.190	0.099	0.053

Table 20: Prompt-only VLM baselines averaged over the six evaluation datasets. The Image + Prompt baseline receives only the image and task instruction. The Image + Saliency + CoT baseline is a stronger prompted baseline because it is additionally given the predicted saliency map from Stage I. OpenVAM outperforms the prompted baselines, showing the benefit of staged saliency-language alignment.

Metrics	Qwen2.5-VL-3B	OpenVAM-3B	Qwen3-VL-4B	OpenVAM-4B	Qwen2.5-VL-7B	OpenVAM-7B	Qwen3-VL-8B	OpenVAM-8B
Object F1 $\uparrow$	0.274	0.461	0.347	0.490	0.336	0.511	0.523	0.546
Location Acc. $\uparrow$	0.367	0.567	0.424	0.606	0.567	0.612	0.597	0.653
Count MAE $\downarrow$	2.546	2.410	2.484	2.291	1.990	1.912	2.088	2.056

Table 21: Fine-grained evaluation of generated saliency rationales. Object F1 measures whether the correct salient objects are identified, Location Accuracy measures whether matched objects are assigned to the correct spatial region, and Count MAE measures whether the model predicts the correct number of salient regions.

Dataset	Size	Methods	JScore $\uparrow$	ROUGE $\uparrow$			BLEU $\uparrow$
				R-1	R-2	R-L	B-1	B-2	B-3	B-4
U-EYE [ 34 ]	Proprietary	Gemini-2.5-Pro	0.739	0.511	0.172	0.253	0.403	0.225	0.129	0.074
	Large	InternVL3.5-37B	0.656	0.498	0.166	0.243	0.386	0.216	0.126	0.073
		Qwen3-VL-32B	0.737	0.475	0.154	0.222	0.360	0.198	0.115	0.067
		Qwen2.5-VL-32B	0.689	0.476	0.143	0.222	0.376	0.199	0.114	0.066
	3B	Qwen2.5-VL-3B	0.537	0.391	0.103	0.198	0.300	0.151	0.080	0.045

Table 22: Evaluation comparison across VLMs on multiple datasets. JScore measures semantic quality, while ROUGE and BLEU measure lexical overlap. Best results are bolded within each model-size group.

Visual Attention

Visual attention studies how people and machines select relevant image regions over time for perception, action, and efficient computation.

See all 60 papers

Oct 4, 2026

Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning

We introduce Self-Saliency, a method for training Vision-Language Models (VLMs) to increase the alignment between their visual attention and the image regions mentioned in their reasoning. Self-Saliency uses a grounding model to localize the objects mentioned in each reasoning step and treats the resulting areas as supervision for the model's visual attention. Previous work on steering visual attention determines target image regions based solely on the image and question. In contrast, we show that conditioning the target regions on the model's generated reasoning improves downstream performance. For proper evaluation, we build a unified, broad suite of 25 visual reasoning benchmarks, where we reproduce the results of previous methods. We find that Self-Saliency significantly outperforms both prior attention-steering methods and baselines that ground image-level text, achieving both a better average rank and a better mean score. Post-training analysis shows that the model primarily adapts its reasoning text to existing attention patterns, producing shorter steps that refer to larger regions. Nevertheless, when controlling for generated text, attention to grounded regions increases significantly across the relevant layer. Finally, we identify a consistent geometric bias in VLM visual attention toward the image border. However, our ablations show that Self-Saliency's gains cannot be explained by simply aligning attention with the center of the image, highlighting the importance of aligning visual attention with the regions mentioned in the model's reasoning.

Uri Berger, Gal Chechik, Gal Dalal

NVIDIA Research · The Hebrew University of Jerusalem · University of Melbourne +1

Vision-Language Grounding VLM Reasoning Attention Intervention in VLMs Reasoning for VLMs

Ask PDF

Jul 17, 2026

Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs

Understanding how vision-language models (VLMs) interpret data visualizations remains an open problem, and is increasingly important as these models are used for analytical tasks where reliable reasoning is essential. We introduce a lightweight, diagnostic saliency map method tailored for text generation over images using transformer models, the current state-of-the-art models in visualization interpretation. Our approach aggregates the language model's attention over the visual tokens across all heads and layers, then maps this attention back onto the vision encoder's patch grid to localise it over the image, producing a direct correspondence between each generated answer token and the image regions it attended to. This yields fast, gradient-free saliency maps that expose how VLMs allocate focus across visual elements during answer generation, enabling inspection of whether model attention aligns with semantically relevant components. We evaluate our approach using a deletion metric which validates the causal faithfulness of our saliency maps to the model's behavior.

Maeve Hutchinson, Abderrahmane Wassim Mehdaoui, Pranava Madhyastha

City St George’s · City St George’s, University of London · The Alan Turing Institute

Data Visualization Visual Attention VLM Interpretability Visual Saliency Chart Understanding

Ask PDF

Aug 31, 2026

Emergent Goal-Directed Attention in Large Vision-Language Models

Human observers prioritize visual information according to task goals. Most computational models of naturalistic viewing are gaze-trained for free viewing, leaving open whether goal-directed attention can emerge in systems without gaze supervision. We tested two off-the-shelf vision-language models (VLMs), Qwen3-VL-32B-Thinking and Gemma-4-26B-A4B-it, on 4,887 naturalistic scenes under visual-search and free-viewing instructions. Model predictions were compared with human fixations on the same images under corresponding tasks. Both models aligned more closely with human fixations under matching goals than under mismatched goals. This crossover persisted in target-absent scenes, where alignment could not be explained by simple visual grounding, and appeared in decoder-layer readouts. Furthermore, model-thinking traces were grounded in target semantics during search and in visual prominence during free viewing. These findings show that general-purpose VLMs can generate human-aligned, goal-directed spatial priorities without gaze-specific training, informing theories of goal-directed attention and offering scalable tools for predicting where people look across tasks.

Han Zhang

University of Michigan Transportation Research Institute, Ann Arbor, MI, USA

Visual Attention Large Vision-Language Models Visual Search VLM Evaluation

Ask PDF

Abstract

Visual Attention

Explore similar work

Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoning

Attention-Guided Saliency Maps for Interpreting Visualization Literacy in VLMs

Emergent Goal-Directed Attention in Large Vision-Language Models