Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

Jae Young Choi; Seon Gyeom Kim; Hyungjun Yoon; Taeckyung Lee; Donggun Lee; Jaeryung Chung; Jihyung Kil; Ryan Rossi

cs.HCFeb 27, 2026

Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

Authors: Jae Young Choi, Seon Gyeom Kim, Hyungjun Yoon, Taeckyung Lee, Donggun Lee, Jaeryung Chung, Jihyung Kil, Ryan Rossi, +2 more

Organizations: KAIST · Adobe Research

Abstract

Large Language Models (LLMs) have emerged as foundation models for IoT applications such as human activity recognition (HAR). However, directly applying high-frequency and multi-dimensional sensor data, such as eye-tracking data, leads to information loss and high token costs. To mitigate this, we investigate a visual prompting strategy that transforms sensor signals into data visualization images as an input to multimodal LLMs (MLLMs) using eye-tracking data. We conducted a systematic evaluation of MLLM-based HAR across three public eye-tracking datasets using three visualization types of timeline, heatmap, and scanpath, under varying temporal window sizes. Our findings suggest that visual prompting provides a token-efficient and scalable representation for eye-tracking data, highlighting its potential to enable MLLMs to effectively reason over high-frequency sensor signals in IoT contexts.

Figures & tables

Figure 1: Visualizations used in this study. (A) Raw Timeline (B) Feature-based Timeline (C) Absolute Duration Heatmap (D) Fixation Count Heatmap (E) Raw Scanpath (F) Feature-based Scanpath.

	GazeBase	SedentaryActivity	DesktopActivity
Sampling Rate	1000 Hz	30 Hz	30 Hz
Sensor Type	Screen-based	Screen-based	Wearable
Device	EyeLink 1000	Tobii Pro X2-30	Pupil Core
Activity Classes	6	8	6
Participants	14 (Round 9)	24	8
Screen Spec	1680 $\times$ 1050 px	24-inch monitor	34-inch monitor

Table 1: Overview of the eye-tracking datasets used in this study.

		GazeBase		SedentaryActivity		DesktopActivity
		0-shot	1-shot	0-shot	1-shot	0-shot	1-shot
Accuracy
Timeline	Raw	0.500	0.811	0.096	0.283	0.200	0.257
	Feat	0.450	0.772	0.067	0.246	0.200	0.261
Heatmap	Raw	0.578	0.811	0.138	0.317	0.183	0.311
	Feat	0.644	0.739	0.129	0.304	0.217	0.278

Table 2: HAR accuracy and token consumption across experimental conditions (10s window). Bold indicates best performance per dataset, and red denotes performance below the textual baseline. Multipliers (

\times\uparrow

) show the relative increase in token usage of textual prompts compared to visual prompts.

Figure 2: One-shot setting accuracy comparison between visualization techniques across three datasets.

Figure 3: Comparison of HAR accuracy and token consumption across varying temporal window sizes (10-second to 100-second) for the (A) SedentaryActivity and (B) DesktopActivity datasets. The line plots illustrate the accuracy trends for each visual prompting strategy (Timeline, Heatmap, Scanpath, and the textual baseline). The grouped bar charts represent the corresponding input tokens for each condition.

Figure 4: Confusion matrices for (A) Heatmap and (B) Scanpath visual prompting, alongside representative examples for PLAY (C, D) and READ (E, F) from the SedentaryActivity 100-second window experiment. Activity names in the matrices are abbreviated to the first three characters. Titles, axis titles and ticks for the examples (C–F) are omitted to enhance visual clarity.

Explore similar work

RAG-HAR+: Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deployment
Jul 29, 2026Hansi Karunarathna, Nirhoshan Sivaroopan, Chamara Madarasingha +2Human Activity Recognition Cost-Aware Inference
University of Sydney, Australia · Curtin University, Australia · Colorado State University, USA
STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition
Jul 3, 2026Nirhoshan Sivaroopan, Albert Zomaya, Kanchana ThilakarathnaOn-Device Language Model Inference Human Activity Recognition
The University of Sydney · Australia
Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Apr 17, 2026Rohit Sinha, Aditya Kanade, Sai Srinivas Kancheti +2Spatial Reasoning Benchmarks Visual Spatial Reasoning
CSE Dept., IIT Hyderabad Hyderabad, India · Microsoft Research Bengaluru, India

Jul 29, 2026J/K move · Enter open · S save

RAG-HAR+: Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deployment

Hansi Karunarathna, Nirhoshan Sivaroopan, Chamara Madarasingha, Anura Jayasumana+1

University of Sydney, Australia · Curtin University, Australia · Colorado State University, USA

Human Activity Recognition Cost-Aware Inference Edge ML Retrieval-Augmented ICL

Open paper AskPDF arXiv

Human Activity Recognition (HAR) from wearable sensors supports applications in healthcare, rehabilitation, fitness tracking, and smart environments. Yet, existing deep learning approaches require dataset-specific training, large labeled corpora, and repeated adaptation to new sensor settings or activity taxonomies. Retrieval-Augmented Generation for Human Activity Recognition (RAG-HAR) addresses this by framing HAR as a training-free, retrieval-augmented task, in which statistical descriptions of sensor windows are used to retrieve similar labeled examples that guide LLM-based classification. We introduce RAG-HAR+, a retrieval-first and cost-optimized extension that strengthens retrieval while reducing dependence on LLM-based inference. RAG-HAR+ uses an offline Retrieval Designer Agent to design dataset-specific feature groups from a diverse pool of motion descriptors, enabling sensor windows to be compared using features better aligned with dataset-specific activity patterns. During inference, RAG-HAR+ uses majority voting over retrieved neighbors for samples with strong retrieval evidence and defers only uncertain cases to an LLM-based Ambiguity Resolver Agent. Across six HAR benchmarks, RAG-HAR+ maintains competitive or improved performance while reducing LLM usage, token consumption, and inference time. We further extend the RAG-HAR mobile prototype to demonstrate the practical feasibility of retrieval-first, LLM-assisted HAR in mobile sensing scenarios.

Abstract

Explore similar work

RAG-HAR+: Towards Cost-Efficient LLM-Based Human Activity Recognition for Edge Deployment

STELLA: Efficient Sensor-to-LLM Translation for On-Device Human Activity Recognition

Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs