Organizations: Technical University of Munich, Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany · Munich Center for Machine Learning (MCML), Germany
Gaze event detection is fundamental to vision science, human-computer interaction, and applied analytics. However, current workflows often require specialized programming knowledge and careful handling of heterogeneous raw data formats. Classical detectors such as I-VT and I-DT are effective but highly sensitive to preprocessing and parameterization, limiting their usability outside specialized laboratories. This work introduces a code-free, large language model (LLM)-driven pipeline that converts natural language instructions into an end-to-end analysis. The system (1) inspects raw eye-tracking files to infer structure and metadata, (2) generates executable routines for data cleaning and detector implementation from concise user prompts, (3) applies the generated detector to label fixations and saccades, and (4) returns results and explanatory reports, and allows users to iteratively optimize their code by editing the prompt. Evaluated on public benchmarks, the approach achieves accuracy comparable to traditional methods while substantially reducing technical overhead. The framework lowers barriers to entry for eye-tracking research, providing a flexible and accessible alternative to code-intensive workflows.
Figures & tables
Figure 1. This figure compares our method with the traditional manual approach. In particular, the LLM-assisted workflow on the right illustrates our pipeline: (1) starting from raw data, the LLM automatically identifies relevant fields, assesses data quality characteristics, and generates an easy-to-understand summary; (2) users specify their requirements in natural language, and the LLM produces executable data-cleaning code together with parameterized implementations of selected event detectors (e.g., I-VT, I-DT); (3) the cleaned data is then passed to a detector set composed of multiple event-labeling algorithms; and (4) the results and explanatory reports are returned, allowing users to iteratively refine the code and parameters by editing prompts and rerunning the detection process.
I-VT
I-DT
precision
recall
F1
precision
recall
F1
GazeCom
fixation
0.9730
0.9782
0.9756
0.9765
0.9240
0.9495
GazeCom
saccade
0.8473
0.8170
0.8319
0.6212
0.8486
0.7173
GazeBase_v2
fixation
0.9922
0.6216
0.7644
0.9513
0.9834
0.9671
GazeBase_v2
saccade
0.2195
0.9559
0.3571
0.7865
0.5477
0.6457
hollywood2_em
fixation
1.0000
0.0025
0.0050
0.9831
0.9565
0.9696
Table 1. Detection performance of the I-VT and I-DT pipelines generated by our framework across four datasets.
Table 3
I-VT
I-DT
precision
recall
F1
precision
recall
F1
hollywood2_em
First time
fixation
1.0000
0.0025
0.0050
0.9831
0.9565
0.9696
hollywood2_em
First time
saccade
0.0998
1.0000
0.1816
0.6839
0.8510
0.7584
hollywood2_em
Second time
fixation
0.9697
0.9672
0.9684
0.9207
0.9950
0.9564
hollywood2_em
Second time
saccade
0.7099
0.7265
0.7181
0.8317
0.2255
0.3548
Eye-tracking dataset
First time
fixation
0.9894
0.0055
0.0110
0.9781
0.2450
0.3919
Table 4. We evaluated the I-VT and I-DT algorithms using four datasets, processing each dataset twice. The results differed between the two runs on two of the datasets.
Method
Event
Precision (detailed)
Precision (simple)
Recall (detailed)
Recall (simple)
F1 (detailed)
F1 (simple)
I-VT
fixation
0.9730
0.9736
0.9782
0.9783
0.9756
0.9759
I-VT
saccade
0.8473
0.8471
0.8170
0.8189
0.8319
0.8328
I-DT
fixation
0.9765
0.9764
0.9240
0.9235
0.9495
0.9492
I-DT
saccade
0.6212
0.6193
0.8486
0.8478
0.7173
0.7157
Table 5. The performance comparison between detailed prompts and simple prompts on the GazeCom dataset.
Method
Fixation F1
Fixation precision
Fixation recall
Saccade F1
Saccade precision
Saccade recall
1D CNN-BLSTM ( Startsev et al., 2019 )
0.939
0.915
0.966
0.893
0.895
0.891
[Agtzidis, Startsev, Dorr] ( Agtzidis et al., 2016 )
0.886
0.930
0.846
0.864
0.901
0.829
I-VMP ( San Agustin, 2010 )
0.909
0.864
0.959
0.680
0.799
0.592
REMoDNaV ( Dar et al., 2021 )
0.822
0.925
0.741
0.692
0.548
0.937
[Larsson et al.] ( Larsson et al., 2015 )
0.912
0.872
0.956
0.861
0.881
0.841
[Berg et al.] ( Berg et al., 2009 )
0.883
0.901
0.867
0.697
0.630
0.780
Table 6. Comparison of fixation and saccade detection performance on GazeCom. ( Startsev et al., 2019 )
Aspect
LLM-driven
Traditional
Domain shift
By modifying assumptions based on different sampling rates, noise levels, and task designs, adaptation can be achieved more quickly.
Thresholds and heuristics often need dataset-specific retuning.
Unit consistency
Can silently assume wrong units or scales; requires explicit schema + logging.
Units and transformations are explicit in code; easier to standardize and verify.
Diagnosis
Flags degenerate patterns and suggests likely causes.
Relies on human debugging
Parameter tuning
Provides reasonable initial velocity or dispersion thresholds and supports iterative refinement via feedback.
Expertise-heavy trial-and-error and performance is sensitive to parameter choices.
Auditability
Needs guardrails (structured outputs, unit tests, fixed seeds or versions) to be auditable.
Deterministic and straightforward to audit once code and params are fixed.
Runtime & scalability
Additional LLM latency and cost; multi-round loops increase overhead in large-scale runs.
Low runtime cost after implementation; scales well for batch processing.
Table 7. Key aspects of LLM-driven eye-tracking analysis compared with traditional pipelines.