cs.CVApr 7, 2026

A Synthetic Eye Movement Dataset for Script Reading Detection: Real Trajectory Replay on a 3D Simulator

Authors: Kidus Zewde, Yuchen Zhou, Dennis Ng, Neo Tiangratanakul, Tommy Duong, Ankit Raj, Yuxin Zhang, Xingyu Shen, 

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large vision-language models have achieved remarkable capabilities by training on massive internet-scale data, yet a fundamental asymmetry persists: while LLMs can leverage self-supervised pretraining on abundant text and image data, the same is not true for many behavioral modalities. Video-based behavioral data -- gestures, eye movements, social signals -- remains scarce, expensive to annotate, and privacy-sensitive. A promising alternative is simulation: replace real data collection with controlled synthetic generation to produce automatically labeled data at scale. We introduce infrastructure for this paradigm applied to eye movement, a behavioral signal with applications across vision-language modeling, virtual reality, robotics, accessibility systems, and cognitive science. We present a pipeline for generating synthetic labeled eye movement video by extracting real human iris trajectories from reference videos and replaying them on a 3D eye movement simulator via headless browser automation. Applying this to the task of script-reading detection during video interviews, we release final_dataset_v1: 144 sessions (72 reading, 72 conversation) totaling 12 hours of synthetic eye movement video at 25fps. Evaluation shows that generated trajectories preserve the temporal dynamics of the source data (KS D < 0.14 across all metrics). A matched frame-by-frame comparison reveals that the 3D simulator exhibits bounded sensitivity at reading-scale movements, attributable to the absence of coupled head movement -- a finding that informs future simulator design. The pipeline, dataset, and evaluation tools are released to support downstream behavioral classifier development at the intersection of behavioral modeling and vision-language systems.

Figures & tables

#ModelBackboneAccuracyParamsGPU VRAM
1L2CS-NetResNet-50 + RetinaFace3.92 ∘ MPII∼\sim 24M + 29M det234 MB
2FAZEDenseNet DTED + MAML∼\sim 3.9 ∘ MPII∼\sim 6.9M30 MB
3GazeTRResNet-18 + TransformerSoTA ETH-XGaze∼\sim 11.4M49 MB
4PureGazeResNet-50 + DeconvSoTA domain gen.∼\sim 31.6M134 MB
5MobileGazeResNet-50 + RetinaFace11.33 ∘ Gaze360∼\sim 24M + 29M det228 MB
6ptgazeResNet-18N/A (3 modes)∼\sim 11.7M50 MB
Table 1: Survey of open-source gaze estimation methods. Seven models produced working inference with pretrained weights; 13 were excluded for missing weights, wrong task, or incompatible frameworks.
Figure 1: Generation pipeline overview. Real interview videos are processed through six trajectory-processing stages and finally recorded on the WesternU 3D Eye Simulator via Playwright automation.
Figure 2: Stage 1: Iris landmarks (red dots) define the eye aperture; normalized iris position (nx,ny)(n_{x},n_{y}) is computed relative to corner and eyelid landmarks.
Figure 3: Stage 2: Normalized iris coordinates (nx,ny)∈[0,1](n_{x},n_{y})\in[0,1] are mapped to cursor position (cx,cy)(c_{x},c_{y}) on the 1280 ×\times 720 px simulator canvas via scale and shift.
Figure 4: Stage 3: Speed correction restores trajectory path length after resampling. α≈1.16\alpha\approx 1.16 – 1.45×1.45\times amplifies the smoothed signal back to the original travel distance.
Figure 5: Stage 4: Per-subject normalization centers each subject’s distribution on the global mean, removing inter-subject baseline offsets.
Figure 6: Stage 5: Circular slicing extracts sessions at uniform stride from the concatenated trajectory. Wrap-around (dashed arrow) enables uniform coverage from a finite array.
Figure 7: Stage 6: Playwright headless browser automation drives the WesternU 3D eye simulator by moving the cursor to each trajectory position and recording the resulting eye video.
PropertyValue
Sessions144 (72 R + 72 C)
Duration/session300 s (5 min)
Frame rate25 fps
Resolution1280 ×\times 720
Total duration12 hours
Total size1.7 GB
Table 2: Dataset statistics for final_dataset_v1 .
Figure 8: Q-Q plots: source vs. generated quantiles for speed, fixation duration, and saccade amplitude, across reading and conversation classes. Each panel shows generated quantiles (Y-axis) against source quantiles (X-axis); the dashed line is the 1:1 reference. KS DD -statistics are annotated in each panel.
MetricDD (Reading)DD (Conv.)
Speed (px/frame)0.0830.034
Fixation duration0.1180.006
Saccade amplitude0.1360.079
Table 3: Two-sample KS test statistics comparing source vs. generated distributions. Lower DD indicates better distributional match.
Figure 9: Trajectory time-series (30-second windows). Left: source reference trajectories from real interview videos. Right: corresponding generated sessions. The generated trajectories preserve the characteristic temporal structure: structured horizontal sweeps for reading, irregular center-biased movement for conversation.
MetricValue
Mean iris error (norm.)0.053
Median iris error0.036
95th percentile error0.161
Correlation (X / Y)0.006 / 0.077
Amplitude ratio (X / Y)0.30 / 0.42
Cursor range (X)100 px (7.8% of 1280)
Table 4: Matched iris comparison metrics (Tudimilla, 120 s, 3000 frames).
Figure 10: Sim fidelity metrics from matched comparison. Left: iris position error statistics. Center: amplitude ratio (sim/real). Right: movement variability (std dev) comparison. The simulator produces 30–42% of the real subject’s iris amplitude.
Figure 11: Apple-to-apple qualitative comparison at t=15t{=}15 s. Left: simulator replay (dark background, eye pinned). Right: original human reference video. Both eyes are normalized to a common center; only iris movement is visible. The simulator preserves the temporal pattern of the source trajectory with reduced visible amplitude at this scale.
More in this topic

Synthetic Data Generation

Synthetic data generation creates artificial examples, simulations, and datasets to train, test, or protect machine-learning systems.

See all 620 papers

Explore similar work

CardsList
May 1, 2026

Multimodal Large Language Models as Synthetic Participants in Video-Based Studies: An Evaluation

Multimodal large language models (MLLMs) have shown strong performance on objective tasks such as video understanding and reasoning. However, it remains unclear whether they can approximate subjective human responses, which depend not only on content comprehension but also on individuals' social contexts. To address this gap, we evaluate MLLMs as synthetic participants in an emerging task: assessing perceived sensory engagement with short videos. Grounded in the Perceived Message Sensation Value (PMSV) framework, we compare ratings from recruited human participants and profile-conditioned MLLM simulations (n=673) using a 17-item scale measuring emotional arousal, dramatic impact, and novelty. We find that even leading MLLMs (Gemini 3 Flash and Qwen 3 Omni) show limited agreement with human participants. The models exhibit distinct downward mean-shift and central-tendency biases in their rating distributions. They both introduce and flatten subgroup differences, while showing inconsistent sensitivity to participant profiles. Prompting strategies affect these metrics differently, modestly improving some aspects while worsening others. These results highlight both the challenges and opportunities of developing MLLMs as synthetic participants in video-based research. Data and code: https://github.com/MINDLab25/mllm-human-simulation-eval

Prabal Shrestha, Bohan Jiang, Haoning Xue +2

Department of Computer Science, Boise State University, Boise, ID, USA · School of Computing and Augmented Intelligence, Arizona State University, Tempe, AZ, USA · Department of Communication, University of Utah, Salt Lake City, UT, USA

Multimodal Large Language ModelsLanguage Model Bias EvaluationMultimodal Model Evaluation
AskPDF
May 12, 2026

LychSim: A Controllable and Interactive Simulation Framework for Vision Research

While self-supervised pretraining has reduced vision systems' reliance on synthetic data, simulation remains an indispensable tool for closed-loop optimization and rigorous out-of-distribution (OOD) evaluation. However, modern simulation platforms often present steep technical barriers, requiring extensive expertise in computer graphics and game development. In this work, we present LychSim, a highly controllable and interactive simulation framework built upon Unreal Engine 5 to bridge this gap. LychSim is built around three key designs: (1) a streamlined Python API that abstracts away underlying engine complexities; (2) a procedural data pipeline capable of generating diverse, high-fidelity environments with varying out-of-distribution (OOD) visual challenges, paired with rich 2D and 3D ground truths; and (3) a native integration of the Model Context Protocol (MCP) that transforms the simulator into a dynamic, closed-loop playground for reasoning agentic LLMs. We further annotate scene-level procedural rules and object-level pose alignments to enable semantically aligned 3D ground truths and automated scene modification. We demonstrate LychSim's capability across multiple downstream applications, including serving as a synthetic data engine, powering reinforcement learning-based adversarial examiners, and facilitating interactive, language-driven scene layout generation. To benefit the broader vision community, LychSim will be made publicly available, including full source code and various data annotations.

Wufei Ma, Chloe Wang, Siyi Chen +3

Johns Hopkins University

Synthetic Data Generation3D Scene Generation
AskPDF
Feb 23, 2026

Eye-Tracking-while-Reading: A Living Survey of Datasets with Open Library Support

Eye-tracking-while-reading corpora are a valuable resource for many different disciplines and use cases. Use cases range from studying the cognitive processes underlying reading to machine-learning-based applications, such as gaze-based assessments of reading comprehension. The past decades have seen an increase in the number and size of eye-tracking-while-reading datasets as well as increasing diversity with regard to the stimulus languages covered, the linguistic background of the participants, or accompanying psychometric or demographic data. The spread of data across different disciplines and the lack of data sharing standards across the communities lead to many existing datasets that cannot be easily reused due to a lack of interoperability. In this work, we aim at creating more transparency and clarity with regards to existing datasets and their features across different disciplines by i) presenting an extensive overview of existing datasets, ii) simplifying the sharing of newly created datasets by publishing a living overview online, https://t.uzh.ch/1Yh, presenting over 55 features for each dataset, and iii) integrating all publicly available datasets into the Python package pymovements which offers an eye-tracking datasets library. By doing so, we aim to strengthen the FAIR principles in eye-tracking-while-reading research and promote good scientific practices, such as reproducing and replicating studies.

Deborah N. Jakobi, David R. Reich, Paul Prasse +3

Department of Computational Linguistics, University of Zurich. · Department of Computer Science, University of Potsdam. · Department of Informatics, University of Zurich.

Eye Tracking
AskPDF