The Language of Looking: Motion Tokens Reveal Domain-Dependent Local Structure in Gaze Sequences

Virmarie Maquiling; Enkelejda Kasneci

cs.LGOct 12, 2026

The Language of Looking: Motion Tokens Reveal Domain-Dependent Local Structure in Gaze Sequences

Authors: Virmarie Maquiling, Enkelejda Kasneci

Organizations: Human-Centered Technologies for Learning, Technical University of Munich, Munich Center for Machine Learning (MCML), Munich, Germany

Abstract

Human gaze is the primary high-bandwidth interface through which we parse the semantics of our environment. While often modeled as a sequence, the degree to which gaze exhibits transferable local structure across domains remains underexplored. Using motion tokenization as a diagnostic lens across four diverse datasets, we investigate the transferability of local sequence structure. We demonstrate that while gaze can share a reusable motion-token vocabulary, identical motion tokens are organized into different local transition patterns depending on the dataset and task context. Our results show the limitations of naive data-pooling under low-semantic motion-token representations and provide a framework for evaluating behavioral sequences through information-theoretic diagnostics.

Figures & tables

Figure 1: Motion-tokenization pipeline. Continuous AEA and RITW recordings are segmented into fixations using adaptive I-VT, whereas COCO and ZuCo provide fixation sequences directly. Adjacent fixations are converted into pseudo-angular displacements and quantized using either the fixed AngDelta grid or a frozen VQDelta codebook. For three fixations $(f_{1},f_{2},f_{3})$ , the resulting displacements become token sequence $(a_{i},a_{j})$ under AngDelta or $(v_{p},v_{q})$ under VQDelta. Screen-based pseudo-angles represent normalized screen displacement, not physical eye rotation. The token sequences are used for target-regret evaluation, order controls, motif overlap, and transition diagnostics. Initial alignment entries are omitted for clarity.

Figure 2: Cross-dataset transfer and structural diagnostics under the unified fixation-level benchmark. Panel A reports mean order-3 target regret across five split seeds. Positive values indicate worse prediction than a target-trained reference evaluated on the same target sequences. Panels B and C report the fixed-seed order-sensitivity and bigram-overlap diagnostics. Panel D shows directed target regret averaged across both tokenizers and five split seeds. The “-fix” suffix indicates that non-fixation events were removed from AEA and RITW to match the fixation-native screen datasets.

Figure 3: Interpretable local structure under AngDelta. Panel D averages row-normalized transitions for each dataset pair after projecting onto pair-specific shared top tokens. Shared tokens can therefore induce different local transition systems across regimes. For readability, an 18-token subset of the shared set is shown.

Tokenizer	Dataset	$H(X_{t}\mid X_{t-1})$	Norm. $H$
AngDelta	AEA-fix	3.37	0.58
AngDelta	RITW-fix	2.79	0.45
AngDelta	COCO	5.97	0.77
AngDelta	ZuCo	3.75	0.48
VQDelta	AEA-fix	4.15	0.69
VQDelta	RITW-fix	4.05	0.65

Table 1: Local transition predictability across datasets and tokenizers.

H(X_{t}\mid X_{t-1})

is the raw first-order conditional entropy in bits. Norm.

H

H(X_{t}\mid X_{t-1})/\log_{2}|V_{\mathrm{used}}|

, which partially accounts for differences in observed vocabulary size. Lower values indicate more constrained next-token dynamics.

Appendix figures & tables12 assets

Supplementary material from the paper’s appendix.

Appendix

Dataset	Regime	Setting	Seq.	Train	Val.	Test	Tokens	Med. len.	Mean len.
AEA-fix	Headworn	Everyday activities	145	119	13	13	3017	11.0	20.8
RITW-fix	Headworn	In-the-wild reading	654	514	57	83	70321	72.0	107.5
COCO	Screen	Visual search	24880	19886	2508	2486	93823	3.0	3.8
ZuCo	Screen	Sentence reading	12847	10301	1281	1265	255064	17.0	19.9

Appendix

Table 2: Dataset statistics for the unified fixation-level benchmark. Counts are reported over exported fixation-level token sequences. All datasets use a deterministic hash-based 80/10/10 train/validation/test split with seed 0. The sequence counts are shared across AngDelta and VQDelta exports.

Component	Setting	Value
Data split	Policy	hash-based split on sequence ID
Data split	Fractions	80/10/10 train/val/test
Data split	Seeds	0–4 for target regret; 0 for fixed-seed diagnostics
AngDelta	yaw bins / pitch bins	24 / 18
AngDelta	yaw range / pitch range	$[-120^{\circ},120^{\circ}]$ / $[-120^{\circ},120^{\circ}]$
AngDelta	outside policy	bucket

Appendix

Table 3: Implementation settings for the unified fixation benchmark.

Transfer regime	Tokenizer	Target regret	Order sens.
headworn $\rightarrow$ screen	AngDelta	$2.836\pm 0.022$	$1.250\pm 0.027$
screen $\rightarrow$ headworn	AngDelta	$2.070\pm 0.092$	$0.857\pm 0.158$
within-headworn	AngDelta	$0.421\pm 0.115$	$1.087\pm 0.127$
within-screen	AngDelta	$2.667\pm 0.018$	$4.365\pm 0.155$
headworn $\rightarrow$ screen	VQDelta	$3.071\pm 0.006$	$1.167\pm 0.026$
screen $\rightarrow$ headworn	VQDelta	$2.455\pm 0.140$	$0.627\pm 0.144$

Appendix

Table 4: Target regret and order sensitivity by descriptive transfer regime over five split seeds. Values are means

\pm

standard deviations over seed-level regime means. The regimes are descriptive because task, stimulus, device, preprocessing, representation, sequence length, and dataset size co-vary.

Direction	Tok.	Bigram Jaccard	Trigram Jaccard
aea_fixation_only->coco_search18_tp	ang_delta	0.064 $\pm$ 0.010	0.011 $\pm$ 0.004
aea_fixation_only->ritw_fixation_only	ang_delta	0.501 $\pm$ 0.032	0.381 $\pm$ 0.021
aea_fixation_only->zuco2	ang_delta	0.192 $\pm$ 0.007	0.070 $\pm$ 0.006
coco_search18_tp->aea_fixation_only	ang_delta	0.074 $\pm$ 0.009	0.017 $\pm$ 0.003
coco_search18_tp->ritw_fixation_only	ang_delta	0.064 $\pm$ 0.006	0.022 $\pm$ 0.002
coco_search18_tp->zuco2	ang_delta	0.050 $\pm$ 0.006	0.015 $\pm$ 0.000

Appendix

Table 5: Motif-overlap uncertainty over split seeds for the unified fixation benchmark. Values are mean

\pm

standard deviation across five split seeds.

Tokenizer	Source	Target	IND PPL	OOD PPL	Src.-relative degradation	Order sens.
AngDelta	AEA-fix	COCO	19.305	99.377	4.464	1.230
AngDelta	AEA-fix	RITW-fix	18.457	11.413	-0.339	0.330
AngDelta	AEA-fix	ZuCo	19.197	55.102	2.039	0.504
AngDelta	COCO	AEA-fix	19.330	53.798	1.788	1.159
AngDelta	COCO	RITW-fix	19.330	34.887	0.805	0.251
AngDelta	COCO	ZuCo	19.330	86.588	3.480	1.178

Appendix

Table 6: Absolute in-domain and out-of-domain perplexities for the unified fixation benchmark. Values are multiseed means over split seeds 0–4. Source-relative degradation is presented as a secondary descriptive quantity and is not used for the primary transfer interpretation.

Tokenizer	Order	Target bits	Mean regret	Positive
AngDelta	1	6.776	1.941	60/60
AngDelta	2	6.232	2.030	58/60
AngDelta	3	6.384	2.150	57/60
AngDelta	4	6.460	2.111	57/60
VQDelta	1	7.934	2.674	60/60
VQDelta	2	7.216	2.521	58/60

Appendix

Table 7: N-gram-order controls over 12 directions and five split seeds. Target bits denotes source-trained cross-entropy on target sequences. Positive gives the number of direction–seed evaluations with positive target regret.

Target	Tokenizer	Best multi-source strategy	Regret
AEA	AngDelta	Balanced pooling	-0.113
AEA	VQDelta	Probability mixture	0.292
COCO	AngDelta	Probability mixture	2.102
COCO	VQDelta	Probability mixture	2.215
RITW	AngDelta	Balanced pooling	0.734
RITW	VQDelta	Probability mixture	1.138

Appendix

Table 8: Best leave-target-out multi-source strategy for each target and tokenizer over five split seeds. Regret is measured against the target-trained reference on the same target sequences.

Codebook policy	Target bits	Mean regret	$\Delta$ target bits
Pooled reference	6.745	2.344	0.000
Source-only	6.575	2.232	-0.170
Leave-target-out	6.465	2.283	-0.280

Appendix

Table 9: VQDelta

K=128

codebook-policy controls at split seed 0. The final column is relative to the pooled shared-vocabulary reference. Token identities are not aligned across separately fitted codebooks.

Bigram Jaccard
Tokenizer	Regime family	$k=25$	$k=50$	$k=100$	$k=200$	$k=500$
AngDelta	within-headworn	0.5090	0.4731	0.4667	0.4074	0.3061
AngDelta	headworn $\rightarrow$ screen	0.0974	0.1247	0.1418	0.1361	0.1357
AngDelta	screen $\rightarrow$ headworn	0.0909	0.1159	0.1203	0.1146	0.1165
AngDelta	within-screen	0.0527	0.0365	0.0446	0.0444	0.0983
VQDelta	within-headworn	0.2825	0.2500	0.2499	0.2598	0.2796

Appendix

Table 10: Top-

k

motif-overlap sensitivity in the unified fixation-level benchmark. Values are regime-averaged Jaccard overlaps for top-

k

bigram and trigram motif sets. Across thresholds, within-headworn overlap remains the strongest regime for both tokenizers, while cross-regime and within-screen overlaps remain lower.

Figure 4: Matched-length source-relative-degradation control. All datasets use fixation-level exports, and sequences are capped at 12 tokens before evaluation. The changes in magnitude show that this secondary metric is sensitive to sequence construction. The figure is not used to infer a stable transfer-regime ordering.

Figure 5: Effect of fixation-only conversion on source-relative degradation. The large changes demonstrate sensitivity to event-interface compatibility.

Figure 6: Descriptive COCO $\leftrightarrow$ ZuCo comparison under native and truncate-to-12 evaluation. The panels report both directed transfers using secondary degradation and structural diagnostics.

Domain Generalization

Domain generalization develops models that perform reliably in new environments without training on target-domain data.

See all 375 papers

Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling
Aug 24, 2026Virmarie Maquiling, Zhuojiang Cai, Enkelejda KasneciMotion Representation Learning Cross-Dataset Generalization
Human-Centered Technologies for Learning, Technical University of Munich (TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany
GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction
Oct 20, 2026Trong-Thang Pham, Anh Nguyen, Hien Van Nguyen +1Egocentric Video QA Efficient VLM Inference
Do Motion Tokenizers for Co-Speech Gesture Generation Encode Gesture Semantics?
Sep 28, 2026Varsha Suresh, Divij Jain, Jia Liu +2Co-Speech Gesture Generation Motion Representation Learning
Saarland University · Max Planck Institute for Informatics, Saarland Informatics Campus · IIT Delhi

Aug 24, 2026J/K move · Enter open · S save

Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling

Virmarie Maquiling, Zhuojiang Cai, Enkelejda Kasneci

Human-Centered Technologies for Learning, Technical University of Munich (TUM), Munich, Germany · Munich Center for Machine Learning (MCML), Munich, Germany

Motion Representation Learning Cross-Dataset Generalization Gaze Estimation Egocentric Video Understanding

Open paper AskPDF arXiv

Gaze is increasingly used as an input signal for vision and multimodal models, yet no consensus exists on how to represent it across datasets. Raw traces preserve detail but are noisy and device-dependent, while coarse event labels are easy to model but can discard local motion structure. We formulate event-aligned, fixed-horizon angular displacement as an interpretable, event-conditioned motion vocabulary and compare it with event-only, spatial, absolute-angle, learned vector-quantized, and continuous representations. To assess transfer alongside target predictability and token collapse, our evaluation combines next-token prediction with target-domain regret, low-order target references, paired bootstrap, order sensitivity, motif overlap, and frozen structural probes. In an event-aligned headset benchmark, angular-motion tokens have lower target-domain regret than frozen-codebook VQ tokens in one transfer direction, while the reverse direction is inconclusive. The probes reveal complementary representation properties, and event-only tokens show that low perplexity can retain little motion information. On a third egocentric dataset, a matched comparison of I-VT, native, and frame-span interfaces shows that event construction materially changes transfer: native events have the lowest regret into EGTEA, while frame-span events have zero motif overlap and fail severely as a source. Motion-based tokenization therefore provides a compact representation for event-aligned egocentric gaze streams, while the evaluation identifies how target predictability and event construction shape cross-dataset conclusions.

Abstract

Domain Generalization

Explore similar work

Motion-Based Tokenization for Cross-Dataset Egocentric Gaze Modeling

GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction

Do Motion Tokenizers for Co-Speech Gesture Generation Encode Gesture Semantics?