cs.AIOct 26, 2026

Toward Fast Human Reaction Modeling from High-Frequency Digital Interaction Data

Authors: Emanuel Dopater, Daniel Gavlas, Lukáš Jochymek, Kannan Paul, Miloslav Szczypka, David Vojáček, Josef Žáček, Jan Plucar, 

Organizations: Mesocort AI Tovární 589, 709 00 Ostrava 9, Czechia · Faculty of Electrical Engineering and Computer Science VŠB–Technical University of Ostrava 17. listopadu 2172/15, 70833 Ostrava, Czechia

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

We investigate whether limbic-like signatures of rapid, salience-driven behavior can be identified in high-resolution Counter-Strike 2 telemetry. This exploratory study introduces Ego-Knowledge, a leak-reduced reconstruction of each player's plausible perceptual state, supported by a millisecond-precision visibility engine and a reaction-centered extraction pipeline. Using a neuroscience-motivated 60--120 ms window, we derive coarse fight-or-flight labels from visibility- and damage-triggered events, with back-damage cases enabling constrained reaction-time analysis and manual label checks. We train two telemetry-native models using only information available by the trigger: EMBER (gradient boosting) and ORBIT (a custom transformer). On the full validation set, EMBER achieves 73.5% accuracy and ORBIT 71.2%, versus a 51.5% majority-class baseline, and both generalize to held-out players. The strongest LLM baseline achieves 62.7% on 1,000 validation cases; on these same cases, EMBER (73.6%) and ORBIT (70.9%) perform better (McNemar p=1.3e-9 and 1.5e-6, respectively), with orders-of-magnitude lower inference time and cost, though the comparison is not compute-matched. Including a short post-trigger window changes forecasting into early response recognition, raising accuracy monotonically to 85.15%. We propose this curve as a candidate probe of the transition from rapid to deliberative control, but roughly uniform per-tick gains do not establish that interpretation. The neuroscience framing is hypothesis-generating, not mechanistic: physiological ground truth, suitable control distributions, and reverse-inference limitations remain open. We outline extensions to auditory reconstruction, larger datasets, and telemetry-native foundation models.

Figures & tables

Fig. 1 : High-level architecture of the Mesocort pipeline. Raw CS2 demo telemetry is transformed into reconstructed game state, player-centric Ego-Knowledge, reaction windows, and fight/flight labels used for behavior modeling, prediction, and evaluation.
Feature groupNo.Description
Round and map context5Round-level and map-level state variables.
Player position and state21Player position, view direction, equipment, movement, and status variables.
Aiming and visibility13Crosshair movement, enemy visibility, and enemy proximity features.
Player input actions12Button states and weapon-firing indicators.
Damage and kill events14Tick-level and round-level damage and kill history features.
Teammate features24Position and visibility features for four teammates.
TABLE I : Summary of input feature groups used by the model.
StatisticValue
Core players selected for Phase 150
Average demos collected per core player690
Initial pool of matches∼\sim 35,000
Time span covered up to∼\sim 7 months
Filtered demos in extracted dataset32,205
Total reaction moments271,541
TABLE II : Phase 1 reaction dataset summary.
Fig. 2 : Visibility engine processing flow: on the left, the scene’s depth map showing the silhouette of the target player; on the right, the pixel segmentation result, where green marks the visible portion and red the portion obstructed by geometry (e.g., a wall), from which the per-frame visibility percentage is computed.
Fig. 3 : Overview of the methodology. Pre-reaction prediction uses only information up to the anchor moment; early response recognition may also use a short post-anchor window.
Fig. 4 : Manual annotation tool. The central view reconstructs the situation in two dimensions; the left panel shows synchronized temporal signals of the analyzed player; the remaining panels hold playback, annotation controls, and gameplay highlights.
Fig. 5 : ORBIT architecture. A sequence of NN tick-level tokens, optionally preceded by global context tokens (map, player, rank), is processed by a Transformer encoder; the final representation feeds a classification head that predicts the behavioral response.
ParameterValue
LEARNING_RATE0.02
MAX_DEPTH8
MIN_CHILD_WEIGHT5.0
SUBSAMPLE0.8
COLSAMPLE_BYTREE0.8
REG_LAMBDA1.0
TABLE III : Hyperparameter configuration of EMBER
SetupTrainVal.Test
Player-disjoint16,6092,9938,730
Extended player-disjoint223,16939,6428,730
Train/validation only230,54740,994–
TABLE IV: Number of rows in train, validation, and test splits.
ParameterValue
TaskBinary classification
Target classesfight , flight
Loss functionCross-entropy loss
Token dimension256
Transformer layers2
Attention heads8
TABLE V : ORBIT hyperparameters used in the experiments.
Fig. 6 : Histogram of reaction times for events in which the player is shot in the back. Base data granularity is 15.625 ms. The distribution provides an initial descriptive view of short-latency behavioral structure; number of situations is 20,114.
Fig. 7 : KDE plots comparing reaction times based on different sources of reaction (KDE plot follows the source histogram).
Sequence-history ticksAccuracy (%)
771.20
1470.98
2170.96
3270.58
6468.93
19268.32
TABLE VI : Validation accuracy of ORBIT models trained with different sequence-history lengths in the pre-reaction prediction setting.
Fig. 8 : Training and validation loss of the selected 7-tick ORBIT model in the pre-reaction prediction setting over 50 epochs. The validation loss reaches its minimum around epoch 30 and then increases while the training loss continues to decrease, indicating overfitting in the later stages of training. 4 4 4 We hypothesize that the early onset of overfitting is partly due to the limited size of the training set, since Transformer architectures are known to be data-hungry and to generalize substantially better as the training data grows [ 60 , 61 ] .
StrategyAccuracy (%)Avg. time (s)Total Cost ($)Precision (%)Recall (%)
Gemini 3.7 Flash Medium Narrative (1/8)62.71.814.7059.6677.51
Gemini 3.7 Flash Medium Full-Dump (3/4)61.66.3160.0358.1981.33
Gemini 3 Flash Medium Full-Dump (1/8)59.72.9714.7256.8179.52
Gemini 3 Flash Medium Full-Dump (1/4)57.33.2815.9755.4272.89
Gemini 3 Flash Medium Full-Dump (3/4)57.42.1040.1155.3674.70
Gemini 3 Flash Medium Narrative (3/4)60.35.0716.9857.6976.10
TABLE VII : Performance comparison of prediction strategies; the LLM and Jev rows are over 1,000 predictions from the current validation set.
Fig. 9 : Validation accuracy of ORBIT in the early response recognition setting as the number of future ticks increases. The history window is fixed to 7 ticks, while the future window is extended up to 64 ticks, corresponding to approximately one second after the anchor moment.
Fig. 10 : Validation accuracy of EMBER in the early response setting as a function of future time steps (maximum difference in ticks between stimulus and the latest moment in the model input) and time step (in ticks). The best accuracy for each future time step is highlighted in red.
Fig. 11 : Global perturbation-based feature importance of ORBIT for samples predicted as fight and flight . Importance is measured as absolute change in predicted-class margin, so the plot should be interpreted as model sensitivity rather than causal evidence.
QuantityValue
Manually inspected situations200
Annotators per situation7
Majority-vote threshold>50%>50\%
Situations with majority-vote label165
Number of situations used for EMBER validation119
Consensus coverage82.5%
TABLE VIII : Summary of majority-vote label validation. Agreement with the automatic labeling system is computed on the majority-vote gold-standard subset.
Fig. 12 : Correctly classified fight example: the analyzed player spots the opponent on lower middle and commits to the duel.
Fig. 13 : Incorrectly classified flight example: the analyzed player, with an AK-47 , faces an AWP in a window but continues the duel.
ComponentEst. (ms)
Δstim\Delta_{\mathrm{stim}} : server →\rightarrow player
Server →\rightarrow client (one-way)12–18
Client processing + render pipeline (200–400 FPS)6–15
Display (144–240 Hz scan-out + pixel response)5–12
Interpolation (enemy-in-view trigger only)16–31
Δresp\Delta_{\mathrm{resp}} : player →\rightarrow demo
TABLE IX : Estimated CS2 system-latency contribution to demo-measured reaction time (typical European competitive conditions). Values are order-of-magnitude estimates, not measurements.
Fig. 14 : Visualization of reaction time transformation based on estimated unmeasured delay and measured ping. Top row = raw reaction time; bottom row = reaction time minus ping and lower band, with shifted upper band. Minimum reaction time filter before adjustment: 46.875 ms (3 ticks)
FeatureDescriptionEncoding / scaling
Round and map context
Premier rankPlayer Premier rank.Min–max scaler
Round numberCurrent round index.Embedding layer
Map IDIdentifier of the map being played.Embedding layer
Bomb plantedIndicates whether the bomb is currently planted.Boolean
Remaining round timeRemaining time in the current round.Standard scaler
TABLE X: Description of the input features used by the model. All features are related for the current game tick.
Fig. 15 : Complete ORBIT global feature-importance ranking. Feature importance is measured as the mean absolute change in the predicted fight or flight class margin after replacing each feature with a neutral baseline value. Features are ordered by decreasing importance.
Fig. 16 : Complete global feature-importance ranking for EMBER. Sorted by gain. EMBER uses a subset of previously laid out feature set.
Fig. 17: General serialization pipeline from the source dataset row to LLM input. All benchmarks share row preparation; encodings differ in how features are rendered. All runs attach labeled train examples and one unlabeled query; the model returns structured JSON with label and reasoning .
PatternDescription
Fight patterns
The Stationary HoldRemaining stationary with a primary weapon (especially AWP ) while the crosshair is aligned with a potential entry point.
Aggressive PursuitMoving quickly toward an enemy or tracking a retreating enemy with a combat-ready weapon.
Tactical AmbushEngaging an unaware enemy from a position of advantage, often described as "stalking" or "initiative" .
Weapon SwitchingTransitioning from a knife or utility to a primary weapon immediately upon spotting an enemy.
Flight patterns
TABLE XI : Fight-or-flight patterns in LLM reasoning
More in this topic

Human Behavior Modeling

See all 38 papers

Explore similar work

CardsList
Jun 16, 2026

EgoCS-400K: An Egocentric Gameplay Dataset for World Models

The shift from video generation to interactive world modeling places new demands on data: beyond captioned videos, world models require temporally aligned video-action-language trajectories grounded in the actions, camera motion, states, and events that drive future scene changes. However, such data is difficult to obtain at scale. Web video datasets offer broad visual coverage but lack executable actions and reliable states; robotic datasets provide action and state supervision but are costly and limited in scene diversity; and existing simulators often lack large-scale human-driven interaction trajectories. In this paper, we introduce EgoCS-400K, a large-scale replay-grounded egocentric Counter-Strike dataset for world models, built from public professional CS and CS2 match demos that preserve human gameplay trajectories and enable parsing, replaying, rendering, and temporal alignment. We extract player states, view directions, movements, keyboard/button inputs, view-angle changes, weapon usage, game events, and round-level context, and render clean first-person videos from the same trajectories. EgoCS-400K contains over 400,000 first-person videos and 10,000 hours of gameplay from more than 1,000 matches and 40,000 rounds, covering 13 maps and 10 player viewpoints per round. It supports a range of interactive visual modeling tasks, including action-conditioned future prediction, state- and event-aware scene rollout, replay-grounded captioning, and agent egocentric action understanding. By connecting visual observations with human actions, camera motion, game states, and events at scale, EgoCS-400K serves as a practical bridge between passive web videos, controllable game simulation, and costly real-world embodied data.

Rongjin Guo, Dong Liang, Yuhao Liu +4

City University of Hong Kong

World Model LearningEgocentric Video UnderstandingInteractive World Models
AskPDF
Sep 16, 2026

REACT: A Fully Spiking State-Space Model for Real-Time Event-Driven Temporal Perception

Robotic systems operating in dynamic environments require visual perception that evolves continuously with the incoming sensory stream. Event cameras provide microsecond temporal resolution and asynchronous sensing, but most learning-based methods accumulate events into frames or temporal bins, introducing an integration delay that can limit fast reaction. Here we propose REACT, a fully spiking state-space model for event-driven temporal perception that processes raw events one by one, without temporal accumulation. REACT uses a complex-valued spiking neuron, C-SiLIF, whose continuous-time dynamics are driven by the physical inter-event interval, allowing its internal state to evolve at the temporal resolution of individual events. We evaluate REACT on gesture recognition and time-to-collision (TTC) estimation from full-field event streams, without a target bounding box or localization input. On EvTTC, REACT achieves a 9.59% relative TTC error with 4.6 ms end-to-end inference latency, within 0.15 percentage points of the best learned method while requiring no target prior. At the dataset's mean approach speed, this latency corresponds to only 4 cm of vehicle motion, compared with 1 m for the fastest competing learned method. REACT further supports anytime TTC prediction, zero-shot transfer to a different driving sequence, and INT8 quantization, reducing the estimated energy consumption from 18.5 to 2.8 mJ per 32,768 events. These results show that event-driven spiking state-space dynamics can provide low-latency, continuously updated temporal perception for reactive robotic systems.

Geoffroy Keime, Nicolas Cuperlier, Benoit R. Cottereau

CerCo, CNRS UMR5549 Université de Toulouse Toulouse, France · IPAL, CNRS IRL, Singapore · ETIS, CY Cergy Paris Université CNRS, ENSEA Cergy, France

Collision AvoidanceEvent-Based VisionRobotic PerceptionState Space Models
AskPDF
Oct 12, 2026

Esports World Model Dataset

Interactive world models must capture how multiple participants jointly change a shared environment while observing different parts of it. Competitive games make this challenge concrete: players cooperate and compete, while fog of war, team vision and camera placement determine what each can see. Modeling these interactions requires data that connect individual observations and actions to shared state and events. We introduce the Esports World Model Dataset (EWM), a collection of human gameplay spanning StarCraft II, Dota 2 and Counter-Strike 2, with 3,053 games and maps and approximately 4,880 player-view hours, including estimated hours for the ongoing CS2 extension. It includes 4,628 completed StarCraft II views, all ten perspectives for 303 Dota 2 matches, and up to ten views per round in CS2. Across these views, replay-clock alignment connects rendered video with game-native actions, structured state and events from the same match. The three games expose complementary observation conditions: private fog of war, shared team vision and first-person viewpoints. In StarCraft II, player-specific entity records further distinguish live observations from remembered snapshots and link co-observed entities across opposing views. Together with temporal alignment, these records allow researchers to distinguish legitimate information differences from disagreement about shared state. By connecting human decisions, shared dynamics and player-specific information, EWM supports research toward jointly controlled multiplayer world models and interactive environments populated by teammates and opponents learned from human play.

Weiyu Ma, Zeyuan Gong, Xuhui Liu +5

King Abdullah University of Science and Technology (KAUST) · Institute of Automation, Chinese Academy of Sciences · Beijing Zhongguancun Academy & Zhongguancun Institute of Artificial Intelligence

Interactive World ModelsWorld Model Learning
AskPDF