Typed Temporal Interaction Features for Simulation-Backed Forecasting of Open-Source Game Release Incidents

Shayma Alkobaisi; Anas Ali

cs.LGSep 12, 2026

Typed Temporal Interaction Features for Simulation-Backed Forecasting of Open-Source Game Release Incidents

Authors: Shayma Alkobaisi, Anas Ali

Organizations: Department of Computer Science and Software Engineering, United Arab Emirates University, Al Ain, United Arab Emirates · Department of Computer Science, National University of Modern languages, Lahore, Pakistan

Abstract

Open-source video-game quality depends on inter-actions among code, assets, configuration, tests, contributors, and issue workflows, yet conventional defect predictors usually flatten or omit these relations. We investigate release-level forecasting of a quality incident within thirty days using GAMEQUALGRAPH-Pilot, a typed temporal feature pipeline with calibrated risk estimates and effort-aware ranking. Because the accessible OS-SGameBench materials do not provide manually audited release dates and outbreak labels, the executed evaluation is explicitly simulation-backed rather than an empirical claim about real games. Five seeded worlds each contain 120 projects and 24 releases, with project-disjoint validation and future cross-project testing. The pilot obtains an AUPRC of 0.520, AUROC of 0.673, Brier score of 0.207, and 29.68% effort-aware recall at a twenty-percent testing budget. Its closest local comparator, Static-Hetero-Reimpl, reaches 0.522 AUPRC; the -0.002 difference is not statistically significant after Holm correction. Inference requires 0.023 milliseconds per release in the measured environment. Ablations and controlled missingness, drift, engine, project-size, alert-threshold, and attribution analyses expose where typed interactions help and where they fail. Results support the reproducibility of the proposed protocol, not deployment effectiveness. Real OSSGameBench release reconstruction, stratified label audits, and official graph-model comparisons remain mandatory before journal submission or operational use in practice. This boundary protects research integrity and supports credible evaluation.

Figures & tables

Study	Task	G	T	A	R	E	Unresolved relation to this work
OSSGameBench [ 4 ]	game repository benchmark	–	✓	✓	–	–	No audited release-outbreak target
GBGallery [ 8 ]	game testing	–	–	✓	–	–	Small runtime-defect benchmark
GlitchBench [ 9 ]	visual glitches	–	–	✓	–	–	No development-artifact trace
Game problems [ 7 ]	postmortems	–	–	✓	–	–	Qualitative and no change mapping
JIT QA [ 1 ]	change risk	–	✓	–	–	✓	Code-change rather than release unit
DeepJIT [ 2 ]	change risk	–	✓	–	–	–	Text/code sequence only

TABLE I: Closest-work matrix. G=graph, T=temporal, A=non-code assets, R=release prediction, E=effort-aware. A dash means the capability is outside the published task, not that the study is deficient.

Symbol	Meaning
$p,r$	project and release indices
$t_{pr}$	release-time information cutoff
$\mathcal{V},\mathcal{E}$	typed nodes and timestamped edges
$\mathcal{K}$	relation-type set
$\bm{x}_{v}$	node or aggregate feature vector
$\bm{h}^{(l)}_{v}$	layer- $l$ representation

TABLE II: Principal notation.

Fig. 1: GAMEQUALGRAPH-Pilot system architecture. Multimodal repository evidence is converted into a timestamp-valid release graph, calibrated incident-risk prediction, and effort-aware testing decision.

Item	Value
Independent seeds	5
Projects per seed	120
Releases per project	24
Generated release rows	14,400
Train projects	80
Validation projects	20

TABLE III: Executed simulation and split statistics.

Component	Setting
Logistic-Code	standardization, balanced logistic regression
RandomForest-All	180 trees, leaf minimum 6
HistGB-All	140 iterations, 15 leaves
Homogeneous-MLP	32–16 units, early stopping
Static-Hetero	six fixed relation summaries
GAMEQUALGRAPH	20 base plus ten relation/temporal summaries; 180 iterations

TABLE IV: Experimental configuration.

Method	AUPRC	AUPRC SD	AUROC	Brier	ECE	ER@20%	ms/release
GAMEQUALGRAPH-Pilot	0.520	0.040	0.673	0.207	0.071	0.297	0.023
HistGB-All	0.515	0.045	0.678	0.208	0.075	0.320	0.017
Homogeneous-MLP	0.438	0.057	0.609	0.218	0.070	0.253	0.007
Logistic-Code	0.509	0.057	0.674	0.208	0.079	0.316	0.005
RandomForest-All	0.499	0.058	0.686	0.207	0.083	0.301	0.053
Static-Hetero-Reimpl	0.522	0.043	0.674	0.208	0.071	0.317	0.023

TABLE V: Cross-project future-holdout results over five seeded simulations. ER@20% is effort-aware recall. Lower is better for Brier, ECE, and latency; higher is better otherwise. These methods share the generated protocol and are directly comparable.

Panel of Figure 7 — Fig. 2: (a) Precision–recall and (b) calibration for the proposed pilot and strongest AUPRC comparator on executed cross-project future holdouts.

Panel of Figure 8 — Fig. 3: Operational sensitivity from executed test predictions: (a) precision and recall across alert thresholds and (b) incident recall across effort budgets.

Panel of Figure 9 — Fig. 4: Generalization panels: (a) mean AUPRC for all local methods and (b) GAMEQUALGRAPH-Pilot AUPRC across simulated engine strata. Method indices follow the main-results table; engine labels are mapped in the source CSV.

Panel of Figure 10 — Fig. 5: (a) Executed modality ablations and (b) permutation AUPRC loss. Numeric labels are mapped to full names in the CSV files and workbook.

Panel of Figure 11 — Fig. 6: Robustness from executed predictions: (a) typed-relation missingness and (b) AUPRC by release index, with the synthetic distribution shift beginning at release 18.

Feature Interaction Modeling

Feature interaction modeling captures how combinations of inputs jointly shape predictions beyond their individual effects.

See all 71 papers

ForecastBench-Sim: A Simulated-World Forecasting Benchmark
Jun 17, 2026Jaeho Lee, Nick Merrill, Ezra KargerForecasting Benchmarks Probabilistic Forecasting
1Forecasting Research Institute · 2UC Berkeley
GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
Apr 3, 2026Shufan Jiang, Chios Chen, Zhiyang ChenAutomated Software Engineering Software Engineering Agents
The University of Hong Kong · Datawhale Org. · Independent Researcher +1
From Explanations to Interventions: Execution-Guided Counterfactual Synthesis in Temporal Graphs
Sep 10, 2026Minwoo Yu, Young-guk HaCounterfactual Explanations Causal Counterfactual Generation
Smart Computing Laboratory, Department of Computer Science & Engineering, Konkuk University, Seoul 05029, Republic of Korea

Jun 17, 2026J/K move · Enter open · S save

ForecastBench-Sim: A Simulated-World Forecasting Benchmark

Jaeho Lee, Nick Merrill, Ezra Karger

1Forecasting Research Institute · 2UC Berkeley

Forecasting Benchmarks Probabilistic Forecasting Counterfactual Prediction Synthetic Benchmark Generation

Open paper AskPDF arXiv

Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score. We introduce ForecastBench-Sim, a simulated-world forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. Forecasters receive a fixed world report (a structured snapshot of the current game state) and answer questions about hidden future states; the benchmark then continues the simulation and scores forecasts. Because the world is simulated, the same setup can generate continuous or binary forecasting questions at arbitrary time horizons, paired intervention worlds for conditional or causal questions, and resolved examples of rare or disruptive outcomes. We describe the benchmark pipeline, question families, scoring protocol, and release artifacts, and report validation slices from model evaluations and an anonymized human pilot. ForecastBench-Sim is intended to complement real-world forecasting benchmarks by providing controlled, immediately resolvable tasks for studying probabilistic reasoning under dynamic world states.

Abstract

Feature Interaction Modeling

Explore similar work

ForecastBench-Sim: A Simulated-World Forecasting Benchmark

GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers

From Explanations to Interventions: Execution-Guided Counterfactual Synthesis in Temporal Graphs