Typed Temporal Interaction Features for Simulation-Backed Forecasting of Open-Source Game Release Incidents
Authors: Shayma Alkobaisi, Anas Ali
Organizations: Department of Computer Science and Software Engineering, United Arab Emirates University, Al Ain, United Arab Emirates · Department of Computer Science, National University of Modern languages, Lahore, Pakistan
Open-source video-game quality depends on inter-actions among code, assets, configuration, tests, contributors, and issue workflows, yet conventional defect predictors usually flatten or omit these relations. We investigate release-level forecasting of a quality incident within thirty days using GAMEQUALGRAPH-Pilot, a typed temporal feature pipeline with calibrated risk estimates and effort-aware ranking. Because the accessible OS-SGameBench materials do not provide manually audited release dates and outbreak labels, the executed evaluation is explicitly simulation-backed rather than an empirical claim about real games. Five seeded worlds each contain 120 projects and 24 releases, with project-disjoint validation and future cross-project testing. The pilot obtains an AUPRC of 0.520, AUROC of 0.673, Brier score of 0.207, and 29.68% effort-aware recall at a twenty-percent testing budget. Its closest local comparator, Static-Hetero-Reimpl, reaches 0.522 AUPRC; the -0.002 difference is not statistically significant after Holm correction. Inference requires 0.023 milliseconds per release in the measured environment. Ablations and controlled missingness, drift, engine, project-size, alert-threshold, and attribution analyses expose where typed interactions help and where they fail. Results support the reproducibility of the proposed protocol, not deployment effectiveness. Real OSSGameBench release reconstruction, stratified label audits, and official graph-model comparisons remain mandatory before journal submission or operational use in practice. This boundary protects research integrity and supports credible evaluation.
Figures & tables
Study
Task
G
T
A
R
E
Unresolved relation to this work
OSSGameBench [ 4 ]
game repository benchmark
–
✓
✓
–
–
No audited release-outbreak target
GBGallery [ 8 ]
game testing
–
–
✓
–
–
Small runtime-defect benchmark
GlitchBench [ 9 ]
visual glitches
–
–
✓
–
–
No development-artifact trace
Game problems [ 7 ]
postmortems
–
–
✓
–
–
Qualitative and no change mapping
JIT QA [ 1 ]
change risk
–
✓
–
–
✓
Code-change rather than release unit
DeepJIT [ 2 ]
change risk
–
✓
–
–
–
Text/code sequence only
TABLE I: Closest-work matrix. G=graph, T=temporal, A=non-code assets, R=release prediction, E=effort-aware. A dash means the capability is outside the published task, not that the study is deficient.
Symbol
Meaning
p,r
project and release indices
tpr
release-time information cutoff
V,E
typed nodes and timestamped edges
K
relation-type set
xv
node or aggregate feature vector
hv(l)
layer- l representation
TABLE II: Principal notation.
Fig. 1: GAMEQUALGRAPH-Pilot system architecture. Multimodal repository evidence is converted into a timestamp-valid release graph, calibrated incident-risk prediction, and effort-aware testing decision.
Item
Value
Independent seeds
5
Projects per seed
120
Releases per project
24
Generated release rows
14,400
Train projects
80
Validation projects
20
TABLE III: Executed simulation and split statistics.
Component
Setting
Logistic-Code
standardization, balanced logistic regression
RandomForest-All
180 trees, leaf minimum 6
HistGB-All
140 iterations, 15 leaves
Homogeneous-MLP
32–16 units, early stopping
Static-Hetero
six fixed relation summaries
GAMEQUALGRAPH
20 base plus ten relation/temporal summaries; 180 iterations
TABLE IV: Experimental configuration.
Method
AUPRC
AUPRC SD
AUROC
Brier
ECE
ER@20%
ms/release
GAMEQUALGRAPH-Pilot
0.520
0.040
0.673
0.207
0.071
0.297
0.023
HistGB-All
0.515
0.045
0.678
0.208
0.075
0.320
0.017
Homogeneous-MLP
0.438
0.057
0.609
0.218
0.070
0.253
0.007
Logistic-Code
0.509
0.057
0.674
0.208
0.079
0.316
0.005
RandomForest-All
0.499
0.058
0.686
0.207
0.083
0.301
0.053
Static-Hetero-Reimpl
0.522
0.043
0.674
0.208
0.071
0.317
0.023
TABLE V: Cross-project future-holdout results over five seeded simulations. ER@20% is effort-aware recall. Lower is better for Brier, ECE, and latency; higher is better otherwise. These methods share the generated protocol and are directly comparable.
Fig. 2: (a) Precision–recall and (b) calibration for the proposed pilot and strongest AUPRC comparator on executed cross-project future holdouts.
Fig. 3: Operational sensitivity from executed test predictions: (a) precision and recall across alert thresholds and (b) incident recall across effort budgets.
Fig. 4: Generalization panels: (a) mean AUPRC for all local methods and (b) GAMEQUALGRAPH-Pilot AUPRC across simulated engine strata. Method indices follow the main-results table; engine labels are mapped in the source CSV.
Fig. 5: (a) Executed modality ablations and (b) permutation AUPRC loss. Numeric labels are mapped to full names in the CSV files and workbook.
Fig. 6: Robustness from executed predictions: (a) typed-relation missingness and (b) AUPRC by release index, with the synthetic distribution shift beginning at release 18.
Forecasting benchmarks for general-purpose AI systems usually inherit the constraints of the real world: outcomes resolve slowly, tail events are rare, and counterfactual questions are difficult to score. We introduce ForecastBench-Sim, a simulated-world forecasting benchmark built on game rollouts from Freeciv, a turn-based strategy game modelled on the Civilization series. Forecasters receive a fixed world report (a structured snapshot of the current game state) and answer questions about hidden future states; the benchmark then continues the simulation and scores forecasts. Because the world is simulated, the same setup can generate continuous or binary forecasting questions at arbitrary time horizons, paired intervention worlds for conditional or causal questions, and resolved examples of rare or disruptive outcomes. We describe the benchmark pipeline, question families, scoring protocol, and release artifacts, and report validation slices from model evaluations and an anonymized human pilot. ForecastBench-Sim is intended to complement real-world forecasting benchmarks by providing controlled, immediately resolvable tasks for studying probabilistic reasoning under dynamic world states.
The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.
Shufan Jiang, Chios Chen, Zhiyang Chen
The University of Hong Kong · Datawhale Org. · Independent Researcher +1
Can a trace explaining model execution also compute the changes needed for a specified alternative prediction? We propose trace-guided intervention search, which uses executable reasoning traces as an intermediate representation for intervention synthesis. A Specified-Foil Counterfactual edits past events so that a frozen temporal predictor selects a designated foil. Our method constructs facts and replacement values from completed original and foil executions and recovered unmet conditions. Proposal generation constructs edits and selects candidates within a fixed cap; exact replay verifies foil top-1 outcomes among retained edits and compositions. Implemented in LiFTER for continuous-time dynamic graphs (CTDGs) and TLogic for temporal knowledge graphs (TKGs), the method improves success over coordinate-based proposal generation by 13.7-34.7 percentage points on four CTDG datasets and 60.0-83.3 points on two TKG datasets under matched downstream search and a proposal cap of 32. Separate shared-candidate comparisons retain 85.7-93.6% of black-box greedy's CTDG success rate with 75.0-80.0% fewer predictor evaluations. A Pulse case study confirms simulator-level survival for five of six interventions. Executable traces thus provide both explanatory evidence and a reusable computational representation for constructing and testing specified alternatives.
Minwoo Yu, Young-guk Ha
Smart Computing Laboratory, Department of Computer Science & Engineering, Konkuk University, Seoul 05029, Republic of Korea