Reusing Multimodal Agents: Efficient Evaluation of Sound Designs in Games
Authors: Thai Nguyen Van, Ibrahim Khan, Mury Fajar Dewantoro, Pratch Suntichaikul, Hai Van Pham, Ruck Thawonmas
Organizations: Graduate School of Information Science and Engineering, Ritsumeikan University, Japan · College of Information Science and Engineering, Ritsumeikan University, Japan · Independent Researcher · School of Information and Communications Technology, Hanoi University of Science and Technology, Vietnam
This paper introduces a novel framework for efficiently evaluating in-game sound designs using multimodal learning and reinforcement learning. Sound design significantly influences gameplay and accessibility; however, objective evaluation methods, defined here as automated assessments based on agent performance rather than human ratings, often require retraining agents from scratch for each sound design. Our approach first pre-trains a multimodal encoder that encodes visual and auditory information into a shared latent space. A reinforcement learning agent is then trained using this latent representation. We fine-tune the multimodal encoder for each sound design while keeping the trained policy fixed, allowing us to isolate the effects of sound changes on agent behavior. To evaluate the proposed method, we conducted experiments on games from multiple genres, comparing the performance of blind agents (agents using only sound input) under various sound designs. Experimental results suggest that this approach can detect differences in agent performance caused by degraded or altered sound design. Moreover, we significantly reduce the computational cost compared to retraining the agent from scratch. Experimental results suggest that this approach provides a practical and scalable solution for evaluating in-game sound designs. The source code, data and supplementary document are available: https://github.com/thainv0212/tog-reusing-agents.
Figures & tables
Fig. 1: Our method pipeline and agent evaluation setting. Only audio is used in the blind setting, while both images and audio are used for the standard setting.
Fig. 3: Multimodal encoder architecture. Audio, Image, and fused modalities are processed by dedicated feature extractors and MLPs before converging into a shared Common MLP module.
Fig. 4: PENDULUM sound designs. (a) Design 1: three listeners positioned at the bottom left and right, and middle top. (b) Design 2: same as (a) but the three listeners are moved to the top left and right, and bottom middle. (c) Design 3: same as (b) but regions where the sound cue is indistinguishable are shaded. (d) Design 4: same as (b) but regions where the sound cue is indistinguishable are shaded.
Fig. 5: HYPERHOT sound designs. (a) Design 1: four listeners (white dots) receive distinct sound cues from enemies (yellow) and bullets (green/blue). (b) Design 2: same as (a), but listener positions are changed. (c) Design 3: enemies and bullets emit the same pitch, and only one listener remains at the bottom left. (d) Design 4: only two enemies and two bullets emit sound, enemies and bullets emit the same pitch, and two listeners remain at the bottom left and bottom right.
Fig. 6: VizDoom basic scenario sound designs. The listener is shown as a green dot at the bottom. (a) Design 1: enemy emits a sound cue whose volume decreases linearly with distance. (b) Design 2: same as (a) but the emitted volume decreases quadratically with distance. (c) Design 3: the player can only hear the enemy’s sound if it is on the left side of the player’s view. (d) Design 4: there is no sound cues indicating the presence of enemy.
Game
Design 1
Design 2
p -value
95% CI
Pendulum
-1.020 ± 0.033
-1.028 ± 0.048
0.273
[-0.016, 0.044]
Hyperhot
1.728 ± 0.183
1.780 ± 0.124
0.364
[-0.299, 0.160]
ViZDoom
0.798 ± 0.014
0.810 ± 0.011
0.076
[-0.027, 0.006]
TABLE I: Comparison of finetuned agent performance between the baseline (Design 1) and Design 2 across three games. Statistical significance is evaluated using the Mann-Whitney U test, and the 95% confidence intervals (CI) for the difference in medians are calculated using a two-sample independent percentile bootstrap method (number of resamples: 9999).
Game
Method
Design 2
Design 3
Design 4
PENDULUM
Ours
-1.028 ± 0.048
-1.382 ± 0.043
-2.464 ± 0.109
No fine-tuning
-5.897 ± 0.298
-5.807 ± 0.057
-5.437 ± 0.065
RL training under blind setting
-1.005 ± 0.091
-2.236 ± 0.068
-2.719 ± 0.244
Finetuned + Policy Training
-1.030 ± 0.090
-1.198 ± 0.047
-2.298 ± 0.152
HYPERHOT
Ours
1.780 ± 0.124
0.953 ± 0.088
0.904 ± 0.081
No fine-tuning
1.701 ± 0.157
0.877 ± 0.091
0.877 ± 0.091
TABLE II: Comparison of agent performance across Designs 2, 3, and 4 in different games. No fine-tuning: The multimodal encoder is trained only on Design 1 and is not fine-tuned.
Game
Method
D2 vs D3
D3 vs D4
D2 vs D4
Average Effect size
PENDULUM
Ours
1.8×10−4/0.84
1.8×10−4/0.84
1.8×10−4/0.84
0.84
RL training under blind setting
1.8×10−4/0.84
3.3×10−4/0.80
1.8×10−4/0.84
0.83
Finetuned + Policy Training
4.4×10−4/0.79
1.8×10−4/0.84
1.8×10−4/0.84
0.82
HYPERHOT
Ours
1.7×10−4/0.84
0.29/0.24
1.8×10−4/0.84
0.64
RL training under blind setting
4.5×10−2/0.44
1.8×10−4/0.84
1.8×10−4/0.84
0.70
Finetuned + Policy Training
1.9×10−2/0.52
2.4×10−4/0.82
1.8×10−4/0.84
0.73
TABLE III: Pairwise performance comparisons ( p -values/effect size) using Mann-Whitney U test of sound designs. D2: Design 2, D3: Design 3, D4: Design 4.
Game
Method
Samples
Trainable Parameters
Backward FLOPs
Training Steps
Total FLOPs
PENDULUM
Ours
20,000
9,664
0.25 GFLOP
70,499
17 TFLOP
RL training under blind setting
150,000
340,268
0.19 GFLOP
150,000
29 TFLOP
Finetuned + Policy Training
20,000 + 150,000
9,664 + 275,972
0.25 GFLOP + 0.36 GFLOP
70,499 + 150,000
17 TFLOP + 54 TFLOP
HYPERHOT
Ours
32,000
9,140,288
2.58 GFLOP
56,250
145 TFLOP
RL training under blind setting
2,000,000
9,248,220
4.8 GFLOP
2,000,000
9632 TFLOP
Finetuned + Policy Training
32,000 + 2,000,000
9,248,220 + 48,140
2.58 GFLOP + 0.024 GFLOP
56,250 + 2,000,000
145 TFLOP + 48 TFLOP
TABLE IV: Comparison of training cost in PENDULUM, HYPERHOT and Vizdoom.
Game
Sound Design
Image-Sound
Image-Joint
Sound-Joint
Average
PENDULUM
Design 1
1.00
1.00
1.00
1.00
Design 2
0.99
1.00
0.99
0.99
Design 3
0.83
0.98
0.87
0.89
Design 4
0.80
0.98
0.85
0.88
HYPERHOT
Design 1
0.68
0.44
0.52
0.55
Design 2
0.64
0.45
0.50
0.53
TABLE V: Latent representation similarities across sound designs in PENDULUM, HYPERHOT and VizDoom.
Game
Design 3
Design 4
Pendulum
-1.081 (+0.301)
-1.893 (+0.571)
Hyperhot
1.538 (+0.585)
1.073 (+0.169)
VizDoom
0.272 (+0.73)
-1.631 (+0.941)
TABLE VI: Performance of fine-tuned policy agents using Design 3 and Design 4 across three games. Values in parentheses denote the performance improvement relative to baseline agents where only the multimodal encoder was fine-tuned
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex: the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only 53.8% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from 51.4% on gameplay-oriented tasks to 33.0% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image- and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from 41.1% to 52.0% when given visual feedback.
Wayne Chi, Yixiong Fang, Arnav Yayavaram +8
Carnegie Mellon University, Pittsburgh, PA, USA · Princeton University, Princeton, NJ, USA
Towards an embodied generalist for real-world interaction, Multimodal Large Language Model (MLLM) agents still suffer from challenging latency, sparse feedback, and irreversible mistakes. Video games offer an ideal testbed with rich visual observations and closed-loop interaction, demanding fine-grained perception, long-horizon planning, and precise control. However, systematically evaluating these capabilities is currently hindered by heterogeneous action interfaces and heuristic verification. To this end, we introduce GameWorld, a benchmark designed for standardized and verifiable evaluation of MLLMs as generalist game agents in browser environments. Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space via deterministic Semantic Action Parsing. GameWorld contains 34 diverse games and 170 tasks, each paired with state-verifiable metrics for outcome-based evaluation. The results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games. Extensive experiments of repeated full-benchmark reruns demonstrate the robustness of the benchmark, while further studies on real-time interaction, context-memory sensitivity, and action validity expose more challenges ahead for game agents. Together, by offering a standardized, verifiable, and reproducible evaluation framework, GameWorld lays a robust foundation for advancing research on multimodal game agents and beyond. The project page is at https://gameworld-bench.github.io.
Mingyu Ouyang, Siyuan Hu, Kevin Qinghong Lin +2
National University of Singapore · University of Oxford
Recent game design agents have made substantial progress in generating playable games. However, program correctness does not ensure an enjoyable experience for players. We present Recursive Game Creator, an experience-oriented harness to advance agentic game development from rough game prototypes into entertaining games. Recursive Game Creator organizes recursive development around four components: Designer, Builder, Player, and Reviewer. The Designer translates user instructions and Reviewer's feedback into detailed plans. The Builder turns these plans into candidate games. The coding-native Player creates and executes reusable policies through programmatic interfaces to efficiently collect diverse gameplay trajectories, mitigating evaluation bias caused by slow GUI-based collection. The Reviewer uses carefully designed trajectory-based metrics to induce player preferences, integrating with visual evidence and explicit textual preferences to evaluate games against game-specific criteria. Finally, the Reviewer accepts the better version and provides improvement reviews for the next round, closing the recursive loop. Our method achieves state-of-the-art overall performance of 77.89 on GameCraft-Bench. On GameASG-Bench, it achieves a strict task success rate of 53.2%, a 34.1% improvement over the same-model baseline, and the highest mean runtime-check pass rate at 93.4% among compared methods. A user study shows longer playtime and higher ratings. Project page is at: https://imbaldy.github.io/recursive-game-creator/.
Jiajun Chen, Haoyu Wu, Mingda Jia +1
HKU MMLab · The University of Hong Kong · Shenzhen Loop Area Institute