GTAGym: AAA Game Tasks Environment for Evaluation and Reinforcement Learning
Organizations: Shanghai Innovation Institute · Fudan University
Abstract
AAA games offer a rich testbed for evaluating agent perception, reasoning, and control. However, evaluating whether agents can complete tasks in AAA games is challenging and often human-annotated because task outcomes are often difficult to verify automatically, let alone applying it for Reinforcement Learning (RL). We introduce GTAGym, an environment and framework for evaluation and RL on GTA V tasks. Its evaluation benchmark, GTAVBench, derived from GTA V main-story missions, contains 103 online tasks reconstructed across nine task families, alongside 47 image-based shooting and sniping tasks using GTA V imagery. A web server provides access to the online environment and evaluation interface, supporting parallel testing through screenshots and keyboard-mouse actions with automatic engine-state outcome checks. We evaluate six frontier LLMs on GTAVBench with textual task instructions and on its GTAVBench-AGI branch, where instructions appear only in the first screenshot and a harness automatically compresses interaction history. The best configuration achieves 70.9% overall accuracy in GTAVBench-AGI, yet aerial navigation, evasion and interception remain difficult. GTAGym also provides a scalable generator that samples task parameters within user-specified ranges and option sets under fixed templates, creating batches of 3D and image-based shooting environments for RL with automatic reward and termination signals. LoRA RL of Qwen3.8-27B on generated on-foot navigation tasks improves the in-domain performance and transfers to out-of-domain task categories, and native GTA V tasks from 15% to 23%.
Figures & tables
| Work | 3D interaction | AAA open-world tasks | State-based scoring | Web execution | RL instance generation |
|---|---|---|---|---|---|
| Procgen [ 3 ] | |||||
| MineDojo [ 5 ] | |||||
| VideoGameBench [ 25 ] | |||||
| GameVerse [ 27 ] | |||||
| GameWorld [ 13 ] | |||||
| OmniGameArena [ 8 ] |
| 3D Environments | Image-Based Environments | |||||||||||
| Navigation | Interaction | Gun Use | ||||||||||
| Model | Road | On-Foot | Aquatic | Vertical | Aerial | Indoor | Following | Interception | Evasion | Shooting | Sniping | Overall |
| No. of tasks | 20 | 16 | 11 | 5 | 8 | 8 | 6 | 13 | 16 | 24 | 23 | 150 |
| GTAVBench | ||||||||||||
| GPT-6-Astra | 66.7±8.5 | 97.9±2.9 | 97.0±4.3 | 100.0±0.0 | 0.0±0.0 | 95.8±5.9 | 83.3±0.0 | 5.1±3.6 | 18.8±5.1 | 76.4±3.9 | 88.4±4.1 | 66.4±3.1 |
| GPT-6.1-Sol | 63.3±2.4 | 91.7±2.9 | 97.0±4.3 | 100.0±0.0 | 0.0±0.0 | 100.0±0.0 | 72.2±7.9 | 5.1±3.6 | 22.9±2.9 | 81.9±2.0 | 85.5±4.1 | 66.0±0.9 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Category | Mission requirements |
|---|---|
| Online 3D track: Navigation | |
| Road | Reach a destination in a road vehicle |
| Aerial | Takeoff, aerial objectives, and landing |
| Aquatic | Depth, boarding, and shore-arrival objectives |
| Indoor | Reach interior goals and interact with doors or objects |
| On-Foot | Reach a destination on foot |
| Field | Parameters |
|---|---|
| keys | List of key and duration_seconds pairs |
| mouse.move | Relative pixel displacement |
| mouse.scroll | Direction (up/down), integer steps – |
| mouse.buttons | Left/right button with down/up event |
| wait_seconds | Timed wait with an empty keyboard array |
| Cap | Pause | Rounds | Game (s) | (s) | In (M) | Out (k) |
|---|---|---|---|---|---|---|
| 0.5 | Yes | 121.2 | 59.7 | 0.494 | 11.20 | 7.66 |
| 1 | Yes | 62.6 | 56.9 | 0.916 | 4.32 | 4.27 |
| 2 | Yes | 42.5 | 61.5 | 1.468 | 2.50 | 3.10 |
| 1 | No | 31.8 | 448.3 | 0.914 | 1.83 | 2.10 |
| Model | Reasoning effort |
|---|---|
| GPT-6-Astra | high |
| GPT-6.1-Sol | high |
| DeepSeek-V4.1-Flash-1017 | max |
| GLM-5.3-Flash | max |
| Kimi-K3 | max |
| Qwen3.8-27B | xhigh |
| Parameter | Type | User-controlled sampling domain |
|---|---|---|
| Start | Position | Coordinates sampled within , intersected with valid spawn locations in the fixed scene. |
| Destination | Position | Coordinates sampled within , intersected with reachable goal locations. The interval constrains elevation and thus underwater depth. For following and interception, this is the target’s destination. |
| Heading | Real angle | An interval in degrees for the agent’s initial yaw. Other pose components follow template initialization rules. |
| Route length | Real distance | An interval in meters for the path length. Applies to the agent’s reference route for navigation and the target’s route for following or interception. |
| Turn count | Integer | Integers in under the template’s turn-count convention, applied to the same route as . |
| Transport | Category | An explicit allowed set specifies the transport domain, filtered by template support. Movement specifies the concrete mode within that domain. |
| Navigation | Interaction | Gun Use | ||||||||
| Sampling parameter | Road | On-Foot | Aquatic | Vertical | Aerial | Indoor | Following | Interception | Evasion | Shooting |
| Start | — | |||||||||
| Destination | — | — | ||||||||
| Heading | — | |||||||||
| Route length | — | — | ||||||||
| Turn count | — | — | ||||||||
| Task category | Tasks | Source missions in GTA V |
|---|---|---|
| On-Foot Navigation | 9 | Architect’s Plans; Bury the Hatchet; Chop; Eye in the Sky; Franklin and Lamar; Friend Request; Mr. Richards; The Construction Assassination; The Long Stretch. |
| Road Driving | 21 | Architect’s Plans; By the Book; Casing the Jewel Store; Chop; Cleaning Out the Bureau; Deep Inside; Father/Son; Friends Reunited; Meltdown; Mr. Philips; Reuniting the Family; Scouting the Port; The Big Score; The Bureau Raid; The Bus Assassination; The Merryweather Heist. |
| Aquatic Navigation | 12 | Daddy’s Little Girl; Derailed; Hood Safari; Minisub; The Merryweather Heist. |
| Vertical Navigation | 7 | Casing the Jewel Store; Complications; Minisub; The Bureau Raid; The Time’s Come. |
| Aerial Navigation | 9 | Minor Turbulence; Monkey Business; Nervous Ron; Predator; The Bureau Raid; The Merryweather Heist; Three’s Company. |
| Indoor Navigation | 9 | Crystal Maze; Friend Request; Meltdown; The Big Score; The Bureau Raid; The Paleto Score. |
| Category | Common error conditions |
|---|---|
| Perception | Misreading visible objects, route geometry, or road connectivity; confusing targets or minimap markers; misaligning camera, map, and movement reference frames. |
| Planning | Skipping a required interaction or movement mode; choosing incompatible subgoals or an incorrect stage order; repeating an ineffective strategy despite sustained evidence of no progress. |
| Calibration | Selecting the wrong control axis or polarity; overshooting or undershooting because an input’s magnitude or duration is inappropriate for the immediate adjustment. |
| Regulation | Initiating or ending a maneuver too early or late; failing to brake or stabilize during approach; neglecting changes in ego-motion, target distance, or threats. |
| Category | Observation and action | Next observation | Failure mechanism |
|---|---|---|---|
| Road navigation | Action: W 1 s + D 1 s. Kimi-K3 approaches the bend at 24.2 m/s. | Speed rises to 30.5 m/s as the vehicle turns approximately , ending on the road edge and pavement. | Excessive steering while accelerating. A full-second steering input carries the vehicle onto the curb rather than aligning it with the road. The next action continues accelerating without a steering correction; the vehicle subsequently travels onto the grass. |
| Interception | Action: W 0.8 s + A 0.22 s. Impact has slowed the target to 0.2 m/s, but the player is still moving at 37.5 m/s. | The player moves 20.4 m and loses contact, still at 36.8 m/s. No blocking hold accumulates. | Excess speed after impact. Rather than braking immediately to establish a blocking position, the agent continues applying throttle and steers away from contact. It brakes only afterward; the target later resumes motion and reaches its endpoint. |
| Aerial navigation | Action: S 1 s. The aircraft is near the runway at approximately 18.6 m height, descending at 9.3 m/s. | Height falls to approximately 9.4 m while descent remains 9.2 m/s. | Late descent-rate correction. Reducing throttle does not sufficiently arrest the rapid descent. The next action adds a brief pitch-up input, but touchdown still ends in a crash. |
| Evasion | Action: W 1 s + D 0.4 s. The previous forward input ended in a building collision, reducing speed from 43.7 m/s to zero. | The vehicle remains pressed against the wall: the entire action produces zero displacement. | Collision and wall entrapment. Continued forward input with steering cannot free the vehicle from the building. The agent starts reversing only on the next decision, after spending a full second unable to move away from the wall. |
| Capability | Observation | Model output action | Why this is an error |
|---|---|---|---|
| Perception | Qwen3.8-27B | ⬇ { " keys ": [ {" key ": " w ", " duration_seconds ": 1}, {" key ": " Shift ", " duration_seconds ": 1} ], " mouse ": {" move ": {" dx ": 0, " dy ": 0}} } | The model reads the minimap destination as “very close, just ahead,” although the purple route and yellow marker lie to the right of the player marker. It keeps running straight into the grass instead of correcting toward the indicated route. |
| Planning | Qwen3.8-27B | ⬇ { " keys ": [ {" key ": " w ", " duration_seconds ": 1.0}, {" key ": " Shift ", " duration_seconds ": 1.0} ], " mouse ": {} } | The model notices a parked car but decides to keep running toward the distant goal. It never attempts to board a vehicle and remains on foot until the countdown expires. Its plan omits the transport stage needed to finish in time. |
| Calibration | Kimi-K3 | ⬇ { " keys ": [ {" key ": " w ", " duration_seconds ": 1}, {" key ": " d ", " duration_seconds ": 1} ], " mouse ": {} } | While accelerating into the bend, the agent holds right steering for a full second. The prolonged input turns the vehicle too far, carrying it off the road onto the pavement and grass. The steering duration is too large for the required heading correction. |
| Regulation | DeepSeek-V4.1-Flash | ⬇ { " keys ": [ {" key ": " w ", " duration_seconds ": 1}, {" key ": " Shift ", " duration_seconds ": 1} ], " mouse ": {} } | The companion’s route bends, but the agent keeps running straight without adjusting its heading. It moves off the road and repeats the input as the companion gap grows. The episode later fails when separation exceeds 20 meters. |
| Model | Perception | Planning | Calibration | Regulation | Overall |
|---|---|---|---|---|---|
| Astra | 20.0 | 100.0 | 69.2 | 67.6 | 65.7 |
| Sol | 0.0 | 40.0 | 53.6 | 60.0 | 49.3 |
| DeepSeek | 20.0 | 26.3 | 22.7 | 21.1 | 22.2 |
| GLM | 29.2 | 21.4 | 25.9 | 19.4 | 24.0 |
| Kimi | 8.0 | 33.3 | 20.7 | 25.6 | 21.6 |
| Qwen-27B | 26.7 | 28.9 | 24.0 | 21.4 | 25.6 |
| Capability / model | Error state | Corrective action | Restored progress |
|---|---|---|---|
| Perception Qwen3.8-27B | Confuse corridor direction After action: mouse (-250, 0) | Recognize the alternating views Before action: W 1s | Advance along the corridor After action: W 1s |
| The agent alternates left/right views while inconsistently interpreting the corridor relative to the character. It then recognizes the oscillation, stops rotating the camera, and makes two consecutive forward moves along the corridor. | |||
| Planning GPT-6-Astra | Leave the required vehicle After action: F 1s | Return and request boarding Before action: F 1s | Resume driving toward the goal After action: W 1s |
| Exiting the car interrupts a task that requires driving. The agent returns to the vehicle, boards again, and resumes forward travel. | |||
| Calibration GPT-6-Astra | Oversteer onto the verge After action: W 1s; A 0.8s | Apply a shorter steering correction Before action: W 1s; D 0.18s | Continue within the road After action: W 1s; A 0.17s |
| A 0.8-second left input carries the car off the road. Subsequent shorter steering inputs bring it back into the lane and restore forward progress. | |||
AI Agent Benchmarks
AI agent benchmarks measure how reliably autonomous systems plan, use tools, interact, and complete tasks across controlled and real-world settings.
See all 1,581 papers