Conflict-Aware Multi-Teacher On-Policy Distillation in Aligned Action Spaces for Cross-Game Generalization
Organizations: LIGHTSPEED · HKUST(GZ) · HKUST
Abstract
Zero-shot cross-game generalization aims to consolidate accessible source-game experts into a single vision-language model (VLM) that controls unseen games from raw pixels without target-game parameter updates. While multi-teacher on-policy distillation (MOPD) mitigates offline exposure bias and single-game reinforcement learning (RL) overfitting, scaling it to autoregressive VLMs faces two bottlenecks. First, token-to-action misalignment from subword length bias distorts discrete action probabilities and uncertainty margins. Second, multi-teacher policy conflict arises as source teachers under incompatible dynamics issue contradictory closed-loop corrections, causing negative transfer. We propose CAMOPD (Conflict-Aware Multi-Teacher On-Policy Distillation), a post-training framework that resolves multi-teacher policy conflict within an aligned semantic action space. CAMOPD first uses complete-action alignment to normalize command completion likelihoods over each game's legal action set, forming a calibrated categorical policy that provides reliable uncertainty margins for selective on-policy retention and visual specialist probes that can be evaluated across source games. To resolve policy conflict without target-game data, conflict-aware multi-teacher consolidation cross-evaluates these specialists across source games to capture both visual identifiability and inter-teacher compatibility, upweighting compatible teachers while suppressing conflicting corrections. On GameWorld's temporal and directional transfer suites with Qwen3.5-4B, CAMOPD achieves 26/30 (86.7%) and 24/30 (80.0%) Task-1 zero-shot success on unseen GeoDash and Doodle Jump, outperforming prompting, weight-merging, and multi-task baselines.
Figures & tables
| Suite | Control Axis | Source Teachers ( ) | Samples ( / ) | Held-Out Target ( ) | Target Tasks (Threshold) |
|---|---|---|---|---|---|
| A | Temporal (when to act) | Boxel Rebound, Stack | 3,519 / 392 | GeoDash ( wait , jump ) | T1 ( ), T2 ( ) |
| B | Directional (where to steer) | Breakout, NS-Shaft, Cubefield | 2,104 / 107 | Doodle Jump ( wait , move_left/right ) | T1 ( ), T2 ( ) |
| Suite A: Held-Out GeoDash | Suite B: Held-Out Doodle Jump | ||||||||||||||
| Method | Family | Task 1 | Task 2 | Mean | Resets | Task 1 | Task 2 | Mean | Resets | ||||||
| SR | PG | SR | PG | SR | PG | SR | PG | SR | PG | SR | PG | ||||
| Base VLM | pretrained | 0/30 | 92.0 | 0/30 | 46.0 | 0.0 | 69.0 | 66 | 3/30 | 63.3 | 0/30 | 17.3 | 5.0 | 40.3 | 69 |
| Pause-and-Plan CoT [ 34 ] | prompted VLM | 13/30 | 88.6 | 0/30 | 59.0 | 21.7 | 73.8 | 56 | 2/30 | 61.1 | 0/30 | 24.3 | 3.3 | 42.7 | 48 |
| Few-shot ICL | prompted VLM | 5/30 | 73.5 | 0/30 | 47.5 | 8.3 | 60.5 | 205 | 12/30 | 73.0 | 0/30 | 23.1 | 20.0 | 48.1 | 46 |
| Boxel Rebound specialist | single teacher (A) | 14/30 | 84.2 | 0/30 | 49.7 | 23.3 | 67.0 | 69 | – | – | – | – | – | – | – |
| Suite A: Held-Out GeoDash | Suite B: Held-Out Doodle Jump | |||||||||||||
| Ablation Variant | Task 1 | Task 2 | Mean | Resets | Task 1 | Task 2 | Mean | Resets | ||||||
| SR | PG | SR | PG | SR | PG | SR | PG | SR | PG | SR | PG | |||
| Token SFT (Offline Token BC) | 0/30 | 65.1 | 0/30 | 31.1 | 0.0 | 48.1 | 462 | 5/30 | 65.5 | 0/30 | 15.9 | 8.3 | 40.7 | 85 |
| + Complete-Action Alignment | 11/30 | 85.6 | 0/30 | 53.7 | 18.3 | 69.7 | 57 | 2/30 | 65.9 | 0/30 | 27.8 | 3.3 | 46.8 | 71 |
| + Selective MOPD (Uniform) | 7/30 | 79.4 | 0/30 | 47.6 | 11.7 | 63.5 | 87 | 8/30 | 57.7 | 0/30 | 41.9 | 13.3 | 49.8 | 41 |
| + Conflict-Aware Consolidation (Full CAMOPD ) | 26/30 | 95.5 | 0/30 | 58.0 | 43.3 | 76.7 | 36 | 24/30 | 91.9 | 0/30 | 39.4 | 40.0 | 65.7 | 12 |
| Specialist | Val. NLL | ||||||
| Suite A | Boxel | Stack | – | ||||
| Boxel Rebound | 0.059 | 0.706 | – | 0.383 | 0.843 | 1.69 | |
| Stack | 0.806 | 0.468 | – | 0.637 | 0.157 | 0.31 | |
| Suite B | Breakout | Cube. | NS-Shaft | ||||
| NS-Shaft | 2.610 | 1.725 | 0.050 | 1.462 | 0.665 | 2.00 | |
| Breakout | 3.050 | 0.986 | 0.984 | 1.673 | 0.247 | 0.74 | |
| Suite A: Held-Out GeoDash | Suite B: Held-Out Doodle Jump | ||||||||||||||
| Policy Variant | Visual Input & Decision Mode | Task 1 | Task 2 | Mean | Resets | Task 1 | Task 2 | Mean | Resets | ||||||
| SR | PG | SR | PG | SR | PG | SR | PG | SR | PG | SR | PG | ||||
| Passive wait | None (constant wait ) | 0/30 | 30.0 | 0/30 | 15.0 | 0.0 | 22.5 | 1952 | 0/30 | 0.0 | 0/30 | 0.0 | 0.0 | 0.0 | 0 |
| Periodic cadence | None (fixed jump / left–right cycle) | 16/30 | 85.9 | 0/30 | 65.6 | 26.7 | 75.7 | 313 | 0/30 | 35.8 | 0/30 | 18.7 | 0.0 | 27.2 | 14 |
| CAMOPD (frozen ) | Frozen initial frame (static) | 1/30 | 92.3 | 0/30 | 46.0 | 1.7 | 69.1 | 69 | 0/30 | 26.8 | 0/30 | 2.1 | 0.0 | 14.4 | 45 |
| CAMOPD (live) | Live frames (closed-loop) | 26/30 | 95.5 | 0/30 | 58.0 | 43.3 | 76.7 | 36 | 24/30 | 91.9 | 0/30 | 39.4 | 40.0 | 65.7 | 12 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Suite | Target Game ( ) | Native Actions ( ) | Base PG | CAMOPD PG | ||||
|---|---|---|---|---|---|---|---|---|
| Suite A | Core Ball | {wait, shoot} | 0.610 | 0.321 | 0.466 | 37.8 | 57.6 | +19.8 |
| GeoDash | {wait, jump} | 0.576 | 0.427 | 0.502 | 69.0 | 76.7 | +7.7 | |
| Chrome Dino | {wait, jump, duck, start} | 0.718 | 0.474 | 0.596 | 42.1 | 46.0 | +3.9 | |
| Suite B | Doodle Jump | {wait, move_left/right} | 0.415 | 0.049 | 0.232 | 40.3 | 65.7 | +25.4 |
| Edge Surf | {wait, turn_left/right, boost, slow_down} | 0.620 | 0.051 | 0.336 | 50.7 | 97.6 | +46.9 | |
| Pac-Man | {wait, move_left/right/up/down} | 0.627 | 0.051 | 0.339 | 16.0 | 28.7 | +12.7 |
| System Prompt ( ): |
| You are controlling the GameWorld game . Task: <task_prompt> Choose exactly one legal action from: . Return exactly one JSON object and no other text: {"tool_name":"<action>","arguments":{}}. |
| User Input ( ): |
| Candidate Action Completion Suffixes for : |
| : {"tool_name":"wait","arguments":{}} |
| : {"tool_name":"move_right","arguments":{}} |
| Scoring ( Eq. 2 ): Mask all system/user prompt tokens and compute . |
| Suite | Source Game ( ) | Legal Action Set ( ) | Privileged State ( ) | Rule Teacher Decision & Confidence Logic |
| Suite A | Boxel Rebound | {wait, jump} | Ground flag, tile grid, subcell | Scans a short lookahead window ( tiles) for spikes, gaps, or walls while grounded: outputs jump ( ) when a hazard enters the jump window, and wait ( ) when airborne or on clear ground. |
| Stack | {wait, drop_block} | Moving & top block coords/sizes | Computes horizontal overlap ratio with the top stack block: outputs drop_block ( ) when overlap ratio , and wait ( ) while approaching alignment. | |
| Suite B | Breakout | {wait[_long], move_left/right[_long]} | Ball , paddle | Reflects the descending ball trajectory against side walls to predict the paddle-plane intercept: emits move_left/right (or *_long on large intercept gaps, of frames; ) when displacement exceeds deadzone, else wait ( ; runtime action wait_long is unused by ). |
| NS-Shaft | {wait, move_left/right} | Player , platform list | Selects the highest reachable safe platform below the player: steers horizontally toward the platform center ( ) and outputs wait ( ) once aligned within the landing margin. | |
| Cubefield | {wait, move_left/right} | Corridor offset, obstacle cones | Projects forward collision cones against approaching cubes: steers toward the wider clearance gap ( ) under imminent collision risk, else recenters or emits wait ( ). |