Faynt: Scaling and Optimizing Policies for Competitive Melee

Ali Janati; Nikita Kuzmin; Rohit Swamy; Charles Niu

cs.LGOct 1, 2026

Faynt: Scaling and Optimizing Policies for Competitive Melee

Authors: Ali Janati, Nikita Kuzmin, Rohit Swamy, Charles Niu

Organizations: Frisson Labs

Abstract

We introduce Faynt, a family of 10M- and 75M-parameter Transformer policies for Super Smash Bros. Melee, each controlling all 26 characters with a single checkpoint. After reinforcement learning (RL), the 10M wins 240 of 244 same-character games (98.4%) against fourteen specialist and multi-character releases on their supported rosters, with a winning record against every release. These opponents retain 21- or 24-frame action delays; Faynt uses no added delay, and we have not isolated the effect of this difference. In a separate evaluation against a privately supplied zero-delay Slippi-AI model, the 10M wins all 68 games across two conditioning settings. We study architecture, optimization, scaling, and hyperparameter transfer to guide pretraining on approximately 840,000 human replays. Post-training combines rank- and outcome-based curricula, 75M-to-10M distillation, and RL restricted to Fox mirror matches. On the initial 152-game benchmark, the supervised 10M wins 69.7% of games, compared with 45.4% for the pretrained 75M, despite higher overall held-out controller-prediction loss. The weighted validation loss used for supervised checkpoint selection agrees with the win-rate ordering of all four pretrained and supervised policies. After supervised post-training, both models take less damage per minute, build larger early leads, and win more often after losing the first life. Optimized inference on recorded game states averages 5.2 ms per decision for the 10M and 8.7 ms for the 75M on an NVIDIA T4, excluding emulator execution and communication. We open-source the weights, both benchmark suites, and a platform for automated model tournaments.

Figures & tables

Configuration	Faynt-10M	Faynt-75M
Trainable parameters	10,163,629	75,305,709
Model width	384	768
Transformer blocks	5	11
Query / key-value heads	6 / 2	12 / 3
Head dimension	64	64
Feed-forward width	768	1,920

Table 1: Model configurations. Parameter counts include the frame encoder, Transformer, and controller heads.

Figure 1: Faynt policy architecture. Dashed connections supply current controller categories directly to the output heads. The main-stick head also conditions on the next joint-controller category, using the recorded target during imitation training and a sample during play. AttnRes denotes Full Attention Residuals.

Figure 2: Compute allocation and optimizer selection. Left: observed endpoints and fitted fixed-compute curves; dashed segments extend below the tested size ranges, and stars mark optima found by extrapolation. Right: six Muon configurations at 20M parameters and 166M targets, with the best AdamW run as reference. Muon’s reference uses learning rate 0.02, momentum 0.95, Nesterov, and five Newton–Schulz iterations; row labels identify individual changes.

Figure 3: Successive halving on the full corpus. Curves join scheduled validation evaluations and end when a recipe is eliminated (crosses); stars mark the halving selections. Counts give active trials. The 10M panel shows selection through rung 3: lower weight decay wins narrowly, while lower Muon learning rate is ultimately selected after recovery. The 75M winner continues alone in rung 5.

Figure 4: Final pretraining runs. Top: validation trajectories before divergence and calibration-law predictions; stars mark the selected checkpoints. Bottom: full continuations, with manual stops dotted. The 10M finalists (navy: lower Muon learning rate; gray: lower weight decay) remain within 0.0005 NLL through 8B targets, then differ in stability. Points are recorded evaluations; lines join successive observations.

Figure 5: Post-training validation loss and learning-rate schedules. Open circles mark each round’s source evaluation; stars mark retained checkpoints. The 10M continues through two distillation rounds. The dashed 75M continuation is stopped at the crossed endpoint. Rates are normalized to the reference learning rates in Appendix E.2 .

Setting	75M	10M, run 1	10M, leash run
Starting policy	supervised 75M	supervised 10M	run 1, step 1,318
Self-play share of policy loss	50%	50%	100%
Learning rate, policy and value	$4\times 10^{-5}$	$10^{-4}$	$10^{-4}$
Policy-gradient weight $w_{\mathrm{PG}}$	1	1	3
Reference KL weight $\lambda_{\to}$	0.1	0.1	0.003
Reference KL weight $\lambda_{\leftarrow}$	0	0	0.003

Table 2: RL settings and checkpoint budgets. Each run retains its model’s supervised endpoint as the reference policy. The leash run inherits the value network from step 1,318 and resets both optimizers. Training time is summed over launches and includes in-loop evaluation; the selected 10M lineage totals 63.5 hours. Common settings and implementation details are in Appendix E.6 .

Figure 6: RL training and evaluated checkpoints. Stars mark the evaluated 10M and 75M policies; the smaller star marks the 10M checkpoint from which the leash run continues. The diamond is the earlier 75M checkpoint used in Section 6 . Dashed segments continue past selected checkpoints; thin segments precede the implementation speedup described in Appendix E.6 .

Figure 7: In-loop evaluation against MIMIC during RL. Each round contains 25.6 seconds on each of 30 emulators. Stock margin is stocks taken minus stocks lost per minute. Open markers follow the checkpoints used; dotted lines mark 75M step 980 and 10M step 632, which falls between evaluation rounds at steps 583 and 706. The 10M continuation starts 45 hours into its first run.

Figure 8: Initial benchmark results before reinforcement learning. Bars show win rates and win counts in each policy’s 152-game evaluation. Baseline schedules substitute the supervised 75M Faynt comparator, so opponent composition differs between Faynt and the reference views. Slippi-AI includes medium-v2 and the scheduled specialists. All reported outcomes have completed the validation checks in Appendix F.3 .

Figure 9: Expanded evaluation after reinforcement learning. Cells show Faynt wins/games; color encodes win rate. Row groups contain five single-character specialists, seven twelve-character releases, and two four-character releases. Supported and forced mirrors jointly cover all 26 characters; extended-roster games keep the opponent on a supported fighter. Column pairs compare 75M RL step 980 and 10M RL step 632. Asterisks mark cells containing separately rerun games (Appendix F.6 ).

Figure 10: Inference optimization and the 60 Hz decision budget. Left: real-time factor on synthetic inputs, with darker bars showing successive optimizations. Right: mean processing time and 99th-percentile latency on recorded-game inputs, including feature conversion. Dashed lines mark one frame. All timings use random weights and exclude the emulator and communication with it; output parity is tested separately with trained checkpoints (Appendix H ).

Figure 11: Prediction loss and win rate can favor different models. The four pretrained and supervised checkpoints are evaluated on fixed validation data and the original 152-game suite. Open markers denote pretrained models; filled markers denote supervised models. Arrows connect checkpoints of the same size. The supervised 10M wins 37 more games than the pretrained 75M despite higher overall loss. Their weighted losses follow their ranking by win rate.

Figure 12: Faynt takes less damage and builds larger early leads. Each size uses the same match configurations across the two stages: 145 at 75M and 139 at 10M. Damage rates are percentage points received per minute, averaged equally across games. Stock lead is the difference in remaining lives after one minute; terminal scores are retained for games that end earlier. The last panel shows the fraction of time Faynt spends in damage-reaction states.

Figure 13: More wins from early leads and deficits. Win rates after taking or losing the first stock on the two-stage comparison set. Labels give wins/games; simultaneous first-stock losses are excluded.

Figure 14: Stock preservation after supervised training. Top: total stock losses, highlighting those at 50% damage or less. Bottom: their share of all stocks lost, with counts. Both stages use the same 145 configurations at 75M and 139 at 10M.

Figure 15: Playing strength improves after reinforcement learning. Open and filled markers show the supervised and RL checkpoints against one privately supplied zero-delay Slippi-AI model under two conditioning settings. Labels give wins/games: 16 games on Final Destination (FD) and 18 across six stages per setting. The RL checkpoints are 10M step 632 and 75M step 980. Appendix F.13 gives the full win/loss records.

Figure 16: 75M performance at two RL checkpoints. Markers show wins in 16 Final Destination games per opponent; labels give step 222 and step 980 counts. Both checkpoints win 18/18 against every opponent in the separate six-stage evaluation (Appendix F.12 ).

Appendix figures & tables47 assets

Supplementary material from the paper’s appendix.

Appendix

	Calibration	Final corpus
Accepted replays	67,129	839,942
Training replays	65,824	823,083
Validation replays	659	8,483
Test replays	646	8,376
All valid targets	1,500,000,244	17,839,509,998
Training targets	1,470,952,166	17,481,727,198

Appendix

Table A.1: Dataset builds. A target is one valid transition from frame

t

t+1

for one player perspective. Manifest identifiers are the abbreviated values recorded in the study. The final split identifier is 9c4ae826ae97 .

AdamW
Learning rate	Batch	NLL
0.0010	65,536	0.860504
0.0003	65,536	0.870815
0.0030	65,536	0.875388
0.0001	65,536	0.914326
0.0010	262,144	0.914411

Appendix

Table C.1: Complete optimizer screen. Muon uses auxiliary AdamW at 0.001 and shared weight decay 0.01. N denotes Nesterov momentum; NS is the number of Newton–Schulz steps. Bold entries are the selected endpoints.

Size	Setting varied	$0.5\times$	$1\times$	$2\times$
5M	Muon rate	0.943590	0.941835	0.939627
5M	Auxiliary rate	0.950677	0.941835	0.926854
5M	Weight decay	0.939114	0.941835	0.937286
50M	Muon rate	0.915042	0.912638	0.927680
50M	Auxiliary rate	0.920879	0.912638	0.914731
50M	Weight decay	0.924369	0.912638	0.917862

Appendix

Table C.2: Hyperparameter transfer at 41.94M targets per run. Each row varies one setting from the selected 20M recipe. The baseline is shared across the three rows at each size, yielding fourteen distinct runs.

Profile	Parameters	$G_{p}$
3M	3,003,373	12,847,328
5M	5,053,805	29,304,944
10M	10,163,629	72,262,720
20M	20,040,877	151,385,600
50M	50,127,021	394,013,760
75M	75,305,709	596,823,632

Appendix

Table C.3: Exact architecture costs and calibration endpoints.

G_{p}

is FLOPs per target. C1, C2, and C3 denote 0.0052, 0.0104, and 0.0208 EFLOP. The smallest measured model has the lowest loss in each group.

10M
Recipe	Multipliers	R1	R2	R3	R4	R5
wd-low	$1,1,0.8$	0.81019	0.80431	0.801534	0.99334	0.94213
muon-low	$0.8,1,1$	0.81022	0.80387	0.801560
aux-low	$1,0.8,1$	0.81252	0.80572
center	$1,1,1$	0.81220	0.80635
aux-high	$1,1.25,1$	0.81294

Appendix

Table C.4: All 24 full-corpus recipes and 47 rung evaluations. Empty cells follow elimination. For interaction recipes,

\ell=1/\sqrt{2}

and

h=\sqrt{2}

; their three-letter names give the multipliers in order. The 75M local-fit recipe uses

(0.77798125836,1.2289867927,1.1422710484)

	10M		75M
Rung	Updates	Active	Updates	Active
1	38,730	8	7,526	16
2	77,461	4	15,052	8
3	116,192	2	22,579	4
4	154,923	1	30,105	2
5	193,654	1	37,631	1

Appendix

Table C.5: Matched-target halving schedule. Active counts refer to the trials evaluated at the end of each rung. The 10M winner continues alone in rungs 4 and 5; the 75M winner continues alone in rung 5.

Recipe	Best step	Validation	Test	Stop targets
10M Muon-low	122,064	0.796817	0.797658	11,811,160,064
10M WD-low	122,064	0.797050		9,663,676,416
75M Muon-low	86,016	0.764821	0.765435	7,516,192,768

Appendix

Table C.6: Recovery checkpoints and complete continuation lengths. The test split is evaluated once for each frozen winner. A blank test entry indicates that the runner-up was excluded from this evaluation.

Model (M)	CPU cores	Microbatch / accumulation	Workers / prefetch	Targets/s, thousands (95% interval)	USD / 1B targets
RTX PRO 6000
20.04	8	64/4	6/2	165.0 [160.3, 170.8]	6.60
20.04	16	64/4	12/1	146.8 [142.9, 150.1]	8.14
20.04	32	64/4	0/2	187.1 [179.1, 195.5]	7.51
50.13	8	64/4	0/2	131.3 [125.4, 136.3]	8.30
50.13	16	64/4	8/2	139.2 [136.3, 142.4]	8.58

Appendix

Table D.1: Complete selected execution and delivery recipes. Microbatch size counts 256-frame windows; accumulation preserves the 65,536-target global batch. Bold rates identify the fastest CPU tier for each GPU/model pair. Prefetch is inactive when there are zero workers; its stored setting is retained. All hosts have 64 GiB RAM. Costs use the GPU, CPU, and memory rates recorded on August 27, 2026 and exclude startup.

Fixed-core fits
GPU	CPU cores	$A$ (k/s)	$\beta$	$R^{2}_{\log}$	Max. residual
RTX PRO 6000	8	139.9	0.159	0.785	6.2%
RTX PRO 6000	16	133.2	0.121	0.815	4.5%
RTX PRO 6000	32	128.7	0.371	0.870	10.4%
H100 SXM	8	180.7	0.751	0.952	13.8%
H100 SXM	16	138.6	0.477	0.652	30.5%

Appendix

Table D.2: Throughput coefficients and fit diagnostics. The normalization size is 50,127,021 parameters. Each fixed-core fit uses three measurements at one CPU allocation; each compact fit uses the best measured CPU tier at each size. Maximum residual is

\max_{i}|T_{i}-\widehat{T}_{i}|/\widehat{T}_{i}

over the three anchors.

Compact-policy candidates, $100
Model (M)	GPU (8 CPU cores)	Hours	Targets/s	Targets (B)	Pred. NLL
5.054	A100	29.50	1,545,647 E	164.148	0.763553
7	A100	29.50	1,071,942 E	113.840	0.756780
8	A100	29.50	922,620 E	97.982	0.754180
10.164	A100	29.50	705,074 E	74.879	0.749773
Larger-policy candidates, $200

Appendix

Table D.3: Fixed-dollar planning comparisons. M denotes measured throughput, F fitted interpolation, and E extrapolation. Bold entries are selected within each candidate set. Exposures count processed targets, including repeated frames. The 75M budget includes the measured cold-start allowance.

Pool	Replays	Eligible perspective
Natural distribution	823,083	Both players
All decided games	823,057	Winner
Master or diamond winners	471,316	Winner
Master winners	256,532	Winner
Diamond winners	214,784	Winner
Diamond subset in later mixture	28,504	Winner

Appendix

Table E.1: Eligible replay pools. Later rounds use every master-winning replay and a deterministic sample of diamond-winning replays, giving approximately 90% master and 10% diamond visits.

Slice	Perspectives	Targets
Master won	2,635	674,560
Master lost	1,932	494,592
Diamond won	2,235	572,160
Diamond lost	2,208	565,248
Platinum won	3,612	924,672
Platinum lost	4,342	1,111,552

Appendix

Table E.2: Rank–outcome validation slices. An additional undecided master game contributes two perspectives and 512 targets; it is excluded from the six decided-outcome slices.

Model	Stage	Frames	$L$	Master-won	$W$
10M	Natural	270	0.77510	0.79191	0.79024
10M	All winners	270	0.77233	0.78693	0.78543
10M	Master + diamond winners	360	0.77166	0.77798	0.77708
10M	Master winners	900	0.77282	0.77078	0.77080
75M	Natural	180	0.74519	0.75550	0.75426
75M	All winners	180	0.74230	0.74977	0.74873

Appendix

Table E.3: Round 1 stage endpoints. Frames are targets processed within the stage, in millions.

L

is overall validation NLL and

W

is the weighted selection loss. The full-corpus stage precedes every filtered stage.

Round	Model	Horizon	Schedule	Objective
1	10M / 75M	1.8B / 1.2B	Cosine $0.1\to 0.001$	NLL
2	10M / 75M	1.2B / 1.5B	Hold $h=0.1$	NLL
3	10M	900M	Hold $h=0.1$	KD, $\alpha=0.5$ , $T=1$
3, stopped	75M		Hold $h=0.1$	NLL
4	10M	900M	Hold $h=0.05$	KD, $\alpha=0.5$ , $T=1$
4, control	10M	300M	Hold $h=0.05$	KD, $\alpha=0.7$ , $T=1$

Appendix

Table E.4: Executed rounds. Horizons are planned processed targets. A blank horizon for the stopped 75M continuation indicates that only its 449M-target stopping point is used here. KD controls each run for 300M targets.

		$\alpha=0.7$		$T=2$
Targets (M)	Main $W$	$W$	$L$	$W$	$L$
0	0.75990	0.75984	0.77071	0.75982	0.77070
50	0.76086	0.76075	0.77074	0.76268	0.77257
100	0.76106	0.76085	0.77042	0.76230	0.77175
150	0.76075	0.76041	0.77141	0.76289	0.77362
200	0.76053	0.76025	0.76993	0.76289	0.77246

Appendix

Table E.5: Round 4 control trajectories. All runs start at step 181,516. The 50–200M evaluations share the held rate; warmup lasts 6M targets for controls and 18M for the main run. Controls begin decaying at 240M, and the 900M main run at 720M.

L

is overall NLL.

Step	Targets (M)	$W$	Overall NLL	Gradient median
127,214	0	0.72492	0.74149
127,976	50	0.72756	0.74431	0.1205
128,738	100	0.72734	0.74400	0.1255
129,500	150	0.72771	0.74383	0.1263
130,262	200	0.72771	0.74504	0.1304
131,024	250	0.72790	0.74235	0.1350

Appendix

Table E.6: The 75M continuation after round 2. The learning-rate multiplier is 0.1 at every trained checkpoint. All metrics are validation measurements except the interval median of the training gradient norm.

Model	Checkpoint	Step	Overall	Master-won	Diamond-won	$W$
10M	Pretrained	122,064	0.79682	0.81315	0.79691	0.81152
10M	Round 1	149,512	0.77281	0.77076	0.77094	0.77078
10M	Round 2	167,800	0.77220	0.76381	0.77001	0.76443
10M	Round 3	181,516	0.77071	0.75889	0.76825	0.75983
10M	Round 4	195,248	0.76896	0.75647	0.76641	0.75746
75M	Pretrained	86,016	0.76481	0.77617	0.76467	0.77502

Appendix

Table E.7: Validation metrics at every selected checkpoint. Round names follow the chronological experiments. Values come from the selected checkpoint rows; final alignment steps can differ slightly from those rows.

Model	Checkpoint	Overall	Master-won	Diamond-won	$W$
10M	Pretrained	0.79766	0.79694	0.79928	0.79717
10M	Round 1	0.77356	0.75469	0.77223	0.75644
10M	Round 2	0.77291	0.74758	0.77119	0.74994
10M	Round 3	0.77137	0.74280	0.76886	0.74541
10M	Round 4	0.76974	0.74044	0.76722	0.74312
75M	Pretrained	0.76543	0.75915	0.76714	0.75995

Appendix

Table E.8: Test results at the same checkpoints. Every selected round improves all four reported metrics relative to the preceding selected round.

	10M		75M
Slice	Pretrained	Round 1	Pretrained	Round 1
Master won	0.81315	0.77078	0.77617	0.73566
Master lost	0.79648	0.76365	0.76250	0.73142
Diamond won	0.79691	0.77098	0.76467	0.73810
Diamond lost	0.79777	0.77552	0.76672	0.74479
Platinum won	0.79945	0.78122	0.76855	0.75292

Appendix

Table E.9: Every rank–outcome slice before and at the end of round 1. Both sizes improve on every slice relative to pretraining, with the largest reductions on master-winning actions.

Model	Checkpoint	Button NLL	Stick NLL	Button $H$	Stick $H$
10M	Pretrained	0.26732	0.52950	0.2612	0.5300
10M	Round 4	0.25502	0.51394	0.2498	0.5178
75M	Pretrained	0.25287	0.51194	0.2521	0.5127
75M	Round 2	0.24311	0.49835	0.2367	0.5001

Appendix

Table E.10: Controller-head diagnostics on all validation slices.

H

denotes the mean predictive entropy of a head, in nats. The final checkpoints retain button entropy within 10% of their pretrained values.

Model	Targets (M)	$L$	$W$
10M	799	0.77472	0.76962
10M	849	0.77552	0.76960
10M	899	0.77423	0.76979
10M	949	0.77441	0.76973
75M	1049	0.74495	0.72936
75M	1099	0.74367	0.72905

Appendix

Table E.11: Final constant-rate evaluations in round 2. Targets count additional training within the round. The learning-rate multiplier is 0.1 throughout.

L

is overall validation NLL;

W

is the weighted selection loss.

Figure E.1: Post-training diagnostics. (a) Weighted validation NLL and training gradient-norm medians rise during the additional 75M cycle, which is stopped at 449M targets. (b) The 10M controls at evaluations with the same held rate of 0.05: teacher weight 0.7 gives lower weighted NLL at all four points, while temperature 2 gives higher NLL. (c) Each run’s own final decay, shown as a percentage of its budget. The dashed line is the temperature control’s source loss, 0.75982; its trained endpoint is worse and the source is retained. Lines connect recorded observations.

Size	Stage	Checkpoint step	Original run designation
75M	Pretrained	86,016	Final pretraining
75M	Supervised post-training	127,214	cur-3
10M	Pretrained	122,064	Final pretraining v2
10M	Supervised post-training	195,248	cur-6-kd
75M	RL	980	RL run
10M	RL	632	RL leash run

Appendix

Table F.1: Frozen Faynt checkpoints. The supervised post-training stage includes curriculum learning for both sizes and knowledge distillation for 10M.

Block	Games	Assignment
MIMIC Master Fox	10	Fox mirrors against fox-master
CPU9 Fox	10	Fox mirrors against the level-9 controller
MIMIC core mirrors	12	Six native characters, two games each
Slippi-AI core mirrors	24	Twelve native characters, two games each
Roster extension	96	Allocation in Table F.3

Appendix

Table F.2: Original external-panel allocation for each Faynt checkpoint.

Opponent	Games	Character and checkpoint assignment
CPU9	28	All fourteen fighters in $E$
MIMIC native	34	Seventeen native fighters outside the common core
MIMIC forced	6	fox-master on Kirby, Zelda, and Pichu
Slippi specialists	4	Donkey Kong: dk_d18_imitation_v2 ; Dr. Mario: doc_d18_imitation_v3
Slippi forced	24	medium-v2 on $E$ except Donkey Kong and Dr. Mario

Appendix

Table F.3: The 96 original extension games. Each assigned fighter is played twice. Forced mirrors place the opponent on a fighter outside its deployed roster.

Policy	Opponent	W–L	Margin	W–L	Margin
		All 26 characters		Common native core
Faynt 10M	MIMIC	35–17	$+1.02$	8–4	$+0.67$
Faynt 10M	Slippi-AI	34–18	$+0.79$	4–8	$-0.83$
Faynt 75M	MIMIC	46–6	$+1.71$	10–2	$+1.08$
Faynt 75M	Slippi-AI	40–12	$+1.31$	7–5	$+0.17$
Slippi-AI	MIMIC	34–18	$+0.38$	11–1	$+1.42$

Appendix

Table F.4: Original pairwise results. Faynt checkpoints are supervised post-trained. Margins are stocks taken minus stocks conceded per game. The common-core columns reuse a subset of the full-roster games.

Release	Roster	Delay	Conditioning
SFIL	1	21	SFAT
diamond	12	21	Master Player
falco_d21_ditto_v4	1	21	Ginger
fox_d21_ditto_hax_v3	1	21	Hax
fox_d21_ditto_v4	1	21	Cody
fox_d24_ditto_v4	1	24	Cody

Appendix

Table F.5: Expanded opponent configurations. Roster size is deployed character count; delay is in game frames.

	Pretrained		Supervised
Block	W–L	Stocks	W–L	Stocks
MIMIC Fox	1–9	20–39	3–7	32–32
CPU9 Fox	8–2	38–22	10–0	40–7
MIMIC core	2–10	33–46	10–2	45–32
Slippi-AI core	2–22	55–93	15–9	82–75
Extension	56–40	320–278	85–11	373–195

Appendix

Table 46

	Pretrained		Supervised
Block	W–L	Stocks	W–L	Stocks
MIMIC Fox	1–9	15–38	5–5	33–32
CPU9 Fox	8–2	38–23	10–0	40–7
MIMIC core	2–10	24–45	8–4	40–32
Slippi-AI core	1–23	38–95	8–16	71–87
Extension	37–59	274–316	75–21	355–219

Appendix

Table 47

MIMIC : block	Games	W–L	Stocks
Faynt Fox	10	7–3	32–32
CPU9 Fox	10	10–0	40–15
Faynt mirrors	12	2–10	32–45
Slippi-AI mirrors	24	2–22	59–93
Roster extension	96	24–72	232–337
Total	152	45–107	395–522

Appendix

Table 48

Slippi-AI : block	Games	W–L	Stocks
MIMIC Master Fox	10	8–2	37–27
CPU9 Fox	10	10–0	40–4
MIMIC mirrors	12	11–1	46–29
Faynt mirrors	24	9–15	75–82
Roster extension	96	34–62	233–322
Total	152	72–80	431–464

Appendix

Table 49

		75M RL		10M RL
Opponent release	Games	W–L	Stocks	W–L	Stocks
fox_d21_ditto_v4	12	2–10	20–46	12–0	48–4
fox_d24_ditto_v4	12	3–9	33–45	12–0	48–6
SFIL	12	9–3	44–32	12–0	48–1
fox_d21_ditto_hax_v3	12	12–0	48–23	12–0	48–2
falco_d21_ditto_v4	12	1–11	22–47	11–1	47–22

Appendix

Table 50

		75M RL		10M RL
Opponent release	Games	W–L	Stocks	W–L	Stocks
fox_d21_ditto_v4	50	7–43	106–191	35–15	177–127
fox_d24_ditto_v4	50	17–33	135–174	43–7 ∗	188–100
SFIL	50	15–35	146–174	41–9	188–98
fox_d21_ditto_hax_v3	50	30–20	173–148	46–4	193–72
falco_d21_ditto_v4	50	4–46	97–193	35–15	175–128

Appendix

Table 51

		75M RL		10M RL
Opponent release	Games	W–L	Stocks	W–L	Stocks
fox_d21_ditto_v4	50	50–0	200–18	50–0 ∗	200–13
fox_d24_ditto_v4	50	50–0 ∗	200–19	50–0 ∗	200–9
SFIL	50	50–0	200–20	50–0	200–17
fox_d21_ditto_hax_v3	50	50–0	200–11	50–0 ∗	200–7
falco_d21_ditto_v4	50	50–0	200–16	50–0	200–16

Appendix

Table 52

Opponent release	Games	75M RL	10M RL
fox_d21_ditto_v4	62	52–10	62–0
fox_d24_ditto_v4	62	53–9	62–0
SFIL	62	59–3	62–0
fox_d21_ditto_hax_v3	62	62–0	62–0
falco_d21_ditto_v4	62	51–11	61–1
gm	52	34–18	52–0

Appendix

Table F.6: Full-roster mirror results after RL. Entries are Faynt wins–losses against each frozen release. Supported and forced components are listed separately in Appendices F.8 and F.10 .

	10M RL 632		75M RL 222		75M RL 980
Opponent	FD	Six stages	FD	Six stages	FD	Six stages
delay0/FoxFD	16–0	18–0	15–1	18–0	14–2	18–0
delay0/FalcoFD	16–0	18–0	16–0	18–0	14–2	18–0
FoxFD0	16–0	18–0	16–0	18–0	16–0	18–0
MarthFD0	16–0	18–0	15–1	18–0	16–0	18–0
PeachFD	16–0	18–0	16–0	18–0	16–0	18–0

Appendix

Table F.7: Results against zero-delay specialists. Entries are Faynt wins–losses. FD denotes Final Destination. Each checkpoint plays 112 games in the FD condition and 126 in the six-stage condition, using the same weights across characters.

Faynt checkpoint	FD	Six stages	FD	Six stages	Total
	Master Player		Cody
10M supervised	6–10	9–9	3–13	5–13	23–45
10M RL step 632	16–0	18–0	16–0	18–0	68–0
75M supervised	9–7	8–10	5–11	3–15	25–43
75M RL step 980	16–0	17–1	13–3	12–6	58–10

Appendix

Table F.8: Results against the privately supplied Slippi-AI checkpoint. Entries are Faynt wins–losses. The two column groups use different conditioning settings of the same opponent model. FD denotes Final Destination; the six-stage condition contains three games per stage. Supervised denotes the post-trained policy before RL.

Model	Stage	Games	Wins	Duration	Faynt dmg/min	Opp. dmg/min	Lead
75M	Pretrained	145	65	193.2	106.4	122.1	$-0.055$
	Supervised	145	117	184.5	95.8	151.4	$+0.531$
10M	Pretrained	139	46	177.1	118.2	110.2	$-0.317$
	Supervised	139	95	186.5	98.7	144.6	$+0.460$

Appendix

Table G.1: Two-stage results on the same configurations. Duration is in seconds. Damage received (dmg/min) is in damage percentage points per minute, averaged equally across games. Stock lead is evaluated at 60 seconds.

Model	Stage	Win after taking first	Win after conceding first
75M	Pretrained	42/65 (64.6%)	23/80 (28.8%)
	Supervised	96/107 (89.7%)	19/36 (52.8%)
10M	Pretrained	40/57 (70.2%)	6/82 (7.3%)
	Supervised	76/93 (81.7%)	19/46 (41.3%)

Appendix

Table G.2: Wins after taking or conceding the first stock on the common two-stage cohorts. Each entry gives wins divided by eligible games, followed by the percentage. Simultaneous first-stock losses are excluded.

Model	Stage	Low-percent / all losses	Share	Mean loss percent
75M	Pretrained	86/456	18.9%	99.6
	Supervised	51/325	15.7%	108.9
10M	Pretrained	98/469	20.9%	92.2
	Supervised	47/348	13.5%	106.4

Appendix

Table G.3: Faynt stock losses on the common cohorts. Low-percent losses occur at 50% damage or less, using the last recorded value before the loss. The final column gives mean damage percentage before a stock loss.

Model	Stage	Neutral	Changed	Damage	Shield
75M	Pretrained	26.08	23.25	11.37	4.61
	Supervised	25.90	24.02	10.22	3.19
10M	Pretrained	22.83	23.12	12.39	4.00
	Supervised	25.67	23.73	10.46	3.15

Appendix

Table G.4: Controller activity and time in action states on the common cohorts. Values are percentages of frames, averaged equally across games. Damage and shield columns use the state definitions above.

Model	Stage	Wins	Taken	Conceded	Margin	Remaining
75M	Pretrained	69	3.066	3.145	$-0.079$	1.884
	Supervised	123	3.763	2.243	$+1.520$	2.171
10M	Pretrained	49	2.559	3.401	$-0.842$	1.857
	Supervised	106	3.546	2.480	$+1.066$	2.179

Appendix

Table G.5: Stock outcomes on the full initial suite. Each row contains 152 games. Taken, conceded, and margin are per-game means; remaining is mean Faynt stocks remaining conditional on winning.

Per frame	Reference	Optimized
Cache write	A loop over the batch in Python reads each row’s validity flag and write position to the host, in every layer	One indexed write each for keys and values per layer; positions stay on the device
Cache read	Each layer’s 256-frame window is copied into chronological order, reading each row’s length and next write position to the host	Attention runs over the ring in storage order through a validity mask, which is exact because rotary phases are applied when keys are written
Grouped-query attention [ 8 ]	PyTorch’s scaled dot-product attention on FP32 inputs expands keys and values to every query head	Batched matrix products per key-value head, without expansion
Checks	Eleven encoder range checks, three cache checks, a reset check, and the controller head’s label checks, each a device-to-host read	None; inputs are valid by construction
Input transfer	One host-to-device copy per input tensor	One copy per data type into static buffers (GPU)
Dispatch	Every kernel launched from Python	One CUDA-graph replay, with sampling and the stick head outside the graph (GPU)

Appendix

Table H.1: Per-frame operations of the two implementations. The optimized implementation computes the same formulas with the same parameters.

Model	Device	Pair	Max $\|\Delta\text{logit}\|$	Mean KL	Max KL	Top-1	Play
10M	CPU	Reference vs. sync-free	$6.1\times 10^{-5}$	$1.5\times 10^{-13}$	$2.6\times 10^{-11}$	0 of 4,800	none in 1,200
10M	GPU	Reference vs. sync-free	$6.1\times 10^{-5}$	$1.5\times 10^{-13}$	$1.6\times 10^{-11}$	0 of 4,800	none in 1,200
10M	GPU	Sync-free vs. optimized	0	0	0	0 of 4,800	none in 1,200
75M	CPU	Reference vs. sync-free	$6.4\times 10^{-4}$	$1.5\times 10^{-13}$	$1.2\times 10^{-10}$	0 of 4,800	none in 1,200
75M	GPU	Reference vs. sync-free	$4.0\times 10^{-4}$	$2.0\times 10^{-13}$	$1.2\times 10^{-10}$	0 of 4,800	none in 1,200
75M	GPU	Sync-free vs. optimized	0	0	0	0 of 4,800	none in 1,200

Appendix

Table H.2: Output parity between implementations on the first 1,200 frames of the recorded game, from both players’ perspectives, with the evaluated checkpoints (10M RL step 632, 75M RL step 980) on a Modal T4 container (GPU rows) and its host CPU (CPU rows). The first three columns of numbers compare the logits of both controller heads; KL divergences between action distributions are per head and in nats. Top-1 counts head outputs whose most likely category differs (1,200 frames

\times

2 players

\times

2 heads). Play reports the first frame on which actions sampled from the same seed differ. Optimized: the sync-free step replayed from a CUDA graph.

Figure H.1: Where the time goes. Real-time factor of the decision loop, paced at 60 Hz with Faynt playing one game, split into building the model input (dark segment) and the policy step with sampling, decoding, and the controller write (light segment), teal for the 10M and navy for the 75M. EPYC rows are the median host’s own run on the recorded game; T4 rows use synthetic frames, and their input build includes the copy to the GPU. The dashed line marks one frame.

	10M				75M
	Play		Watch		Play		Watch
GPU	Time	p99	Time	p99	Time	p99	Time	p99
NVIDIA T4	5.2	6.8	6.8	8.9	8.7	12.1	9.5	11.9
NVIDIA L4	3.2	4.7	4.0	5.5	5.4	6.4	6.2	7.5
NVIDIA A10G	3.1	4.6	3.7	5.1	4.6	5.9	5.6	6.9

Appendix

Table H.3: The whole loop of the optimized implementation on the recorded game, paced at 60 Hz: mean processing time per frame (Time) and 99th-percentile latency (p99), in ms. Play and watch are as defined in Appendix H.1 . Decisions later than one frame are 0.6% for the T4 75M and 0.3% for the A10G 10M when playing, and none in any other cell.

Figure H.2: Serving several games from one forward pass. 99th-percentile latency of the optimized implementation, one row per game, paced at 60 Hz on synthetic frames, on an NVIDIA T4, L4, and A10G GPU; teal is the 10M and navy the 75M. The dashed line marks one frame. Each point is a single run, and decisions later than one frame occur only for the T4 10M at 8 games (0.8%) and the T4 75M at 64 games (0.4%).

Explore similar work

Does Scaling Reinforcement Learning Really Require More Training?
Oct 1, 2026Bangji Yang, Jiajun Fan, Hongba Ma +2Offline Reinforcement Learning Sequential Scaling
University of Illinois at Urbana-Champaign · Tsinghua University
Mach-Mind-4-Flash Technical Report
Jul 10, 2026Foundation Model TeamAgentic Reinforcement Learning Mixture-Of-Agents
Foundation Model, Li Auto Inc.
Probing the Impact of Scale on Data-Efficient, Generalist Transformer World Models for Atari
May 9, 2026Jooyeon KimWorld Models Scaling Laws
Graduate School of Artificial Intelligence UNIST

Oct 1, 2026, cs.LGJ/K move · Enter open · S save

Does Scaling Reinforcement Learning Really Require More Training?

Bangji Yang, Jiajun Fan, Hongba Ma, Ruihan Guo+1

University of Illinois at Urbana-Champaign · Tsinghua University

Offline Reinforcement Learning Sequential Scaling Scaling Intermediate Checkpoints Mathematical Reasoning

Open paper AskPDF arXiv

Scaling reasoning typically spends more compute on reinforcement learning (RL) or on inference. We show that a completed RL training history can yield policies stronger than the checkpoints visited by its optimizer. We call this policy-space scaling: expanding the deployable policy set accessible from a fixed RL history, without extending training or increasing per-query inference computation. We instantiate it with SURGE (Scaling Up RL Gradient-free via Eigenspace fusion). SURGE combines two checkpoints from the same RL run: a high-accuracy anchor and a competitive donor that generates shorter responses. It expresses both checkpoints as changes from their shared initialization, then spectrally decomposes the anchor's update to retain its dominant component and incorporate the donor's complementary component. With a fixed target for how much of the anchor update to retain, SURGE determines the block size from the weights without testing candidate policies. We evaluate two 1.5B mathematical-reasoning histories, DeepSeek and Nemotron, and one 7B coding history, OLMo. SURGE improves benchmark-average accuracy over both input checkpoints while using fewer reasoning tokens than the anchor. It reaches 54.17% on DeepSeek AIME24 against a measured native maximum of 50.83%, and 83.7% on OLMo HumanEval+ against 82.8%. These gains exceed the observed training curves. Geometric controls support the importance of RL-update structure beyond weight displacement or token reduction alone. Each constructed model runs as a single policy. Our findings identify stored RL history as a reusable scaling resource: the capability available from a training run need not end at its best checkpoint.

Abstract

Explore similar work

Does Scaling Reinforcement Learning Really Require More Training?

Mach-Mind-4-Flash Technical Report

Probing the Impact of Scale on Data-Efficient, Generalist Transformer World Models for Atari