Language models exhibit strong reasoning capabilities, yet adapting them to structured domains remains challenging and can yield inconsistent outcomes. We identify representation compatibility, the extent to which a model effectively processes a representation for a structured task, as a key factor in adaptation. We study this in chess, which provides a controlled testbed with precise semantics, computable optimal actions, and multiple state representations, including a symbolic encoding (FEN) and a spatial format (ASCII). We find that models often process semantically equivalent inputs substantially differently, affecting both learning and generalization. Building on this observation, we propose representation-aligned auxiliary supervision, which uses environment-derived tasks expressed in compatible representations to improve adaptation to structured domains. Across models and representations, auxiliary supervision consistently improves optimal-move prediction relative to target-only training under identical target data. Tasks that expose environment dynamics provide larger and most consistent gains than surface-level or static supervision, while remaining competitive with substantially increasing the amount of target-task data. Moreover, ASCII-trained models transfer more effectively to FEN than FEN-trained models do to ASCII, even surpassing the FEN target-only baseline on FEN evaluation. The gains also extend beyond optimal-move prediction to open-ended, factually grounded commentary generation. Overall, our results show that auxiliary supervision in model-compatible representations can enable effective adaptation in structured domains.
Figures & tables
Figure 1: Representation compatibility. Given a chess position in FEN, Qwen3-8B reconstructs an ASCII board before reasoning about the task, illustrating how a model’s processing of a structured state can depend substantially on its representation. FEN is a compact linear encoding, while ASCII provides an explicit 8×8 spatial representation.
Figure 2: Representation compatibility in next-state prediction. Across model scales and families, ASCII inputs consistently yield higher board accuracy (left) and format compliance (right) than FEN prior to domain adaptation. The gap narrows with model size but remains substantial.
Training
Qwen3-8B
Qwen3-14B
Olmo3-7B-T
FEN
ASCII
FEN
ASCII
FEN
ASCII
Baseline
0.0
0.001
0.020
0.041
0.0
0.0
OM only (FEN)
0.181
0.146
0.199
0.161
0.172
0.152
OM + dynamics (FEN)
0.192
0.153
0.223
0.166
0.185
0.077
OM only (ASCII)
0.168
0.181
0.194
0.206
0.157
0.176
OM + dynamics (ASCII)
0.173
0.201
0.202
0.226
0.185
0.199
Table 1: Environment-derived supervision improves optimal-move prediction. OM trains on optimal-move prediction only; OM + dynamics additionally includes next-state and legal-move supervision. Training representations are shown in parentheses; FEN and ASCII columns report accuracy under each representation. Shading indicates in-distribution performance.
Figure 3: Environment-derived supervision improves beyond target-data scaling and matched-data controls. (a) Optimal-move accuracy as the amount of target-task training data increases, compared with 100k target-task examples plus 200k environment-derived dynamics examples. (b) Optimal-move accuracy under matched-data auxiliary supervision conditions. All results use Qwen3-14B, with models trained and evaluated in either FEN or ASCII.
Training Data
Avg. Len.
Fmt
EM
BA
FEN → FEN (short)
267
1.000
0.539
0.916
FEN → FEN (long)
622
1.000
0.613
0.930
ASCII → ASCII (short)
258
1.000
0.934
0.988
ASCII → ASCII (long)
681
0.999
0.713
0.942
FEN → ASCII → FEN (short)
276
1.000
0.824
0.971
FEN → ASCII → FEN (long)
664
1.000
0.912
0.981
Table 2: Representation on performance and reasoning efficiency. Each row corresponds to Qwen3-8B fine-tuned on a representation and reasoning length combination. Avg. Len. denotes average response length. Fmt : format compliance; EM : exact match; BA : board accuracy.
Figure 4: Contribution of environment-derived auxiliary tasks. Optimal-move accuracy for models trained with different combinations of tasks. The tasks improve over target-only training, while their combination (OM + dynamics) yields the largest gains across the two training representations.
Initialization
FEN
ASCII
Claim Acc. ↑
Error Rate ↓
Claim Acc. ↑
Error Rate ↓
Baseline (Qwen3-8B)
20.1
61.5
17.1
56.6
No auxiliary supervision
33.7
39.0
44.0
32.2
OM + dynamics (FEN)
50.7 (+17.0)
23.2 (-15.8)
41.7 (-2.3)
35.7 (+3.5)
OM + dynamics (ASCII)
44.0 (+10.3)
36.3 (-2.7)
51.0 (+7.0)
27.9 (-4.3)
Table 3: Auxiliary supervision on commentary generation. Claim accuracy and error rate for Qwen3-8B across auxiliary warm-up conditions. Columns indicate the representation used for commentary training and evaluation; rows indicate the preceding training or auxiliary warm-up condition. For the OM + dynamics rows, the auxiliary model is trained in the representation shown in parentheses. Shading indicates matched auxiliary and commentary representations.
Appendix figures & tables7 assets
Supplementary material from the paper’s appendix.
Appendix
Hyperparameter
Value
Training
Epochs
3
Batch size
256
Gradient clipping
1.0
Precision
bf16
Optimization
Learning rate
1e-5
LR scheduler
cosine
Appendix
Table 4: Supervised fine-tuning hyperparameters.
Training
Mean WPL ↓
Median WPL ↓
FEN
ASCII
FEN
ASCII
OM only (FEN)
21.8 ( ± 27.2)
24.8 ( ± 27.2)
7.8
12.9
OM + dynamics (FEN)
21.0 ( ± 26.7)
23.8 ( ± 26.8)
7.1
11.1
OM only (ASCII)
22.6 ( ± 26.2)
22.0 ( ± 27.2)
9.4
8.0
OM + dynamics (ASCII)
22.5 ( ± 26.5)
20.9 ( ± 27.0)
10.0
6.4
Appendix
Table 5: Comparison of move quality in win probability loss (Qwen3-8B). We report mean and median win probability loss (WPL), as a softer measure of move quality based on changes in engine-estimated win probability. Shading indicates in-distribution performance.
Training
FEN → FEN
ASCII → ASCII
FEN → ASCII → FEN
Fmt
EM
BA
Fmt
EM
BA
Fmt
EM
BA
Best Baseline
1.000
0.613
0.930
1.000
0.934
0.988
1.000
0.912
0.981
FEN + ASCII
1.000
0.741
0.952
1.000
0.908
0.983
1.000
0.016
0.647
FEN + ASCII + F2A2F
1.000
0.805
0.965
1.000
0.937
0.989
1.000
0.917
0.982
FEN + ASCII + F2A2F + Conv
1.000
0.860
0.974
1.000
0.953
0.991
1.000
0.939
0.987
Appendix
Table 6: Representation mixture and learning. Performance on next-state prediction under different data compositions. Fmt : format compliance; EM : exact match accuracy; BA : board accuracy.
Training Data
Source
Fmt
EM
BA
FEN → ASCII → FEN
–
1.000
0.912
0.981
S1 + S3 + S2
Same
0.975
0.003
0.644
S1 + S3 + S2
Diff
0.972
0.004
0.633
Appendix
Table 7: Limits of sub-skill composition. Left: models trained on constituent reasoning skills fail to compose them into the FEN → ASCII → FEN pipeline. Right: adding FEN-to-ASCII conversion to FEN-based reasoning does not induce the use of ASCII as an intermediate representation. Source indicates whether the constituent tasks use the same set of positions (Same) or disjoint sets (Diff).
Category
Tool
Purpose
Position Query
get_piece_at (fen, sq)
Return the piece on a given square
get_legal_moves (fen)
List all legal moves in the position
is_check (fen)
Determine whether the side to move is in check
get_material (fen)
Compute material balance / piece counts
get_squares (fen, piece, color)
Find locations of a piece type
Attack / Defense
get_attacks (fen, sq)
List squares attacked by a piece on a square
Appendix
Table 8: Tool suite. We implement a set of tools that enables the LLM judge to verify the factual correctness of diverse chess claims through chained tool calls.
Figure 13
Figure 5: Error analysis for commentary generation. Distribution of error types across six models. Board comprehension errors reflect failures in static board parsing; Legality errors reflect failures in multi-step board state tracking; Tactical errors are those regarding downstream consequences of piece localization; Quality errors reflect failures in position understanding.
We study how reasoning evolves in a language model -- from supervised fine-tuning (SFT) to reinforcement learning (RL) -- by analyzing how a set of theoretically-inspired datasets influences language model performance in chess. We find that fine-tuning a model to directly predict the best move leads to effective RL and the strongest downstream performance -- however, the RL stage elicits unfaithful reasoning (reasoning inconsistent with the chosen move). Alternatively, training on multi-move trajectories yields comparable downstream performance with faithful reasoning and more stable RL. We analyze multiple qualitative and quantitative measures and highlight how these evolve from SFT through RL; we find several SFT-checkpoint metrics -- spanning evaluation performance, hallucination rates, and reasoning quality -- to be predictive of post-RL model performance. Finally, we ground our results with an experiment measuring chess information density in our custom datasets. We release models as well as training data, evaluations, and code that allowed us to surpass leading open-source reasoning models in chess with a 7B-parameter model. Code, models, and data are available at https://github.com/lucasdino/lang-chess.
Lucas Dionisopoulos, Nicklas Majamaki, Prithviraj Ammanabrolu
Department of Computer Science and Engineering, University of California San Diego, San Diego, United States.
Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.
Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov +6
Reinforcement learning with verifiable rewards has been used to elicit impressive performance from language models in many domains. But, broadly beneficial deployments of AI may require us to train models with strong capabilities in "fuzzy", hard-to-supervise domains. In this paper, we develop methods to align language models in fuzzy domains where human experts are still able to provide high-quality supervision signal, but only for a small number of model outputs, using online natural language feedback. Specifically, we train models by iteratively optimizing against proxy reward signals, stopping at the point of over-optimization, collecting fresh expert supervision, and updating the proxy reward. We construct proxy reward models from language models using in-context learning (ICL) and fine-tuning. We test our methods by eliciting creative writing and alignment research capabilities in Qwen3-8B and Haiku 4.5 respectively. For Qwen3-8B, ICL methods recover up to 35% of performance with 50x fewer expert samples, while fine-tuning methods recover 80% with up to 20x fewer samples and 100% with 3x fewer samples. For Haiku 4.5, ICL methods recover up to 35% of performance with 30x fewer samples, and fine-tuning methods recover 100% with 10x fewer samples. Our results suggest that online natural language feedback can substantially improve the data efficiency of expert supervision.
Christine Ye, Joe Benton
Stanford University · Anthropic Fellows Program for Safety Research · Anthropic