cs.LGNov 10, 2026

Trait Hitchhiking in LLM Post-Training

Authors: Adil Soubki, Elizabeth S. Z. Tan, Miles Cranmer

Organizations: Department of Applied Mathematics and Theoretical Physics, University of Cambridge

Abstract

Reinforcement learning (RL) is now a standard stage of language model post-training, and the compute devoted to it is growing rapidly. While it has been shown to improve capabilities, particularly for tasks with verifiable rewards, it can also change behaviours and internal properties that the reward does not explicitly target. We present a formalism to describe the evolution of "traits" during RL post-training and, in doing so, recover the Price equation from evolutionary biology. We use this to develop a mathematical model for what we call "trait hitchhiking": the amplification or suppression of non-targeted traits as training redistributes probability across completions. We evaluate this Price update expression on language models trained with GRPO and show that the Price estimates closely track the sampled trait dynamics. These results identify trait hitchhiking as a mechanism by which RL post-training can silently amplify undesirable traits.

Figures & tables

Appendix figures & tables14 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Demystifying Reinforcement Learning Post-Training of Language Models

    Aug 24, 2026Donovan Clay, Saket Gollapudi, Sankar Harilal +4RL for Language ModelsRL Post-Training

  2. On Language Drift during RLVR Post-Training

    Oct 1, 2026Michael Sullivan, Alexander KollerRL for Language Model ReasoningReinforcement Learning with Verifiable Rewards