cs.LGNov 8, 2026

Primitives and Paths: Predicting What Post-Training Can Do from What the Base Model Already Has

Authors: Anqi Qu

Organizations: University of Oxford

Abstract

Does reinforcement learning teach language models new capabilities, or merely amplify ones acquired during pre-training? The literature offers evidence for both: pass@kk studies suggest RL reweights latent behaviours while some controlled composition studies demonstrate gains beyond simple amplification. We argue that this question has no answer independent of the capability state of the base model, and propose a two-dimensional state space for post-trainability. The first axis, effective full-solution support, measures whether successful trajectories can be sampled under a feasible exploration budget. The second axis, atomic primitive coverage, measures whether the constituent skills of a solution can be executed when the burden of composition is removed. Together, these dimensions motivate three regimes -- amplification, composition-limited, and acquisition-limited -- each predicting a distinct empirical signature and calling for a distinct intervention. We propose a prospective measurement protocol and investigate the framework using synthetic composition tasks across three small language models. We find that primitive mastery alone does not guarantee that RL can learn a composition. Checkpoints that execute every primitive reliably can fail to improve when complete solutions are never sampled during training, whereas modest composition supervision that establishes nonzero support enables substantial RL gains. The framework reframes mid-training as preparation for post-training: its role is to move models into states from which RL can operate effectively.

Figures & tables

Explore similar work

CardsList
  1. RL Post-Training Builds Compositional Reasoning Strategies

    Jul 8, 2026Azwar Abdulsalam, Nishil Patel, Andrew SaxeCompositional ReasoningRL for Language Model Reasoning

  2. Compositional Reasoning in Language Models under Reinforcement Learning Post-Training

    Sep 16, 2026Yu He, Yingxi Li, Yifei Wang +1Compositional ReasoningRL Post-Training

  3. How Does RL Post-training Induce Skill Composition? A Case Study on Countdown

    Dec 1, 2025Simon Park, Simran Kaur, Sanjeev AroraCompositional GeneralizationRL Post-Training