In this work, we reveal that Large Language Models (LLMs) possess intrinsic behavioral plasticity-akin to chameleons adapting their coloration to environmental cues-that can be exposed through token-conditional generation and stabilized via reinforcement learning. Specifically, by conditioning generation on carefully selected token prefixes sampled from responses exhibiting desired behaviors, LLMs seamlessly adapt their behavioral modes at inference time (e.g., switching from step-by-step reasoning to direct answering) without retraining. Based on this insight, we propose Token-Conditioned Reinforcement Learning (ToCoRL), a principled framework that leverages RL to internalize this chameleon-like plasticity, transforming transient inference-time adaptations into stable and learnable behavioral patterns. ToCoRL guides exploration with token-conditional generation and keep enhancing exploitation, enabling emergence of appropriate behaviors. Extensive experiments show that ToCoRL enables precise behavioral control without capability degradation. Notably, we show that large reasoning models, while performing strongly on complex mathematics, can be effectively adapted to excel at factual question answering, which was a capability previously hindered by their step-by-step reasoning patterns.
Figures & tables
Figure 1: Like chameleons changing their coloration in response to external stimuli (top), language models can adapt their behavior according to the token prefix (bottom). ⊕ stands for concatenation.
Figure 2: Top : Conditioning response generation on a direct-answer prefix enables LRM to switch from step-by-step reasoning to direct knowledge retrieval, revealing a new perspective on language model behavioral plasticity. Middle : ToCoRL leverages token-conditional generation to guide exploration during RL training, inducing the emergence of new factual answering behaviors. Bottom : After an initial direct answer, the ToCoRL-trained model adopts an emergent recalibrative reasoning behavior for factual problems (see Section 5.2 ), while its complex math problem–solving behavior remains unchanged. We provide concrete examples for further illustration in Fig. 4 .
Accuracy
Avg. length
Med. length
The Normal Gen
18.9%
1255
1128
ToCo Gen
20.7%
891
477
Table 1: The following table shows the accuracy (avg@8), the average response length (Avg. length) and medium response length (Med. length) in token number, when tested on SimpleQA benchmark. Note that we use ‘Normal Gen’ and ‘ToCo Gen’ to denote normal generation and token-conditional generation respectively.
Qwen3-30A3B
SimpleQA
Omnisci
AIME’24
AIME’25
Thinking
18.9
16.1
84.8
80.5
Instruct
22.1
10.5
75.2
61.3
w. ToCoRL
28.3
18.6
89.0
81.5
w. GRPO
23.6
16.7
89.5
81.4
w. Adapt-Think
23.9
16.0
86.9
78.9
Table 2: Evaluation results for experiments based on Qwen3-30B-A3B-2507-Thinking. We use ‘Thinking’ and ‘Instruct’ to denote the corresponding two open-source models, and ‘w.’ to denote the training method used for factual problem queries. ‘Adapt-Think’, ‘I-T GRPO’, and ‘Omnisci’ are abbreviations for Adaptive-Thinking, Instruct-Tuning GRPO, and AA-Omniscience, respectively, and we consistently use these notations in subsequent tables.
Figure 3: For clearer visualization, we plot the curves of these metrics over the first training epoch. Compared to ToCoRL, GRPO extends reasoning length while keeping the original reasoning style. Since unnecessary associations and unverified information persist, longer reasoning yields minor improvements, and response lengths eventually return to their initial levels. Adaptive-Thinking penalizes all reasoning, so evaluation scores rise quickly at first as ineffective context is removed. However, without additional reasoning capacity, the LRM’s potential is underutilized, limiting further gains. Instruct-tuning RL initially shortens reasoning, but the prompt instruction gradually loses control over the behavior.
Qwen3-30A3B
SimpleQA
Omnisci
AIME’24
AIME’25
Base
24.2
12.0
15.1
9.4
SFT
19.6
14.0
56.5
38.7
w. ToCoRL
28.1
15.8
72.1
61.7
w. GRPO
22.6
14.3
72.0
61.9
w. Adapt-Think
24.8
13.8
47.9
30.4
Table 3: Results for experiments based on Qwen3-30B-A3B-SFT, obtained by enabling the base model (denoted as ‘Base’) with a unified reasoning behavior. We refer to it as ‘SFT’ for brevity.
(λ,k)
SimpleQA
Omnisci
AIME’24
AIME’25
(1,6)
28.3
18.6
89.0
81.5
(1,7)
27.5
18.1
88.2
80.6
(1,8)
27.6
18.3
88.5
80.9
(0.5,6)
28.1
18.3
88.3
80.5
(2,6)
27.2
17.8
88.0
80.0
Table 4: Results for ablation studies on hyperparameters ( λ and k ). We skip testing k<6 since the required < think > \n prefix (occupying 2 tokens) makes the prefix too short to form a direct attempt at answering the problem.
πInst
SimpleQA
Omnisci
AIME’24
AIME’25
Qwen3-235A22B
28.3
18.6
89.0
81.5
Qwen3-30A3B
28.0
18.6
89.1
81.3
Qwen2.5-14B
27.8
18.3
89.0
81.1
Table 5: Results for ablation studies on different prefix providers ( πInst ). For brevity, the ‘Instruct’ suffix is omitted for all models. Note that Qwen3 models in the table refer to the 2507 version.
Response Provider
SimpleQA
Omnisci
AIME’24
AIME’25
30A3B-w. ToCoRL
29.1
16.0
74.5
62.0
235A22B-Instruct *
28.3
15.5
68.6
55.2
30A3B-Instruct *
25.3
14.5
69.3
59.2
Table 6: We use ’Response Provider’ to denote the source of the factual responses used for new SFT datasets as described above. All providers are from the Qwen3 series; * indicates manually constructed responses, otherwise are generated normally.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
Qwen3-235B-A22B-2507
Qwen3-30B-A3B-2507
Thinking
Instruct
Thinking
Instruct
AIME’25
92.3
70.3
80.5
61.3
SimpleQA
47.0
54.3
19.2
22.1
Appendix
Table 7: Comparison of accuracy on complex math benchmark (AIME’25) and factual problem benchmark (SimpleQA). In the same model series, the Thinking version, which conduct step-by-step reasoning before arriving at final answer, consistently perform better on complex math benchmark and worse on factual problem benchmark.
Figure 4: Detailed demonstration of the behavior change brought by token-conditional generation and ToCoRL. Concrete queries and responses are provided.
Figure 5: Since most experiments are conducted with large reasoning models, we only illustrate the procedure in this setting. Moreover, we only show the case where the length of token prefix k=3 as an example. That is, besides the < think > token, two initial tokens from direct answer (in orange) are used for constructing token prefix.
Inference-time language model alignment steers or selects outputs during generation to better meet safety, preference, and task goals without retraining.
Strong reasoning depends not only on model knowledge but also on how effectively cognitive behaviors are deployed during generation. Existing methods often rely on explicit behavior-level control, making them insufficiently adaptive when failures and required corrections vary across reasoning states, tasks, and models. To this end, we propose Latent Reward Steering (LRS), an adaptive inference-time framework that promotes cognitive behaviors by optimizing the sparse-autoencoder (SAE) latent states that implicitly carry them. Rather than relying on predefined cognitive behaviors or steering directions derived from them, LRS trains a latent reward model on reasoning traces by final answer correctness to estimate the quality of intermediate latent states. During inference, reward gradients provide state-specific correction directions for fragile latent states, while a reward and confidence gate restricts intervention to states the reward signal flags as fragile. Experiments on multiple reasoning LLM backbones and benchmarks show that \ours consistently improves performance over various baselines, and post-hoc analyses further indicate that \ours implicitly promotes good cognitive behaviors that fix the original reasoning errors. Code is available at: https://github.com/jiakanglee/Latent-Reward-Steering.
Jiakang Li, Guanyu Zhu, Can Jin +8
Rutgers University · South China Agricultural University · Columbia University +4
Large Language Models (LLMs) often produce plausible but poorly-calibrated answers, limiting their reliability on reasoning-intensive tasks. Recent research suggests that Chain-of-Thought (CoT) reasoning paths are inherent in pre-trained LLMs and can be elicited by simply altering the decoding process, where the presence of a CoT path correlates with higher answer confidence. Building on these insights, we present Reinforcement Learning from Self-Feedback (RLSF), a post-training stage that utilises the model's intrinsic confidence as a self-generated reward. By generating multiple CoT decoding beams from a frozen LLM, we compute the confidence of each final answer span and rank the resulting traces accordingly to create synthetic preferences. These preferences are subsequently utilised to fine-tune the policy through standard preference optimisation, requiring no human labels, gold answers, or externally curated rewards. RLSF simultaneously (i) refines the model's probability estimates--restoring well-behaved calibration--and (ii) strengthens step-by-step reasoning, yielding improved performance on arithmetic reasoning and multiple-choice question answering. By converting a model's own uncertainty into structured self-feedback, RLSF affirms reinforcement learning on intrinsic model behaviour as a principled and data-efficient component of the LLM post-training pipeline. Our results demonstrate that leveraging these inherent reasoning capabilities provides a robust path for enhancing model reliability without manual prompt engineering or external supervision.
Carel van Niekerk, Renato Vukovic, Benjamin Ruppik +3
Steering vectors offer a training-free mechanism for controlling reasoning behaviors in large language models, but constructing effective vectors requires identifying genuine behavioral signals in the model's hidden states. For behaviors that can be toggled via prompts, this is straightforward. However, many reasoning behaviors -- such as self-reflection -- emerge spontaneously and resist prompt-level control. Current methods detect these behaviors through keyword matching in chain-of-thought traces, implicitly assuming that every detected boundary encodes a genuine behavioral signal. We show that this assumption is overwhelmingly wrong: across 541 keyword-detected boundaries, 93.3% are behaviorally unstable, failing to reproduce the detected behavior under re-generation from the same prefix. We develop a probabilistic model that formalizes intrinsic reasoning behaviors as stochastic events with context-dependent trigger probabilities, and show that unstable boundaries dilute the steering signal. Guided by this analysis, we propose stability filtering, which retains only boundaries where the model consistently reproduces the target behavior. Combined with a content-subspace projection that removes residual question-specific noise, our method achieves 0.784 accuracy on MATH-500 (+5.0 over the strongest baseline). The resulting steering vectors transfer across models in the same architecture family without re-extraction, improving Nemotron-Research-Reasoning-1.5B (+5.0) and DeepScaleR-1.5B-Preview (+6.0). Code is available at https://github.com/zhmzm/stability-steering.