Beyond Surface Alignment: Grounding the Dynamics of Situational Understanding and Generative Control in LLMs

Chenghao Yang

cs.CLAug 30, 2026

Beyond Surface Alignment: Grounding the Dynamics of Situational Understanding and Generative Control in LLMs

Authors: Chenghao Yang

Organizations: DEPARTMENT OF COMPUTER SCIENCE THE UNIVERSITY OF CHICAGO CHICAGO, ILLINOIS

Abstract

The current alignment tuning paradigm for Large Language Models (LLMs) prioritizes surface-level behaviors -- fluency, safety, and tonal consistency. While effective for casual chat, this thesis argues that such surface alignment masks a lack of grounding, creating models that are stylistically confident but situationally brittle. We propose a framework of Grounded Alignment, analyzing how models process context (Input) and structure generation (Output), then aligning these grounded behaviors to human needs. First, we evaluate failures in Situational Grounding. SitTest shows that despite large context windows, state-of-the-art models struggle to maintain a consistent "mental model" of a changing environment. ReCode further shows that models rely on surface heuristics rather than deep syntactic dependencies: they "read" extensive histories without truly "understanding" the evolving situation. Second, we evaluate Generative Grounding. We introduce the Branching Factor (BF) to map LLM generation, finding that standard alignment tuning constricts this landscape into premature stylistic collapse. Hindsight further shows that models often fail to understand their own generations. Finally, we propose Dynamic Control for grounded interaction. AI Realtor demonstrates context engineering to compensate for poor situational grounding. Base-Aligned Model Collaboration decouples exploration from stylistic constraints. We also present Annealed Sampling for verifiable reinforcement learning and apply these ideas to Addiction Support, where model-generated rationalization offers a communication interface for high-stakes domains. Collectively, this work moves beyond surface alignment toward agents anchored in both context and generation.

Figures & tables

Figure 1

Normal Instruction	Step-EM / State-EM
Normal Instruction	2-shot	3-shot	5-shot
NL Functor + NL Argument	$22\%/92\%$	$34\%/93\%$	$36\%/95\%$
SL Functor + SL Argument	$19\%/92\%$	$22\%/93\%$	$52\%/96\%$
Counter-Intuitive Instruction (On NL)
NL Functor + NL Argument	$10\%/77\%$	$6\%/75\%$	$0\%/84\%$
SL Functor + SL Argument	$13\%/89\%$	$20\%/90\%$	$12\%/90\%$

Table 1: Experiment results on ChatGPT Robustness check for state tracking in 10-box environment. Metrics here are presented in the format of “Step-EM / State-EM”. We use 50 samples for each experiment setting.

Panel of Figure 2 — Figure 2: Results for Intermediate State Probing. Purple vertical lines indicate where in-context demonstrations (for steps prior to the test step) end.

Normal Initialization	Step-EM / State-EM
Normal Initialization	2-shot	5-shot
NL Functor + NL Argument	$22\%$ / $92\%$	$36\%$ / $95\%$
SL Functor + SL Argument	$19\%$ / $92\%$	$52\%$ / $96\%$
Compressed Initialization
NL Functor + NL Argument	$46\%$ / $95\%$	$60\%$ / $97\%$
SL Functor + SL Argument	$48\%$ / $95\%$	$54\%$ / $97\%$

Table 2: Compressed Initialization Testing experiment results for 10-box environment on ChatGPT. Metrics here are shown in the format of "Step-EM/State-EM". We use 50 samples with various number of steps for experiments.

Figure 3: Diagram for fine-grained error analysis

Panel of Figure 4 — Figure 4: Fine-Grained Error Analysis for Logical Inconsistency

Panel of Figure 1 — Figure 1: InCoder-6B predicts correctly on nominal prompt (left) but fails on the prompt where docstrings are paraphrasing with BackTranslation (right). We underline the perturbed positions and wrong model completions.

Perturbations	MBPP Docstrings
Nominal	Write a function to find all words which are at least 4 characters long in a string by using regex.
BackTranslation	Write a function to find all words in a string at least 4 characters long using regex.
ButterFingers	Wri h e a function to find all words which are a r leas v 4 characters long in a string by using regex.
ChangeCharCase	Wri T e a f U ncti O n to find All wo R ds whic H are at le A st 4 C ha R acter S L on G in a string by u SI ng re G ex.
EnglishInflectionalVariation	Writes a functions to found all word which was at least 4 character long in a string by use regex.
SwapCharacters	rW ite a function to find all words which are at el ast 4 ch ra acters long in a string by su ing regex.

Table 1: Illustrations for docstring perturbations on a MBPP sample.

Panel of Figure 3 — Figure 3: An original prompt with partial code (a) and its perturbed versions (b, c).

Perturbations on Function Names	MBPP
Nominal	find_char_long
CamelCase	findCharLong
ButterFingers	fin f _char_long
SwapCharacters	find_c ah r_long
ChangeCharCase	fin D _cha R _long
InflectionalVariation	found_chars_long

Table 2: Illustrations for function name perturbations on a MBPP sample.

HumanEval	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
HumanEval	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Docstring	Nominal $\uparrow$	0.232	0.140	0.262	0.195	0.305	0.195	0.104	0.152	0.122
	RP 5 @1 $\uparrow$	0.122	0.049	0.104	0.073	0.128	0.098	0.024	0.067	0.037
	RD 5 @1(%) $\downarrow$	47.37	65.28	60.47	62.50	58.00	50.00	76.47	56.00	70.00
	RR 5 @1(%) $\downarrow$	20.73	14.63	27.44	18.90	35.37	18.90	14.63	15.85	10.98
Function	Nominal $\uparrow$	0.232	0.140	0.262	0.195	0.305	0.195	0.104	0.152	0.122

Table 3: ReCode benchmark robustness evaluation on popular code generation models for HumanEval.

MBPP	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
MBPP	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Docstring	Nominal $\uparrow$	0.317	0.191	0.361	0.221	0.407	0.241	0.128	0.199	0.133
	RP 5 @1 $\uparrow$	0.137	0.050	0.147	0.042	0.163	0.045	0.011	0.031	0.013
	RD 5 @1(%) $\downarrow$	56.96	73.66	59.38	80.93	59.85	81.28	91.20	84.54	90.00
	RR 5 @1(%) $\downarrow$	36.86	34.39	41.89	36.76	46.72	44.66	25.57	35.32	30.08
Function	Nominal $\uparrow$	0.317	0.191	0.361	0.221	0.407	0.241	0.128	0.199	0.133

Table 4: ReCode benchmark robustness evaluation on popular code generation models for MBPP.

Category	Metric	HumanEval	MBPP
Docstring	RP 5 @1 $\uparrow$	0.078	0.071
	RD 5 @1 $(\%)\downarrow$	60.67	75.31
	RR 5 @1 $(\%)\downarrow$	19.72	36.92
Function	RP 5 @1 $\uparrow$	0.113	0.142
	RD 5 @1 $(\%)\downarrow$	41.61	46.59
	RR 5 @1 $(\%)\downarrow$	12.06	21.01

Table 5: Average robustness numbers across all models. MBPP is more challenging for robustness evaluation.

Figure 4: Robust Drop s @1 and Robust Relative s @1 under different

s

. Larger

s

indicates stronger perturbations evaluated and larger performance drops.

Figure 5: Robust Drop 1 @k and Robust Relative 1 @k under different

k

using sampling

n=100

. Robust Drop remains stable while Robust Relative increases with k.

	HumanEval	MBPP
Naturalness (Nominal) $\uparrow$	0.92	0.92
Naturalness (Perturbed) $\uparrow$	0.75	0.80
Semantics Similarity $\uparrow$	0.92	0.92

Table 6: Human evaluation for practical naturalness and semantic similarity by 5 annotators. Either 0, 0.5, or 1 is assigned to each data point indicating quality level.

	HumanEval		MBPP
	Syntax	Format	Syntax	Format
CodeBLEU (syntax) $\uparrow$	0.95	0.98	0.93	0.96
CodeBLEU (dataflow) $\uparrow$	0.94	1.00	0.92	1.00

Table 7: Average CodeBLEU syntax and format scores between non-perturbed codes and perturbed ones with our syntax and format transformations.

Panel of Figure 5 — Figure 5: Substituting external random tokens for the model’s own prefix separates autoregressive self-narrowing from alignment. Self-Conditioned is ordinary generation; External Random replaces the prefix with i.i.d. random tokens up to a cut point (dashed) and continues, with the $x$ -axis aligned to the true generation position. External content surges BF, after which continued autoregression drives it down again, in both Base (top) and Aligned (bottom) models. Full setup, cut points, and the Qwen3-4B-Instruct anomaly are discussed in Appendix 15.L .

Figure 6: Unexpected environment feedback raises BF. Change in next-step BF relative to a matched control (progress-as-expected), for adversarial and random-noise feedback in a one-turn agentic setting; the effect shrinks for more heavily aligned models. Setup and per-model results in Appendix 15.L .

Panel of Figure 7 — Figure 7: Attributing BF Reduction. (a) Heatmap of average BF ratio (Base/Aligned) across tasks and models. The numbers indicate the average ratio over all constraint levels. Note that for OLMo-2, we follow the convention to treat the DPO version as the aligned model ( olmo20242 ) . Since both base and aligned models in each column share the same tokenizer, the within-family tokenizer scaling cancels in the ratio; cross-family ratio magnitudes should be interpreted as patterns rather than precise comparisons. (b)-(e) Pareto Analysis of BF across various IFs. $AT$ indicates whether the model is aligned. $C$ denotes the prompt complexity. $S$ refers to model size, and $G$ refers to model generation.

Models	Default ( $T$ =0.6, $p$ =0.9)	$T$ =0.6, $p$ =1.0	$T$ =1.0, $p$ =0.9	Min ( $T$ =1.0, $p$ =1.0)	$\frac{\text{Default}-\text{Min}}{\text{Default}}\%$
Llama-3-70B-Instruct	78.50 ( $\pm$ 2.09)	77.60 ( $\pm$ 2.23)	77.50 ( $\pm$ 2.60)	75.90 ( $\pm$ 2.85)	3.31
Llama-3-70B	78.00 ( $\pm$ 3.52)	74.00 ( $\pm$ 3.80)	72.00 ( $\pm$ 4.38)	63.50 ( $\pm$ 5.02)	18.59
DeepSeek-R1-Distill-Llama-8B	66.30 ( $\pm$ 3.51)	65.70 ( $\pm$ 3.84)	62.70 ( $\pm$ 4.14)	59.70 ( $\pm$ 4.65)	9.95
Llama-3.1-8B-Instruct	63.00 ( $\pm$ 4.01)	61.50 ( $\pm$ 4.37)	57.50 ( $\pm$ 4.92)	50.50 ( $\pm$ 5.34)	19.84
Llama-3.1-8B	54.00 ( $\pm$ 4.61)	53.50 ( $\pm$ 4.92)	47.00 ( $\pm$ 5.21)	37.00 ( $\pm$ 5.48)	31.48

Table 1: Experiment Results across decoding methods on STEM subset of MMLU. We follow the common practice of using 5-shot CoT prompting.

\frac{\text{Default}-\text{Min}}{\text{Default}}\%

indicates the maximum relative performance drop when deviating from the default decoding configuration.

Model	Maj@1 Std	Maj@3 Std	Maj@8 Std	Maj@16 Std	BF
DeepSeek-R1-Distill-Llama-70B	14.34	8.29	4.99	3.21	1.23
Llama-3-70B-Instruct	16.37	11.40	7.50	5.12	1.28
Llama-3-70B	27.78	19.53	13.22	9.23	1.31
DeepSeek-R1-Distill-Llama-8B	27.10	20.91	13.93	9.14	1.23
Llama-3.1-8B-Instruct	31.54	24.64	17.30	12.90	1.31
Llama-3.1-8B	36.41	29.78	20.43	14.05	1.35

Table 2: Majority Voting@K standard deviation on MMLU-STEM with

200

samples. We compute the standard deviation over

100

bootstrapping trials, each using

64

samples per instance. We set

T=0.6,p=0.9

to match standard benchmarking settings, differing from

T=1.0,p=0.9

setup in Section 4 . Lower temperature concentrates probability mass on fewer tokens, reducing BF and complicating direct comparisons. However, bootstrapping (100 runs) reveals minimal variability (

\approx 0.01

), confirming that the BF differences reported here remain significant. Consequently, BF remains a strong predictor of standard deviation.

Figure 8: Resampling from different output positions to assess the effect of interrupting BF reduction . We resample new continuations at the 25th and 200th output token of DeepSeek-Distilled Llama-8B MMLU outputs. Results show substantial performance drops at both positions.

Panel of Figure 9 — Figure 9: Nudging Experiments over Just-Eval-Instruct.

Figure 10: Stage-wise Contribution to BF Reduction. We analyze OLMo-2 models on the Creative StoryGen task.

Figure 11: BF across post-training algorithms. Using checkpoints from an RLHFlow/OpenRLHF pipeline on Creative StoryGen, we find that SFT drives the largest BF drop, DPO and PPO yield similar further reductions, and Iterative DPO partially preserves distributional breadth.

Figure 1: Example of Self-Reflection Prompting

Metric	Standard Prompting	Exploration-Only	Self-Reflection
TruthfulQA
Rouge-1	$57.5\pm 1.1$	$57.2$	60.8
BLEURT	$66.8\pm 1.9$	$60.7$	72.8
HotpotQA
Accuracy*	$80.3\pm 0.5$	80.8	$76.2$
EM	$\mathbf{50.5\pm 0.4}$	$47.3$	$37.0$

Table 1: Self-reflection SR 2 V experiment results on QA datasets. Bold-facing indicates the best-performing method under each metric. *Evaluated manually.

Figure 2: Performance Decomposition on Question Difficulty and Response Accuracy.

Figure 3: Performance Decomposition on Question Difficulty and Response Accuracy (Artificial Responses). Dotted lines show “turning points” at which reflection loses effectiveness, for Easy/Medium/Hard questions.

Figure 4: Majority Voting Analysis

Figure 5: Proposed guide for using Self-Reflection.

Seed=1
Seed=2

Figure 1: Heatmaps of explanation scores of an example from Yelp-Polarity based on two runs of KernelSHAP (KS) using different random seeds. KS is run on a fine-tuned BERT model using

200

samples per instance (approx.

3.47

s per instance on average using a single A100 GPU, more than

150

times slower than one forward inference of the BERT model). The darker each token is, the higher its explanation score. Clearly, interpretation results are significantly different when using different seeds.

Figure 2: Illustration of our proposed Amortized Model. Black-outlined circles represent original inputs without Shapley Values, while circles with colored outlines or colored fills denote inputs with Shapley Values.

Setting	Spearman	Top-5 Inter.	Top-10 Inter.	MSE	Running Time
SVS-25	$0.84(\pm 0.00)$	$3.41(\pm 0.00)$	$7.02(\pm 0.00)$	$0.01(\pm 0.00)$	$183.72$ s/it
KS-25	$0.04(\pm 0.00)$	$0.43(\pm 0.01)$	$1.45(\pm 0.01)$	$0.00(\pm 0.00)$	$1.92$ s/it
KS-200	$0.16(\pm 0.00)$	$1.09(\pm 0.01)$	$2.47(\pm 0.00)$	$0.82(\pm 0.29)$	$3.47$ s/it
KS-2000	$0.37(\pm 0.00)$	$2.45(\pm 0.01)$	$4.38(\pm 0.05)$	$0.03(\pm 0.00)$	$33.40$ s/it
KS-8000	$0.63(\pm 0.00)$	$3.73(\pm 0.02)$	$6.93(\pm 0.01)$	$0.01(\pm 0.00)$	$123.29$ s/it

Table 1: Ranking stability experiments on the Yelp-Polarity dataset. Each local explanation setting is evaluated across 5 runs with different random seeds. “Top-K Inter.” denotes top-K intersection. All values in this table are absolute values. Here we can see a clear trade-off between stability and computation cost.

Setting	Spearman	Top-5 Inter.	Top-10 Inter.	MSE	Running Time
SVS-25	$0.75(\pm 0.00)$	$3.54(\pm 0.02)$	$7.46(\pm 0.02)$	$0.02(\pm 0.00)$	$128.07$ s/it
KS-25	$0.06(\pm 0.00)$	$0.97(\pm 0.01)$	$3.41(\pm 0.03)$	$0.01(\pm 0.00)$	$0.33$ s/it
KS-200	$0.24(\pm 0.00)$	$1.79(\pm 0.01)$	$4.37(\pm 0.03)$	$0.07(\pm 0.00)$	$2.04$ s/it
KS-2000	$0.52(\pm 0.00)$	$3.19(\pm 0.00)$	$6.09(\pm 0.00)$	$0.03(\pm 0.00)$	$20.39$ s/it
KS-8000	$0.76(\pm 0.00)$	$4.08(\pm 0.02)$	$7.74(\pm 0.02)$	$0.01(\pm 0.00)$	$89.48$ s/it

Table 2: Ranking stability experiments on the MNLI dataset.

Method	MNLI		Yelp-Polarity
Method	Spearman	MSE	Spearman	MSE
SVS-25	0.75	1.90e-2	0.84	6.64e-3
KS-25	0.17	9.95e-2	0.12	4.34e-2
KS-200	0.35	7.73e-2	0.24	5.77e-2
KS-2000	0.60	2.54e-2	0.51	1.86e-2
KS-8000	0.74	1.25e-2	0.70	6.25e-3

Table 3: Spearman’s correlation and MSE of variants of SV methods against SVS-25, a proxy of exact SV on MNLI and Yelp-Polarity. As we show in Section 4 , MSE correlates poorly with ranking stability and Spearman’s correlation should be considered as the main metric . We only list MSE for reference. Bold-faced numbers are the best in each column. Results are averaged over 5 runs. Our amortized model achieves better approximation compared to KS-200 and FastSHAP baseline, but not as good as much more time-consuming methods KS-2000/8000. SVS-25 is listed as an upper bound.

Training Data Proportion	Spearman (MNLI)	Spearman (Yelp-Polarity)
10%	0.45	0.40
30%	0.57	0.65
50%	0.65	0.71
70%	0.65	0.72
100%	0.77	0.76

Table 4: Training time sensitivity study. To evaluate how much the amortized model will be influenced by randomness during training, we sample training data 5 times with different random seeds and then compute the averaged Spearman’s correlation among all pairs of runs. The standard deviation is less than 1e-2. Our amortized model is stable against training time randomness with only 10% of data.

Method	MNLI	Yelp-Polarity
Method	Spearman	Spearman
SVS-2	0.41	0.52
SVS-3	0.47	0.60
SVS-5	0.55	0.69
SVS-25	0.75	0.84
Our Amortized Model	0.42	0.61

Table 5: Approximation results for the Shapley explanation methods on MNLI and Yelp-Polarity datasets. Bold-faced numbers are the best in each column. Results are averaged over 5 runs. Spearman’s correlations are computed against SVS-25. Adapt-m means here how many sampled ordering

\sigma

s we used here to do local adaption (

m

in Algorithm 1 ).

Model	Acc
BOW	67.3
ShapCal (KS-2000)	67.4
ShapCal (Amortized)	68.0

Table 6: Calibration Experiments for Amortized Models. The explanation scores can help the calibrator achieves better accuracy on out-of-domain data than KS-2000.

Figure 1: Illustration of the Design Pipeline of AI Realtor .

Figure 2: Illustration of the inductive feature schema construction pipeline.

Figure 5: Faithfulness Scores for Hallucination Checks.

Figure 1: BACo is an inference-time token-level model collaboration framework that combines a base model’s diversity with its aligned counterpart’s quality. (A) A comparison of generated outputs. The aligned model produces high-quality but low-diversity outputs, while the base model produces high-diversity but low-quality outputs. BACo optimizes both diversity and quality by dynamically routing between them. The probabilities of token(s) are in grey next to text boxes. (B) Illustration of the diversity-quality trade-off space. Single models face a steep trade-off, where improving diversity by adjusting configuration (e.g., by increasing temperature) degrades quality. BACo achieves a better Pareto curve and allows for easy traversal across this frontier by adjusting the router’s threshold.

	Diversity	Quality
Model	(#Clusters) $\uparrow$	(Reward) $\uparrow$
Llama-3-8B	8.13	1.28
Llama-3-8B-Instruct	2.58	7.62

Table 1: Diversity and quality of Llama-3-8B ’s base and aligned models in our preliminary experiment. The results demonstrate a clear diversity-quality trade-off in the two model’s performance. Diversity is measured by the number of semantic equivalent clusters of the output group, and quality is the average reward per output from another LLM.

Figure 2: Illustration of the indicators on diversity-quality space: Coverage , the area under a method’s trade-off curve (blue shading for the blue method); Dominance , the proportion of the global Pareto frontier (highlighted curves) contributed by the method.

Routers	Lexical		Semantic		Overall
	Cov.	Dom.	Cov.	Dom.	Cov.	Dom.
- Rand	0.493	26.3%	0.409	17.0%	0.451	21.7%
- Judge	0.302	2.6%	0.254	0.6%	0.278	1.6%
- P	0.433	4.8%	0.397	8.5%	0.415	6.7%
- FC	0.419	3.2%	0.382	4.7%	0.401	4.0%
- P-Punc	0.495	30.7%	0.452	31.3%	0.474	31.0%

Table 3: Averaged performance of routers within BACo on NoveltyBench across all diversity–quality spaces. The -P-Punc router achieves the best overall performance. While the random router ( -Rand ) attains moderately strong results, mainly from increased surface-level lexical diversity, its performance drops sharply on semantic metrics, confirming that unguided switching fails to produce meaningful diversity. In contrast, -P-Punc delivers the most balanced and consistent results across both lexical and semantic evaluations, showing combination of designed routing strategies leads to more meaningful diversity.

Figure 4: Diversity-accuracy trade-off comparison on GSM8K. At comparable accuracy levels, BACo achieves higher diversity than the aligned baseline, indicating that the observed gains are not artifacts of open-ended evaluation metrics.

Figure 5: Pairwise intra-model repetition distribution under Artificial Hivemind. BACo ’s generation falls more in lower-similarity bins (bottom rows).

Method	Lexical		Semantic		Overall
	Cov.	Dom.	Cov.	Dom.	Cov.	Dom.
Base	0.142	9.8%	0.142	13.2%	0.142	11.5%
Aligned	0.273	43.8%	0.128	19.9%	0.200	31.8%
AACo	0.022	4.5%	0.006	7.0%	0.014	5.7%
BACo	0.495	42.0%	0.452	59.8%	0.474	50.9%

Table 4: Comparison of base-aligned and aligned-aligned collaboration (denoted as “ AACo ”) on NoveltyBench across all diversity–quality spaces. AACo yields little improvement on the diversity–quality trade-off compared with BACo , particularly in semantic diversity. The results demonstrate the necessity of involving a base and aligned model.

Figure 1: Average entropy shrinks with output positions for Llama-3-8B-Instruct on MMLU dataset.

Figure 2: A “forking” experiment on DeepSeek-Llama-3 shows early branching (high-entropy region) yields higher Maj@

k

on MMLU than late branching (low-entropy region).

Figure 3: The annealing schedule with different decay rates $d$ . A larger $d$ slows the cooling, front-loading exploration over more tokens. We set $c=10,\tau_{\mathrm{max}}=1.2,\tau_{\mathrm{min}}=0.1$ for illustration.

Figure 4: Pass@16 and Worst@16 performance evaluation in RL training. While EAD improves exploration of high-quality samples (even the worst outperform temperature sampling), the gain diminishes over time; importance sampling can supplement to correct bias and sustain training.

Figure 5: Pass@16 performance on Qwen-2.5-Math-7B. EAD enables better exploration than fixed-temperature sampling, yielding sustained gains in Pass@16 throughout training.

Figure 6: Entropy Dynamics in RL Training. Under commonly-used temperature sampling, trained with RL algorithm would make entropy decrease, sharply shrinking the exploration space for RL from beginning. While EAD could help RL algorithm to escape local minimum and do exploration when needed in the middle of RL training.

Figure 8: EAD is compatible with various RL algorithms and can significantly improve the model performance over time.

Figure 9: Inference-Time Scaling Evaluation for Different Decoding Methods using off-the-shelf Qwen2.5 models. We could see that EAD improves traditional temperature sampling. We set

\tau_{\text{max}}=1.2,\tau_{\text{min}}=0.1,d=25

for EAD.

Figure 10: Reverse Annealing experiment results. The reverse schedule fails to surpass normal temperature sampling.

Panel of Figure 11 — Figure 11: Ablation on hyperparameter sensitivity. Left: Ablation on decay cap $d_{\max}$ . “Vanilla” denotes EAD with the default $d_{\max}=40,000$ . Setting $d_{\max}=25$ (where $d_{0}=25$ ) mimics a fixed temperature schedule (effectively $\alpha=0$ ), leading to performance drops as the schedule fails to adapt to longer responses. EAD remains robust provided $d_{\max}$ allows for smooth temperature reduction (e.g., $d_{\max}\geq 200$ ). Right: Ablation on growth factor $\alpha$ and initial decay $d_{0}$ . With a small $d_{0}$ , a positive growth factor ( $\alpha>0$ ) is required to adapt to increasing response lengths and maintain a smooth intra-sequence schedule. While a large $d_{0}$ ensures smoothness even with $\alpha=0$ , it limits exploration, resulting in lower learning efficiency.

Figure 12: Stress-test of EAD using the DAPO recipe on AIME24. EAD outperforms the temperature sampling baseline across various hyperparameter setups.

I’m 18m and I’ve been taking norcos since I was 16 but just on and off. Starting this year I’ve been taking it every day basically and now I’m tired of it. I still get high so ig my addiction isn’t that bad as others but I don’t want to get to that point. I’m tired of chasing the high. I’ve spent at least 3k on norcos this year and I can’t control myself . I try to go a day sober but my mind is telling me I need and then withdrawals starts […]

Table 1: A self-disclosure from a user on Reddit going through the cycle of Opioid Addiction.

Medical Use	Oxycodone for wisdom teeth removal I just got 4 wisdom teeth plus another tooth in my palette removed and got prescribed 1 or 2 5mg tablets of oxy (Endone) each time . He recommended to avoid it if I could since I’m 43kg and have no tolerance. […]
Misuse	Oxy nod but no euphoria? Hi everyone, I tried oxy for the first time a few weeks back snorting a prolonged 20mg tablet and felt pretty good . Wednesday I dropped 9 of the 5mg capsules over a couple hours and was nodding strongly […]
Addiction	Well y’all were right. The sickness came. And is the worst i’ve ever experienced. Took subs, went into pwd accidentally and jump started the methadone sickness. I am to the point that I just have to get off this godforsaken mountain and go back to my ex and get back in the clinic bc at this rate i’m afraid i’m gonna end up killing myself .[…].
Recovery	It’s my birthday! One year off opiates It’s been 365 days since I decided to take back control of my body. I was highly dependent and addicted to prescribed opiates […]
Relapse	So high. 18 hours later. Still so high. So I’m pissed at myself. I was clean from heroin for 11 months and last night I did some . And for no reason too […]
Not Using	Partners of an Opiate addict in recovery How do you guys do this? I feel like I am having an incredibly hard time "moving on". I have nightmares of my partner oding, dying, and pretty much anything else that involves drug use . I over analyze everything […]

Table 2: Example for each Opioid Usage category. The underlined bold text represents the title of each post. Highlighted text represents salient spans annotated by humans as explanations for the label.

Category	Novice	Expert
Misuse	22.10	20.0
Addiction	29.15	12.53
Recovery	18.89	25.49
Relapse	4.65	3.96
Medical Use	7.05	3.52
Not Using	18.17	34.51

Table 3: Distribution (%) of OUD categories in novice- and expert-annotated data.

Is it safe to mix 20mg oxy with a few standard drinks? Seen online it’s dangerous but I don’t trust a lot of those harm reduction websites, most of them are whack. EDIT: I didn’t do it don’t worry, thanks for the info

Table 4: An annotation example demonstrating the role of explanation annotations for understanding annotator disagreement: the red is associated with “Misuse” and the blue with “Not Using”.

Given the following title, text, and explanation from the text, please identify the appropriate opioid usage category among the following types: ’Medical Use’, ’Misuse’ ,’Recovery’, ’Relapse’, ’Addiction’, ’Not Using’. Title: {{title}} Text: {{text}} Explanation: {{explanation}}

Table 5: Zero-shot instructional prompts for T0pp for w/ Explanation setting

Given the following title and text, please identify the appropriate opioid usage category among the following types: ’Medical Use’, ’Misuse’, ’Recovery’, ’Relapse’, ’Addiction’, ’Not Using’. Please provide an explanation for your answer by extracting the relevant span from the text that justifies your choice. { 13 in-context samples with the format below } Title: {{title}} Text: {{text}} Label: {{label}} Explanation: {{explanation}}

Table 6: Few-shot instructional prompt for GPT-3 for w/ Explanation setting.

Mode	Test Set	Zero-Shot	Few-Shot		Supervised
Mode	Test Set	T0pp	GPT3	GPT4	DeBERTa	T5-3B	T5-11B
w/o Explanation	Expert	48.9 / 46.9	62.2 / 57.1	55.4 / 50.2	67.6 / 65.7	63.5 / 61.2	71.4 / 70.4
w/o Explanation	Novice	62.0 / 60.8	66.4 / 65.4	63.3 / 60.0	74.0 / 74.4	72.7 / 71.4	80.9 / 81.5
w/ Explanation	Expert	48.9 / 47.4	61.1 / 54.5	63.2 / 59.1	73.8 / 72.6	64.4 / 65.5	76.6 / 77.0
w/ Explanation	Novice	62.7 / 58.5	66.9 / 65.9	67.3 / 64.8	81.3 / 81.9	78.7 / 77.3	84.0 / 84.0

Table 7: Performance of different models on expert and novice-annotated test data in a zero-shot/few-shot/supervised setting. w/o Explanation and w/ Explanation models refers to the setting where Explanations are excluded or included as part of the input. Results are presented in "Accuracy/F1" format.

Explanation	Test Set	T5-11B	DeBERTa
Gold	Expert	76.6	73.8
Gold	Novice	84.0	81.3
Silver	Expert	70.3	70.3
Silver	Novice	78.0	78.0
Random	Expert	69.6 ( $\pm$ 1.3)	65.8 ( $\pm$ 1.1)
Random	Novice	68.3 ( $\pm$ 1.7)	67.9 ( $\pm$ 1.4)

Table 8: Accuracy of T5-11B and DeBERTa w/ Explanation model on expert and novice annotated test sets by varying the quality of explanations. We can observe the importance of including gold explanations.

Setting	Not Using - Misuse		Recovery - Addiction		Not Using - Addiction
Setting	$\rightarrow$	$\leftarrow$	$\rightarrow$	$\leftarrow$	$\rightarrow$	$\leftarrow$
T5-11B- w/o Explanation	20%	4.5%	20%	0%	14%	0%
DeBERTa- w/o Explanation	18%	0%	18%	3.6%	16%	5.5%
T5-11B- w/ Explanation	16%	8%	14%	0%	13%	0%
DeBERTa- w/ Explanation	15%	2.3%	15%	3.6%	15%	0%

Table 9: Model error analysis over expert annotation data.

\rightarrow

means the expert-annotated label is on the left side and the predicted label is on the right side, and

\leftarrow

vice versa. Percentages in the table represent the error rate in each expert labeled category. The results demonstrate that the main confusion for the models exists in “Not Using - Misuse”, “Recovery - Addiction”, and “Not Using - Addiction”. These problems surface in an asymmetrical pattern – one mis-classification direction matters more in the confusion.

Dataset	Agreement	T5-11B (w/o expl)	T5-11B (w/ expl)	DeBERTa (w/o expl)	DeBERTa (w/ expl)
Novice	Unanimous Consent	95.5 / 94.0	97.1 / 95.7	87.1 / 85.9	90.0 / 89.6
	Arguable (Majority Vote)	68.0 / 66.7	72.5 / 71.3	62.5 / 60.8	73.8 / 75.2
	Arguable (All Annotations)	84.0 / 81.4	92.5 / 91.8	86.3 / 84.3	91.3 / 92.1
Expert (FirstBatch)	Unanimous Consent	92.7 / 85.0	96.4 / 97.1	83.6 / 72.7	89.1 / 83.2
	Arguable (Majority Vote)	60.3 / 59.9	64.0 / 66.0	56.9 / 58.2	65.0 / 66.2
	Arguable (All Annotations)	84.6 / 83.3	86.8 / 85.8	82.5 / 82.7	86.9 / 87.3

Table 10: Ablation study on dataset annotation disagreement. Results are presented in "Accuracy/F1" format. “Unanimous Consent”: all annotators agree on the same label; “Arguable (Majority Vote)”: annotators have some disagreements, and majority voting is used as the correct label; “Arguable (All annotations)”: disagreements exist, and any annotator label is considered correct. We observe that models perform better on data with unanimous agreement than on arguable data.

Appendix figures & tables65 assets

Supplementary material from the paper’s appendix.

Appendix

Normal Instruction	Step-EM / State-EM
Normal Instruction	2-shot	5-shot
NL Functor + NL Argument	$22\%$ / $92\%$	$36\%$ / $95\%$
Synthetic Functor + Synthetic Argument	$19\%$ / $92\%$	$52\%$ / $96\%$
Actions w/ Distractor
NL Functor + NL Argument	$18\%$ / $91\%$	$44\%$ / $96\%$
Synthetic Functor + Synthetic Argument	$34\%$ / $93\%$	$50\%$ / $96\%$

Appendix

Table 1: Action understanding experiment results for 10-box environment on ChatGPT. Metrics here are shown in the format of "Step-EM/State-EM". We use 50 samples with various number of steps for experiments.

Normal Instruction	Step-EM / State-EM
Normal Instruction	2-shot	3-shot	5-shot
NL Functor + NL Argument	$22\%/92\%$	$34\%/93\%$	$36\%/95\%$
Synthetic Functor + NL Argument	$14\%/90\%$	$20\%/93\%$	$30\%/94\%$
NL Functor + Synthetic Argument	$44\%/95\%$	$30\%/94\%$	$74\%/98\%$
Synthetic Functor + Synthetic Argument	$19\%/92\%$	$22\%/93\%$	$52\%/96\%$
Counter-Intuitive Instruction (On NL)

Appendix

Table 2: Experiment results on ChatGPT Robustness check for state tracking in 10-box environment. Metrics here are presented in the format of “Step-EM / State-EM”. We use 50 samples with various number of steps for experiments.

Normal Instruction	State-EM, 2-shot
Normal Instruction	ChatGPT	Text-Davinci-003
NL Functor + NL Argument	$92\%$	$96\%$
Synthetic Functor + NL Argument	$90\%$	$94\%$
NL Functor + Synthetic Argument	$95\%$	$94\%$
Synthetic Functor + Synthetic Argument	$92\%$	$88\%$

Appendix

Table 3: Experiment Results Comparing ChatGPT and Davinci-003. Metrics here are State-EM. We use 50 samples with 2-shot setting and various number of steps for experiments.

Figure 1: An baseline example of the prompt with partial code derived from original MBPP prompt for robustness evaluation on code.

Figure 2: An example of the DeadCodeInserter perturbation.

Figure 3: An example of the For-While Switch perturbation.

Figure 4: An example of the OperandSwap perturbation.

Figure 5: An example of the VarRenamerCB perturbation.

Figure 6: An example of the VarRenamerNaive perturbation.

Figure 7: An example of the VarRenamerRN perturbation.

Figure 8: An example of the Tab-Indent perturbation.

Figure 9: An example of the Line Split perturbation.

Figure 10: An example of the Doc2Comments perturbation.

Figure 11: An example of the NewlineRandom perturbation.

Figure 12: An example of the NewlineAfterCode perturbation.

Figure 13: An example of the NewlineAfterDoc perturbation.

Panel of Figure 14 — Figure 14: HumanEval showcase 1 illustrating failure case under deadcode insertion .

Panel of Figure 15 — Figure 15: HumanEval showcase 2 illustrating failure case under NewlineAfterCode insertion.

Panel of Figure 16 — Figure 16: HumanEval showcase 3 illustrating failure case under ButterFingers perturbations on docstrings.

Fleiss Kappa	HumanEval	MBPP
Naturalness (Nominal) $\uparrow$	0.362	0.301
Naturalness (Perturbed) $\uparrow$	0.435	0.326
Semantics Similarity $\uparrow$	0.658	0.461

Appendix

Table 1: Fleiss Kappa of human evaluation.

Categories	Perturbations	HumanEval	MBPP
Docstring	BackTranslation	0.91	0.95
	ButterFingers	0.87	0.89
	ChangeCharCase	1.00	1.00
	EnglishInflectionalVariation	0.96	0.93
	SwapCharacters	0.90	0.87
	SynonymInsertion	0.91	0.88

Appendix

Table 2: Cosine similarity for each type of perturbations where perturbed and unperturbed docstrings/function names are embedded by the SOTA sentence transformer.

Categories	Perturbations	HumanEval		MBPP
		CodeBLEU	CodeBLEU	CodeBLEU	CodeBLEU
		(syntax)	(dataflow)	(syntax)	(dataflow)
Syntax	DeadCodeInserter	0.85	0.79	0.72	0.67
	For-While Switch	0.92	0.90	0.84	0.86
	OperandSwap	0.91	1.00	0.90	1.00
	VarRenamerCB	1.00	0.99	0.93	0.99

Appendix

Table 3: CodeBLEU syntax and format similarity scores between unperturbed codes and perturbed ones with each type of our syntax and format transformations.

HumanEval	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
HumanEval	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.232	0.140	0.262	0.195	0.305	0.195	0.104	0.152	0.122
BackTranslation	RP 5 @1 $\uparrow$	0.213	0.116	0.238	0.159	0.244	0.152	0.098	0.134	0.098
	RD 5 @1(%) $\downarrow$	7.89	17.39	9.30	18.75	20.00	21.88	5.88	12.00	20.00
	RR 5 @1(%) $\downarrow$	4.27	6.10	8.54	6.10	10.98	5.49	3.05	3.05	3.66
ButterFingers	RP 5 @1 $\uparrow$	0.165	0.098	0.171	0.122	0.189	0.116	0.067	0.098	0.067

Appendix

Table 4: Robustness evaluation for each type of docstring perturbations on HumanEval.

MBPP	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
MBPP	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.317	0.191	0.361	0.221	0.407	0.241	0.128	0.199	0.133
BackTranslation	RP 5 @1 $\uparrow$	0.304	0.186	0.360	0.222	0.387	0.230	0.119	0.177	0.128
	RD 5 @1(%) $\downarrow$	4.21	2.69	0.28	-0.47	4.80	4.68	7.20	11.34	3.85
	RR 5 @1(%) $\downarrow$	6.26	6.06	7.91	6.06	6.26	7.08	4.00	5.95	5.44
ButterFingers Perturbation	RP 5 @1 $\uparrow$	0.210	0.092	0.240	0.100	0.280	0.126	0.044	0.082	0.057

Appendix

Table 5: Robustness evaluation for each type of docstring perturbations on MBPP.

HumanEval	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
HumanEval	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.232	0.140	0.262	0.195	0.305	0.195	0.104	0.152	0.122
CamelCase	RP 5 @1 $\uparrow$	0.238	0.140	0.256	0.201	0.293	0.165	0.098	0.152	0.116
	RD 5 @1(%) $\downarrow$	-2.63	0.00	2.33	-3.13	4.00	15.62	5.88	0.00	5.00
	RR 5 @1(%) $\downarrow$	1.83	1.22	3.05	3.05	3.66	3.05	3.05	1.22	0.61
ButterFinger	RP 5 @1 $\uparrow$	0.195	0.104	0.232	0.177	0.274	0.159	0.098	0.140	0.091

Appendix

Table 6: Robustness evaluation for each type of function name perturbations on HumanEval.

MBPP	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
MBPP	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.317	0.191	0.361	0.221	0.407	0.241	0.128	0.199	0.133
CamelCase	RP 5 @1 $\uparrow$	0.316	0.196	0.367	0.219	0.408	0.245	0.116	0.194	0.134
	RD 5 @1(%) $\downarrow$	0.32	-2.69	-1.42	0.93	-0.25	-1.70	9.60	2.58	-0.77
	RR 5 @1(%) $\downarrow$	5.44	5.44	7.29	5.34	7.08	4.52	5.75	5.03	3.18
ButterFinger	RP 5 @1 $\uparrow$	0.312	0.185	0.370	0.203	0.412	0.231	0.110	0.175	0.117

Appendix

Table 7: Robustness evaluation for each type of function name perturbations on MBPP.

HumanEval	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
HumanEval	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.402	0.293	0.518	0.366	0.549	0.390	0.189	0.323	0.250
DeadCodeInserter	RP 5 @1 $\uparrow$	0.116	0.079	0.152	0.110	0.159	0.091	0.055	0.079	0.079
	RD 5 @1(%) $\downarrow$	71.21	72.92	70.59	70.00	71.11	76.56	70.97	75.47	68.29
	RR 5 @1(%) $\downarrow$	37.80	30.49	41.46	32.93	45.12	37.20	17.07	30.49	27.44
ForWhile TransformerFirst	RP 5 @1 $\uparrow$	0.384	0.226	0.500	0.305	0.537	0.384	0.159	0.280	0.213

Appendix

Table 8: Robustness evaluation for each type of code syntax perturbations on HumanEval.

MBPP	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
MBPP	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.450	0.285	0.535	0.331	0.571	0.379	0.219	0.292	0.176
DeadCodeInserter	RP 5 @1 $\uparrow$	0.043	0.020	0.044	0.024	0.055	0.025	0.015	0.015	0.009
	RD 5 @1(%) $\downarrow$	90.41	93.17	91.75	92.86	90.29	93.50	92.96	94.72	94.74
	RR 5 @1(%) $\downarrow$	52.05	37.99	57.39	39.12	60.57	44.87	29.26	37.78	24.95
ForWhile TransformerFirst	RP 5 @1 $\uparrow$	0.432	0.259	0.497	0.303	0.532	0.346	0.182	0.245	0.149

Appendix

Table 9: Robustness evaluation for each type of code syntax perturbations on MBPP.

HumanEval	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
HumanEval	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.402	0.293	0.518	0.366	0.549	0.390	0.189	0.323	0.250
Tab-Indent	RP 5 @1 $\uparrow$	0.415	0.305	0.518	0.354	0.561	0.396	0.146	0.299	0.244
	RD 5 @1(%) $\downarrow$	-3.03	-4.17	0.00	3.33	-2.22	-1.56	22.58	7.55	2.44
	RR 5 @1(%) $\downarrow$	3.66	4.88	8.54	4.88	3.66	4.27	7.93	9.76	7.93
Line Split	RP 5 @1 $\uparrow$	0.384	0.274	0.500	0.378	0.524	0.390	0.171	0.305	0.244

Appendix

Table 10: Robustness evaluation for each type of code format perturbations on HumanEval.

MBPP	Metric	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	CodeGen	InCoder	InCoder	GPT-J
MBPP	Metric	2B mono	2B multi	6B mono	6B multi	16B mono	16B multi	1B	6B	6B
Nominal	RP 5 @1 $\uparrow$	0.450	0.285	0.535	0.331	0.571	0.379	0.219	0.292	0.176
Tab-Indent	RP 5 @1 $\uparrow$	0.452	0.302	0.530	0.339	0.566	0.385	0.208	0.325	0.176
	RD 5 @1(%) $\downarrow$	-0.46	-5.76	0.96	-2.48	0.90	-1.63	4.69	-11.62	0.00
	RR 5 @1(%) $\downarrow$	6.37	6.98	5.85	8.01	6.88	6.78	9.24	12.22	7.60
Line Split	RP 5 @1 $\uparrow$	0.445	0.275	0.524	0.326	0.556	0.378	0.187	0.283	0.163

Appendix

Table 11: Robustness evaluation for each type of code format perturbations on MBPP.

Panel of Figure 17 — Figure 17: Robust Drop 1 @1 and Robust Relative 1 @1 on CodeGen-16B-mono under different $k$ using sampling $n=100$ . Robust Drop remains stable while Robust Pass and Robust Relative increases with k.

Category	Metric	$n=1$	$n=10$	$n=100$
Docstring	Nominal $\uparrow$	0.287	0.308	0.306
	RP 1 @1 $\uparrow$	0.128	0.140	0.143
	RD 1 @1(%) $\downarrow$	55.32	54.46	53.34
	RR 1 @1(%) $\downarrow$	15.85	16.77	16.55
Function	Nominal $\uparrow$	0.287	0.308	0.306
	RP 1 @1 $\uparrow$	0.183	0.180	0.183

Appendix

Table 12: Generation variances for three robustness metrics with different sampling

n

on CodeGen-16B-mono and HumanEval.

Figure 1: Accuracy vs. Correctness Margin for each artificial response

Figure 2: Synthesized Artificial Contexts Example

Metric	Standard Prompting	Exploration-Only	Self-Reflection
TruthfulQA
Rouge-1	$57.5\pm 1.1$	$55.1$	59.0
BLEURT	$66.8\pm 1.9$	$70.1$	72.9
HotpotQA
Accuracy	$80.2\pm 0.4$	$69.7$	$71.9$

Appendix

Table 1: Self-Reflection experiment results using iterative prompting. Bold-faced numbers at each row indicate the best-performing method under each metric.

Metric	Standard Prompting	Exploration-Only	Self-Reflection
TruthfulQA
Rouge-1	$53.8\pm 0.4$	$51.7$	53.8
BLEURT	$60.9\pm 0.6$	$58.2$	63.0
HotpotQA
Accuracy*	$61.0\pm 1.0$	62.9	$57.5$

Appendix

Table 2: Self-reflection SR 2 V experiment results on QA datasets using LLaMA-2-chat. Bold-facing indicates the best-performing method under each metric. *Evaluated manually.

Figure 3: Performance Decomposition on Question Difficulty and Response Accuracy (LLaMA-2-chat).

Metric	Standard Prompting	Exploration-Only	Self-Reflection
TruthfulQA
Rouge-1	$59.1\pm 1.0$	$61.3$	63.3
BLEURT	$71.5\pm 0.4$	73.9	$71.7$
HotpotQA
Accuracy*	$89.8\pm 0.3$	90.9	$89.2$

Appendix

Table 3: Self-reflection SR 2 V experiment results on QA datasets using Mixtral-8x7B-v0.1. Bold-facing indicates the best-performing method under each metric. *Evaluated manually.

Figure 4: Performance Decomposition on Question Difficulty and Response Accuracy (Mixtral-8x7B-v0.1).

Figure 5: Performance Decomposition on Question Difficulty and Response Accuracy (Artificial Responses) for Mixtral-8x7B-v0.1.

Figure 6: Majority Voting Analysis (Mixtral-8x7B-v0.1).

Figure 3: Monte Carlo (MC) sampling systematically underestimates entropy. The plot shows that the estimated entropy of sequences from Llama-3-8B-Instruct increases with the number of MC samples (

M

). A small sample size fails to cover the vast output space, leading to an underestimation of the true entropy. This bias is difficult to eliminate without incurring substantial computational costs.

Panel of Figure 6 — Figure 6: Additional verification of BF dynamics on XSUM, Aya, and the Qwen3 family. Values are exponential-moving averages of per-step perplexity (smoothing $0.1$ ).

Panel of Figure 8 — Figure 8: BF Output Dynamic for OLMo-2 (7B & 13B) across alignment stages. We compare Base and DPO models on Creative StoryGen and MMLU. Note that we treat DPO as the aligned model for OLMo-2, following the convention in ( olmo20242 ) .

Panel of Figure 10 — Figure 10: Output Perplexity Dynamics in Nudging Experiments.

Panel of Figure 12 — Figure 12: Correlational Analysis of BF and Distinct-N. We can find there is no consistent correlation between Distinct-N and BF.

Figure 13: Signed $R^{2}$ values heatmap investigating correlation between BF and Min-K $\%$ .

Hyperparameter	Value(s)
Training Batch Size	1024
Max Prompt Length	1024
Max Response Length	3072
Mini Batch Size	256
Micro Batch Size Per GPU	4
Learning Rate	$10^{-6}$

Appendix

Table 1: Hyperparameter Setup for Running Minimal-RL recipe.

Figure 2: Compared with normal temperature sampling, EAD can naturally incentivize the model to generate longer reasoning chains.

Method	Time/Sample (s)	Tok/s	Peak Memory
Aligned	$9.8\pm 3.9$	40.0	15.1 GiB
Response Ensemble	$10.9\pm 4.0$	34.8	15.4 GiB
Diverse Beam Search	$17.0\pm 5.9$	17.0	15.8 GiB
Logits Ensemble	$25.5\pm 0.5$	20.1	30.1 GiB
In-Context Resampling	$90.5\pm 35.3$	4.1	15.5 GiB
Paraphrase Prompting	$117.9\pm 41.1$	3.3	15.1 GiB

Appendix

Table 1: Wall-clock runtime comparison on a single A100 80 GB. Tok/s is computed over the final output. BACo is comparable to or faster than several inference-time diversity baselines while substantially outperforming them on diversity–quality ( Table 2 ).

Method	Lexical		Semantic		Overall
	Cov.	Dom.	Cov.	Dom.	Cov.	Dom.
Base	0.142	9.8%	0.142	13.1%	0.142	11.4%
Aligned	0.273	40.1%	0.128	17.2%	0.200	28.6%
Nudging	0.192	6.8%	0.161	7.6%	0.176	7.2%
Decoding	-	0.8%	-	1.0%	-	0.9%
Prompting best	-	8.0%	-	6.5%	-	7.3%

Appendix

Table 2: Comparison results on NoveltyBench. For space-saving, we present the best method in each category.

Method	Lexical		Semantic		Overall
	Cov.	Dom.	Cov.	Dom.	Cov.	Dom.
Base	0.000	1.9%	0.000	6.8%	0.000	4.38%
Aligned	0.253	59.2%	0.077	29.1%	0.165	44.1%
Nudging	0.430	11.4%	0.387	15.6%	0.408	13.5%
BACo best	0.473	27.4%	0.454	48.5%	0.463	38.0%

Appendix

Table 3: Comparison results on WildChat. For space saving, we present the best router, - P-Punc , as BACo ’s representative.

Method	Lexical		Semantic		Overall
	Cov.	Dom.	Cov.	Dom.	Cov.	Dom.
Base	0.151	26.3%	0.153	28.1%	0.152	27.2%
Aligned	0.282	47.7%	0.106	41.2%	0.194	44.4%
Nudging	0.205	9.7%	0.194	6.5%	0.199	8.1%
BACo best	0.367	16.3%	0.174	24.2%	0.271	20.3%

Appendix

Table 4: Comparison results on Narrative-Discourse. For space saving, we present the best router, - P-Punc , as BACo ’s representative.

Method	Lexical		Semantic		Overall
	Cov.	Dom.	Cov.	Dom.	Cov.	Dom.
Base	0.142	7.6%	0.142	10.9%	0.142	9.2%
Aligned	0.273	36.5%	0.128	15.8%	0.200	26.1%
In-context Prompt	-	0.0%	-	2.2%	-	1.1%
Paraphrase Prompt	-	8.0%	-	5.6%	-	6.8%
Diverse BS Decoding	-	0.8%	-	1.0%	-	0.9%

Appendix

Table 5: Comparison of all methods (baselines and BACo routers) on NoveltyBench. BACo All reports the best Cov. across all routers and the Dom. sum over all routers. The lower half of the table provides the performance of individual routers. Note that routers distribute Dom. values given to the metric definition; therefore, Dom. values in the top and bottom halves of the table are not directly comparable.

Figure 1: Diversity-quality trade-off comparison between BACo and the aligned model baseline on IFEval. BACo demonstrates a more effective optimization of the trade-off. The table presents all the trade-off spaces tested.

Figure 2: Diversity-accuracy trade-off comparison between BACo and the aligned model baseline on GSM8K. BACo demonstrates a more effective optimization of the trade-off. The table presents all the trade-off spaces tested.

Method	Overall Cov.	Overall Dom.
Base	0.098	11.7%
Aligned	0.209	30.5%
Nudging	0.281	5.5%
Others	-	8.3%
BACo - P-Punc	0.236	44.0%

Appendix

Table 6: Results of BACo on Olmo2-7B and Olmo2-7B-Instruct on NoveltyBench.

Context Engineering

See all 12 papers

Jun 15, 2026

SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data

As large language models (LLMs) are increasingly deployed in real-world applications, alignment is no longer governed by a single universal notion of safety or helpfulness, but instead by provider- or application-specific model specifications. These specifications are typically long, structured, and frequently updated, yet existing alignment pipelines lack a systematic mechanism to operationalize them as training signals. In this paper, we propose specification-grounded alignment, a new alignment paradigm that treats provider-authored model specifications as the primary alignment target rather than abstract principles or static benchmarks. To instantiate this paradigm, we introduce SpecAlign, a framework that synthesizes alignment data directly from specification documents. SpecAlign combines structured rule annotation, controllable specification instantiation, and multi-agent adversarial data synthesis to generate fine-grained, boundary-aware preference pairs that capture both compliant behaviors and meaningful specification violations. Experiments across multiple model specifications and backbone models demonstrate that training with SpecAlign consistently improves rule compliance while preserving general capabilities and avoiding over-conservative behavior. These results suggest that grounding alignment in explicit model specifications enables rapid, precise, and scalable adaptation of LLM behavior to evolving policy requirements.

Wenjie Wang, Yue Huang, Zhengqing Yuan +6

University of Notre Dame · Carnegie Mellon University · LMU Munich +1

LLM Alignment Synthetic Data Generation

Ask PDF

Jun 3, 2026

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

Safety-aligned Large Language Models (LLMs) remain vulnerable to interventions during inference that redirect generation toward harmful outputs. Recent work attributes this to shallow safety, where alignment concentrates in the first few output tokens. We show that shallow safety is a special case of a broader inference-time vulnerability, in which short token injections at any generation step can substantially alter subsequent safety behavior. We also find that a model's alignment with refusal directions in its hidden states does not predict its robustness to such injection, revealing that internal state alone does not determine generation behavior under perturbation. To address this, we align models directly on generation trajectories constructed by simulating mid-sequence perturbation, and show that this improves robustness to mid-sequence injection and generalizes to attacks that exploit early-token generation. Our work argues that robust safety alignment requires training on the generation process itself, not only its outputs.

Kyungmin Park, Taesup Kim

Hankuk University of Foreign Studies · Seoul National University

Adversarial Attacks on LLMs LLM Safety Alignment Adversarial Robustness of Language Models Inference-Time Language Model Alignment

Ask PDF

Oct 2, 2026

DNAlign: Dynamic Null-Space Safe Alignment for LLMs

Ensuring the safe and reliable deployment of large language models (LLMs) remains a fundamental challenge. Existing safety alignment approaches either incur high computational cost or unintentionally disrupt the model's core knowledge, leading to degraded fluency and factual accuracy on benign tasks. This reveals a persistent trade-off between safety and utility. We propose DNAlign, a lightweight alignment framework that integrates control-theoretic optimization with null-space projection. By treating the LLM as a dynamic system, the proposed framework introduces controllable perturbations to steer generation toward safe behavior. A key component is the projection module, which restricts these perturbations to the harmful-related subspace derived from neutral hidden states, thereby preserving general knowledge and response quality. A value function trained on human preference data adaptively optimizes the control signals to align with human safety preferences. Extensive evaluations across multiple LLM backbones demonstrate that our framework consistently reduces harmful outputs while maintaining fluency, coherence, and factual utility. It achieves superior overall performance compared to prior alignment baselines without sacrificing generation diversity. These results indicate that the proposed framework provides an effective and practically deployable solution for safe LLM alignment. Code is available at https://anonymous.4open.science/r/DNAlign.

Jisheng Dang, Yushuo Zhao, Dewei Liu +6

School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China · National University of Singapore, Singapore 119077 · School of Medical Technology, Beijing Institute of Technology, Beijing 100081, China

LLM Alignment LLM Safety Alignment Inference-Time Language Model Alignment

Ask PDF

Abstract

Context Engineering

Explore similar work

SpecAlign: Efficient Specification-Grounded Alignment of Large Language Models via Synthetic Data

Inference-Time Vulnerability Beyond Shallow Safety: Alignment Along Generation Trajectories

DNAlign: Dynamic Null-Space Safe Alignment for LLMs