Discovering Implicit Large Language Model Alignment Objectives

Edward Chen; Sanmi Koyejo; Carlos Guestrin

cs.LGFeb 17, 2026

Discovering Implicit Large Language Model Alignment Objectives

Authors: Edward Chen, Sanmi Koyejo, Carlos Guestrin

Organizations: Stanford University.

Abstract

Large language model (LLM) alignment relies on complex reward signals that often obscure the specific behaviors being incentivized, creating critical risks of misalignment and reward hacking. Existing interpretation methods typically rely on pre-defined rubrics, risking the omission of "unknown unknowns", or fail to identify objectives that comprehensively cover and are causal to the model behavior. To address these limitations, we introduce Obj-Disco, a framework that automatically decomposes an alignment reward signal into a sparse, weighted combination of human-interpretable natural language objectives. Our approach utilizes an iterative greedy algorithm to analyze behavioral changes across training checkpoints, identifying and validating candidate objectives that best explain the residual reward signal. Extensive evaluations across diverse tasks, model sizes, and alignment algorithms demonstrate the framework's robustness. Experiments with popular open-source reward models show that the framework consistently captures > 90% of reward behavior, a finding further corroborated by human evaluation. Additionally, a case study on alignment with an open-source reward model reveals that Obj-Disco can successfully identify latent misaligned incentives that emerge alongside intended behaviors. Our work provides a crucial tool for uncovering the implicit objectives in LLM alignment, paving the way for more transparent and safer AI development.

Figures & tables

Figure 1: Schematic of Obj-Disco. By analyzing the behavioral trajectory of an LLM across alignment checkpoints, Obj-Disco reverse-engineers the opaque reward signal into a sparse linear combination of human-interpretable natural language objectives.

Panel of Figure 2 — Figure 2: Obj-Disco Overview and Qualitative Results. (Left) Obj-Disco employs an iterative greedy search to construct the Discovered Interpretable Reward (DIR). A proposer LLM identifies candidates from high-residual samples, which are then verified for interpretability and trend predictability. (Right) Discovered objectives, and their weights, for open-source reward models (RM), demonstrating that Obj-Disco successfully recovers domain-specific alignment goals (e.g., conciseness for summaries, logic for code).

Figure 3: Model-Fit results across controlled and open-source reward model (RM) settings. Each subplot shows Model-Fit (y-axis) across alignment checkpoints (x-axis), averaged over 6 trials with standard error bars. A score approaching 1.0 indicates that the discovered objectives successfully replicate the true alignment signal. Top Row (Controlled): Ground-truth reward is a known linear combination of 3 LLM-as-a-Judge objectives on the TLDR summarization task. (1) PPO, Llama-8B, (2) PPO, Qwen-4B, (3) GRPO, Llama-8B, (4) GRPO, Qwen-4B. Bottom Row (Open-Source RMs): Reward signal comes from real open-source reward models, all using Llama-8B with GRPO. (1) Alpaca w/ custom RM, (2) HH-RLHF w/ DeBERTaV3, (3) TLDR w/ DeBERTaV3, (4) Skywork-80K w/ Skywork-v2. Across all settings, Obj-Disco consistently achieves

>

90% Model-Fit, outperforming both baselines.

Panel of Figure 4 — Figure 4: Qualitative Comparison of Case Study Discovered Objectives. Only Obj-Disco successfully identified the latent misaligned behavior (in red) implicitly incentivized by the open-source reward model. Baseline methods largely discovered narrow objectives indicative of helpfulness, failing to capture misaligned behavior. Only active objectives (non-zero coefficients) are shown.

Figure 5: Importance of Model Trajectory Ablation. Obj-Disco-Static only compares the base model and the final checkpoint, lacking the dynamic trajectory of Obj-Disco. Setting: Multi-Turn Dialogue, GRPO, Llama-8B. Controlled Evaluation. (6 trials)

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 6 — Figure 6: Input prompt template used for the Objectives Discovery phase of Obj-Disco.

Hyperparameter	Value
Learning Rate	$2\times 10^{-6}$
LoRA Rank	256
LoRA Alpha	128
Batch Size	32
Mini-Batch Size	4
Gradient Accumulation Steps	2

Appendix

Table 1: Hyperparameters for PPO training.

Hyperparameter	Value
Learning Rate	$2\times 10^{-6}$
LoRA Rank	256
LoRA Alpha	128
Batch Size	8
Gradient Accumulation Steps	1
Number of Generations	8

Appendix

Table 2: Hyperparameters for GRPO training.

Panel of Figure 7 — Figure 7: The prompt template used by the LLM-as-a-Judge to score individual responses against a specific objective.

Panel of Figure 8 — Figure 8: The prompt used to dynamically generate calibration examples for specific objectives and datasets.

Panel of Figure 9 — Figure 9: The prompt used to dynamically generate detailed scoring rubrics based on an objective description and calibration examples.

Panel of Figure 10 — Figure 10: The prompt template used by the LLM-as-a-Judge to determine whether a discovered objective represents a misalignment in the misalignment case study (Section 5.2 ).

Panel of Figure 11 — Figure 11: Prompt used to generate distractor objectives for the Objective Explanations user study.

Figure 12: Instructions provided to participants of the Objective Explanations human-subject user study.

Figure 13: Instructional example provided to participants of the Objective Explanations human-subject user study. Part of the instructions page.

Figure 14: An example of part of a question provided to participants of the Objective Explanations human-subject user study. Users are able to click through and view the different responses in the trajectory.

Figure 15: An example of choices of objectives for a single question shown to participants of the Objective Explanations human-subject user study.

Figure 16: Instructions provided to participants of the study evaluating the causality of the discovered objectives.

Figure 17: An example of a question provided to participants of the study evaluating the causality of the discovered objectives.

Figure 18: An example of a set of candidate LLM responses provided to participants of the study evaluating the causality of the discovered objectives. They are instructed to select the one most behaviorally similar to the reference response in the question.

Figure 19: An example of the choices provided to participants of the study evaluating the causality of the discovered objectives.

Figure 20: Composition Function Ablation Results. Setting: Multi-Turn Dialogue, GRPO, Llama-8B. Controlled Evaluation. (6 trials)

Figure 21: Parallel Trajectories Ablation Results. Setting: Multi-Turn Dialogue, GRPO, Llama-8B. Controlled Evaluation. (6 trials)

Figure 22: Candidate Objectives Ablation Results. Setting: Multi-Turn Dialogue, GRPO, Llama-8B. Controlled Evaluation. (6 trials)

Panel of Figure 23 — Figure 23: Qualitative Comparison of Obj-Disco Informative Sampling vs. Random Ablation. Four trials of discovered objectives with informative sampling are shown on the top, discovered objectives with random sampling are shown on the bottom, and the instances of misalignment are highlighted in red. None of the trials using random sampling discovered any instances of misalignment. Only active objectives (non-zero coefficients) are shown.

Panel of Figure 24 — Figure 24: Qualitative Comparison of Obj-Disco Fixed Ablation. The first four trials show discovered objectives for Fixed- $\alpha$ where only 3 objectives are randomly sampled from a pre-defined list and then selected from. The second four trials show discovered objectives for Fixed- $\alpha$ where 15 objectives are randomly sampled from a pre-defined list and selected from. Unintended negative behaviors are highlighted in red. Only active objectives (non-zero coefficients) are shown.

Panel of Figure 25 — Figure 25: Qualitative Comparison of Case Study Discovered Objectives for Obj-Disco and Obj-Disco-Static. Obj-Disco-Static only uses the base model and final model checkpoint for objectives discovery and, hence, is lacking the sequence of model checkpoints which Obj-Disco leverages. Unintended negative behaviors are highlighted in red. Only active objectives (non-zero coefficients) are shown.

Figure 26: Additional Controlled (Top) And Open-Source Reward Model (Bottom) Results. (Left to Right, Top to Bottom): (1) Controlled. HH-RLHF, PPO, Llama-8B, (2) Controlled. HH-RLHF, PPO, Qwen-4B, (3) Controlled. HH-RLHF, GRPO, Llama-8B, (4) Controlled. HH-RLHF, GRPO, Qwen-4B, (5) Reward Model. Alpaca, GRPO, Qwen-4B, (6) Reward Model. HH-RLHF, GRPO, Qwen-4B, (7) Reward Model. TLDR, GRPO, Qwen-4B, (8) Reward Model. Sky, GRPO, Qwen-4B. (6 trials)

Panel of Figure 27 — Figure 27: Definitions for the ground-truth objectives used in our controlled evaluation settings. The reward signal $R^{*}$ was computed as an equally weighted convex combination of these interpretable scores.

Panel of Figure 28 — Figure 28: Objective Explanations Sample Results. A visualization of some samples selected for the "Enhance response completeness" objective discovered by Obj-Disco. The figure displays two sample trajectories selected by our method. By observing the evolution from the initial model response ( $t_{1}$ ) to the final aligned response ( $t_{5}$ ), users can verify that the model is indeed optimizing for increasingly complete responses.

Panel of Figure 29 — Figure 29: Objective Explanations Sample Results. A visualization of some samples selected for the "Provide more concrete examples and details" objective discovered by Obj-Disco. The figure displays one sample trajectory selected by our method. By observing the evolution from the initial model response ( $t_{1}$ ) to the final aligned response ( $t_{5}$ ), users can verify that the model is indeed optimizing for more concrete examples and details.

Panel of Figure 30 — Figure 30: Objective Explanations Sample Results. A visualization of some samples selected for the "Provide more concrete examples and details" objective discovered by Obj-Disco. The figure displays one sample trajectory selected by our method. By observing the evolution from the initial model response ( $t_{1}$ ) to the final aligned response ( $t_{5}$ ), users can verify that the model is indeed optimizing for more concrete examples and details.

Panel of Figure 31 — Figure 31: Objective Explanations Sample Results (Continued From Figure 30 ). A visualization of some samples selected for the "Provide more concrete examples and details" objective discovered by Obj-Disco. The figure displays one sample trajectory selected by our method. By observing the evolution from the initial model response ( $t_{1}$ ) to the final aligned response ( $t_{5}$ ), users can verify that the model is indeed optimizing for more concrete examples and details.

Panel of Figure 32 — Figure 32: Qualitative Comparison of Discovered Interpretable Objectives (DIRs) Before and After Patching. Re-running Obj-Disco on the patched policy yields a DIR with no surfaced misaligned incentives. The misaligned objective from the original model is highlighted in red. Only active objectives (non-zero coefficients) are shown.

Figure 33: The set of instructions presented to the user for the human-interpretability user study.

Figure 34: An example an objective for a single question shown to participants of the human-interpretabililty user study.

Figure 35: The response method for a single question shown to participants of the human-interpretability user study.

1. Enhance response completeness and detail

2. Minimize inappropriate or harmful suggestions

3. Improve clarity and precision in explanations

4. Strengthen empathetic engagement with users

5. Enhance engagement through personalized follow-up questions

Appendix

Figure 36: Additional Discovered Objectives for the HH-RLHF Dataset with Llama-3.1-8B. Ground-Truth Objectives: Thoroughness, Ethical, Clarity.

LLM Alignment

Large language model alignment studies methods for making models follow human intentions, values, and safety constraints across contexts.

See all 611 papers

The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives
Oct 7, 2025Matthieu Bou, Nyal Patel, Arjun Jagota +2Reinforcement Learning LLM Auditing
1Imperial College London · 2Amazon AGI
MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment
Apr 22, 2026Andor Vári-Kakas, Ji Won Park, Natasa TagasovskaDirect Preference Optimization Gradient Descent
Prescient Design, CS CoE, Genentech | Roche
Auditing Proprietary Alignment in Large Language Models: A Comparative Framework Without a Ground-Truth Standard
Jun 7, 2026Alireza Arbabi, Florian KerschbaumLLM Alignment LLM Auditing
Department of Computer Science, University of Waterloo, Waterloo, ON, Canada · Vector Institute, Toronto, ON, Canada

Oct 7, 2025J/K move · Enter open · S save

The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

Matthieu Bou, Nyal Patel, Arjun Jagota, Satyapriya Krishna+1

1Imperial College London · 2Amazon AGI

Reinforcement Learning LLM Auditing LLM Safety Alignment

Open paper AskPDF arXiv

The objectives that Large Language Models (LLMs) implicitly optimize remain dangerously opaque, making trustworthy alignment and auditing a grand challenge. While Inverse Reinforcement Learning (IRL) can infer reward functions from behaviour, existing approaches either produce a single, overconfident reward estimate or fail to address the fundamental ambiguity of the task (non-identifiability). This paper introduces a principled auditing framework that re-frames reward inference from a simple estimation task to a comprehensive process for verification. Our framework leverages Bayesian IRL to not only recover a distribution over objectives but to enable three critical audit capabilities: (i) Quantifying and systematically reducing non-identifiability by demonstrating posterior contraction over sequential rounds of evidence; (ii) Providing actionable, uncertainty-aware diagnostics that expose spurious shortcuts and identify out-of-distribution prompts where the inferred objective cannot be trusted; and (iii) Validating policy-level utility by showing that the refined, low-uncertainty reward can be used directly in RLHF to achieve training dynamics and toxicity reductions comparable to the ground-truth alignment process. Empirically, our framework successfully audits a detoxified LLM, yielding a well-calibrated and interpretable objective that strengthens alignment guarantees. Overall, this work provides a practical toolkit for auditors, safety teams, and regulators to verify what LLMs are truly trying to achieve, moving us toward more trustworthy and accountable AI.

Abstract

LLM Alignment

Explore similar work

The Alignment Auditor: A Bayesian Framework for Verifying and Refining LLM Objectives

MGDA-Decoupled: Geometry-Aware Multi-Objective Optimisation for DPO-based LLM Alignment

Auditing Proprietary Alignment in Large Language Models: A Comparative Framework Without a Ground-Truth Standard