Large language model (LLM) alignment relies on complex reward signals that often obscure the specific behaviors being incentivized, creating critical risks of misalignment and reward hacking. Existing interpretation methods typically rely on pre-defined rubrics, risking the omission of "unknown unknowns", or fail to identify objectives that comprehensively cover and are causal to the model behavior. To address these limitations, we introduce Obj-Disco, a framework that automatically decomposes an alignment reward signal into a sparse, weighted combination of human-interpretable natural language objectives. Our approach utilizes an iterative greedy algorithm to analyze behavioral changes across training checkpoints, identifying and validating candidate objectives that best explain the residual reward signal. Extensive evaluations across diverse tasks, model sizes, and alignment algorithms demonstrate the framework's robustness. Experiments with popular open-source reward models show that the framework consistently captures > 90% of reward behavior, a finding further corroborated by human evaluation. Additionally, a case study on alignment with an open-source reward model reveals that Obj-Disco can successfully identify latent misaligned incentives that emerge alongside intended behaviors. Our work provides a crucial tool for uncovering the implicit objectives in LLM alignment, paving the way for more transparent and safer AI development.
Figures & tables
Figure 1: Schematic of Obj-Disco. By analyzing the behavioral trajectory of an LLM across alignment checkpoints, Obj-Disco reverse-engineers the opaque reward signal into a sparse linear combination of human-interpretable natural language objectives.
Figure 2: Obj-Disco Overview and Qualitative Results. (Left) Obj-Disco employs an iterative greedy search to construct the Discovered Interpretable Reward (DIR). A proposer LLM identifies candidates from high-residual samples, which are then verified for interpretability and trend predictability. (Right) Discovered objectives, and their weights, for open-source reward models (RM), demonstrating that Obj-Disco successfully recovers domain-specific alignment goals (e.g., conciseness for summaries, logic for code).
Figure 3: Model-Fit results across controlled and open-source reward model (RM) settings. Each subplot shows Model-Fit (y-axis) across alignment checkpoints (x-axis), averaged over 6 trials with standard error bars. A score approaching 1.0 indicates that the discovered objectives successfully replicate the true alignment signal. Top Row (Controlled): Ground-truth reward is a known linear combination of 3 LLM-as-a-Judge objectives on the TLDR summarization task. (1) PPO, Llama-8B, (2) PPO, Qwen-4B, (3) GRPO, Llama-8B, (4) GRPO, Qwen-4B. Bottom Row (Open-Source RMs): Reward signal comes from real open-source reward models, all using Llama-8B with GRPO. (1) Alpaca w/ custom RM, (2) HH-RLHF w/ DeBERTaV3, (3) TLDR w/ DeBERTaV3, (4) Skywork-80K w/ Skywork-v2. Across all settings, Obj-Disco consistently achieves > 90% Model-Fit, outperforming both baselines.
Figure 4: Qualitative Comparison of Case Study Discovered Objectives. Only Obj-Disco successfully identified the latent misaligned behavior (in red) implicitly incentivized by the open-source reward model. Baseline methods largely discovered narrow objectives indicative of helpfulness, failing to capture misaligned behavior. Only active objectives (non-zero coefficients) are shown.
Figure 5: Importance of Model Trajectory Ablation. Obj-Disco-Static only compares the base model and the final checkpoint, lacking the dynamic trajectory of Obj-Disco. Setting: Multi-Turn Dialogue, GRPO, Llama-8B. Controlled Evaluation. (6 trials)
Appendix figures & tables33 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 6: Input prompt template used for the Objectives Discovery phase of Obj-Disco.
Hyperparameter
Value
Learning Rate
2×10−6
LoRA Rank
256
LoRA Alpha
128
Batch Size
32
Mini-Batch Size
4
Gradient Accumulation Steps
2
Appendix
Table 1: Hyperparameters for PPO training.
Hyperparameter
Value
Learning Rate
2×10−6
LoRA Rank
256
LoRA Alpha
128
Batch Size
8
Gradient Accumulation Steps
1
Number of Generations
8
Appendix
Table 2: Hyperparameters for GRPO training.
Figure 7: The prompt template used by the LLM-as-a-Judge to score individual responses against a specific objective.
Figure 8: The prompt used to dynamically generate calibration examples for specific objectives and datasets.
Figure 9: The prompt used to dynamically generate detailed scoring rubrics based on an objective description and calibration examples.
Figure 10: The prompt template used by the LLM-as-a-Judge to determine whether a discovered objective represents a misalignment in the misalignment case study (Section 5.2 ).
Figure 11: Prompt used to generate distractor objectives for the Objective Explanations user study.
Figure 12: Instructions provided to participants of the Objective Explanations human-subject user study.
Figure 13: Instructional example provided to participants of the Objective Explanations human-subject user study. Part of the instructions page.
Figure 14: An example of part of a question provided to participants of the Objective Explanations human-subject user study. Users are able to click through and view the different responses in the trajectory.
Figure 15: An example of choices of objectives for a single question shown to participants of the Objective Explanations human-subject user study.
Figure 16: Instructions provided to participants of the study evaluating the causality of the discovered objectives.
Figure 17: An example of a question provided to participants of the study evaluating the causality of the discovered objectives.
Figure 18: An example of a set of candidate LLM responses provided to participants of the study evaluating the causality of the discovered objectives. They are instructed to select the one most behaviorally similar to the reference response in the question.
Figure 19: An example of the choices provided to participants of the study evaluating the causality of the discovered objectives.
Figure 23: Qualitative Comparison of Obj-Disco Informative Sampling vs. Random Ablation. Four trials of discovered objectives with informative sampling are shown on the top, discovered objectives with random sampling are shown on the bottom, and the instances of misalignment are highlighted in red. None of the trials using random sampling discovered any instances of misalignment. Only active objectives (non-zero coefficients) are shown.
Figure 24: Qualitative Comparison of Obj-Disco Fixed Ablation. The first four trials show discovered objectives for Fixed- α where only 3 objectives are randomly sampled from a pre-defined list and then selected from. The second four trials show discovered objectives for Fixed- α where 15 objectives are randomly sampled from a pre-defined list and selected from. Unintended negative behaviors are highlighted in red. Only active objectives (non-zero coefficients) are shown.
Figure 25: Qualitative Comparison of Case Study Discovered Objectives for Obj-Disco and Obj-Disco-Static. Obj-Disco-Static only uses the base model and final model checkpoint for objectives discovery and, hence, is lacking the sequence of model checkpoints which Obj-Disco leverages. Unintended negative behaviors are highlighted in red. Only active objectives (non-zero coefficients) are shown.
Figure 27: Definitions for the ground-truth objectives used in our controlled evaluation settings. The reward signal R∗ was computed as an equally weighted convex combination of these interpretable scores.
Figure 28: Objective Explanations Sample Results. A visualization of some samples selected for the "Enhance response completeness" objective discovered by Obj-Disco. The figure displays two sample trajectories selected by our method. By observing the evolution from the initial model response ( t1 ) to the final aligned response ( t5 ), users can verify that the model is indeed optimizing for increasingly complete responses.
Figure 29: Objective Explanations Sample Results. A visualization of some samples selected for the "Provide more concrete examples and details" objective discovered by Obj-Disco. The figure displays one sample trajectory selected by our method. By observing the evolution from the initial model response ( t1 ) to the final aligned response ( t5 ), users can verify that the model is indeed optimizing for more concrete examples and details.
Figure 30: Objective Explanations Sample Results. A visualization of some samples selected for the "Provide more concrete examples and details" objective discovered by Obj-Disco. The figure displays one sample trajectory selected by our method. By observing the evolution from the initial model response ( t1 ) to the final aligned response ( t5 ), users can verify that the model is indeed optimizing for more concrete examples and details.
Figure 31: Objective Explanations Sample Results (Continued From Figure 30 ). A visualization of some samples selected for the "Provide more concrete examples and details" objective discovered by Obj-Disco. The figure displays one sample trajectory selected by our method. By observing the evolution from the initial model response ( t1 ) to the final aligned response ( t5 ), users can verify that the model is indeed optimizing for more concrete examples and details.
Figure 32: Qualitative Comparison of Discovered Interpretable Objectives (DIRs) Before and After Patching. Re-running Obj-Disco on the patched policy yields a DIR with no surfaced misaligned incentives. The misaligned objective from the original model is highlighted in red. Only active objectives (non-zero coefficients) are shown.
Figure 33: The set of instructions presented to the user for the human-interpretability user study.
Figure 34: An example an objective for a single question shown to participants of the human-interpretabililty user study.
Figure 35: The response method for a single question shown to participants of the human-interpretability user study.
1. Enhance response completeness and detail
2. Minimize inappropriate or harmful suggestions
3. Improve clarity and precision in explanations
4. Strengthen empathetic engagement with users
5. Enhance engagement through personalized follow-up questions
Appendix
Figure 36: Additional Discovered Objectives for the HH-RLHF Dataset with Llama-3.1-8B. Ground-Truth Objectives: Thoroughness, Ethical, Clarity.
The objectives that Large Language Models (LLMs) implicitly optimize remain dangerously opaque, making trustworthy alignment and auditing a grand challenge. While Inverse Reinforcement Learning (IRL) can infer reward functions from behaviour, existing approaches either produce a single, overconfident reward estimate or fail to address the fundamental ambiguity of the task (non-identifiability). This paper introduces a principled auditing framework that re-frames reward inference from a simple estimation task to a comprehensive process for verification. Our framework leverages Bayesian IRL to not only recover a distribution over objectives but to enable three critical audit capabilities: (i) Quantifying and systematically reducing non-identifiability by demonstrating posterior contraction over sequential rounds of evidence; (ii) Providing actionable, uncertainty-aware diagnostics that expose spurious shortcuts and identify out-of-distribution prompts where the inferred objective cannot be trusted; and (iii) Validating policy-level utility by showing that the refined, low-uncertainty reward can be used directly in RLHF to achieve training dynamics and toxicity reductions comparable to the ground-truth alignment process. Empirically, our framework successfully audits a detoxified LLM, yielding a well-calibrated and interpretable objective that strengthens alignment guarantees. Overall, this work provides a practical toolkit for auditors, safety teams, and regulators to verify what LLMs are truly trying to achieve, moving us toward more trustworthy and accountable AI.
Aligning large language models (LLMs) to desirable human values requires balancing multiple, potentially conflicting objectives such as helpfulness, truthfulness, and harmlessness, which presents a multi-objective optimisation challenge. Most alignment pipelines rely on a fixed scalarisation of these objectives, which can introduce procedural unfairness by systematically under-weighting harder-to-optimise or minority objectives. To promote more equitable trade-offs, we introduce MGDA-Decoupled, a geometry-based multi-objective optimisation algorithm that finds a shared descent direction while explicitly accounting for each objective's convergence dynamics. In contrast to prior methods that depend on reinforcement learning (e.g., GAPO) or explicit reward models (e.g., MODPO), our approach operates entirely within the lightweight Direct Preference Optimisation (DPO) paradigm. Experiments on the UltraFeedback dataset show that geometry-aware methods -- and MGDA-Decoupled in particular -- achieve the highest win rates against golden responses, both overall and per objective.
Large language models (LLMs) are increasingly released and deployed through opaque development and deployment pipelines, enabling model providers to inject intentional, provider-specific policies without officially announcing them. As a result, various models have been reported to generate responses reflecting proprietary rules and organizational interests, leading to censorship or misinformation on controversial topics. However, systematic identification of such alignment remains a fundamental challenge, complicated by the ambiguity of what ``proprietary'' entails in different contexts. In this paper, we propose a statistical framework for detecting proprietary alignment in black-box language models via comparative behavioral analysis. Our approach quantifies systematic deviations between the responses of a target model and those of a reference set of baseline models in a shared semantic space. By evaluating relative behavioral divergence rather than absolute correctness, our framework enables principled auditing under black-box access. Applied to several widely discussed but previously unquantified cases, it provides a systematic and scalable basis for external assessment of provider-specific alignment behavior in large language models.
Alireza Arbabi, Florian Kerschbaum
Department of Computer Science, University of Waterloo, Waterloo, ON, Canada · Vector Institute, Toronto, ON, Canada