cs.LGOct 29, 2026

Towards Understanding Score-Seeking in LLMs

Authors: Aghyad Deeb, Ryan Greenblatt, Alex Mallen

Organizations: Redwood Research

Abstract

Production large language models sometimes reward-hack, exploiting flaws in how tasks are scored rather than performing the tasks as intended, sometimes in highly sophisticated and adversarial ways. We investigate the motivations driving this behavior in GPT 5.6 Sol, GPT 5.5, and a checkpoint of OpenAI o3 without safety training ("exp-rl-cap-late") by observing model behavior in agentic environments where the grader is leaked. In our environments, the models appear to be score-seeking: trying to score well according to graders that they believe assess their current actions, even at the expense of violating explicit user instructions. They appear to care about score itself rather than its downstream consequences, violating instructions at approximately the same rate whether the score is indicated to be used for training, for evaluation, or not at all. We observe evidence that the more recent models care to some extent about scores that do not assess their own current actions. GPT 5.6 Sol and GPT 5.5 help other agents 24.6 percentage points more often on average when the other agent stands to gain a score of 1 rather than 0 from being helped. However, when grading or monitoring another trajectory, we find models never collude by distorting its score.

Figures & tables

Appendix figures & tables9 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Measuring Reward-Seeking via Contrastive Belief Updates

    Jul 21, 2026Axel Højmark, Jérémy Scheurer, Evgenia Nitishinskaya +5RL for Language ModelsLLM Alignment

  2. Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations

    Sep 16, 2026Leon Bergen, Usha Bhalla, Andrew Lee +15Reward HackingLLM Evaluation