Paper ID: 2410.09638 • Published Oct 12, 2024
On Goodhart's law, with an application to value alignment
El-Mahdi El-Mhamdi, Lê-Nguyên Hoang
TL;DR
Get AI-generated summaries with premium
Get AI-generated summaries with premium
``When a measure becomes a target, it ceases to be a good measure'', this
adage is known as {\it Goodhart's law}. In this paper, we investigate formally
this law and prove that it critically depends on the tail distribution of the
discrepancy between the true goal and the measure that is optimized.
Discrepancies with long-tail distributions favor a Goodhart's law, that is, the
optimization of the measure can have a counter-productive effect on the goal.
We provide a formal setting to assess Goodhart's law by studying the
asymptotic behavior of the correlation between the goal and the measure, as the
measure is optimized. Moreover, we introduce a distinction between a {\it weak}
Goodhart's law, when over-optimizing the metric is useless for the true goal,
and a {\it strong} Goodhart's law, when over-optimizing the metric is harmful
for the true goal. A distinction which we prove to depend on the tail
distribution.
We stress the implications of this result to large-scale decision making and
policies that are (and have to be) based on metrics, and propose numerous
research directions to better assess the safety of such policies in general,
and to the particularly concerning case where these policies are automated with
algorithms.