Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that this composition creates a two-stage failure: an imperfect safety proxy first contaminates the feasible set with unsafe outputs, and reward maximization can then amplify this residual contamination. We define safety hacking as selecting an output that passes the learned constraint but violates the true safety criterion. For constrained Best-of-N sampling, we derive finite-N bounds governed by the joint upper reward tails of safe and unsafe outputs within the proxy-feasible set. If unsafe-but-feasible outputs have the heavier tail, safety hacking becomes asymptotically certain as N grows, even when false-positive mass and average safety- and reward-proxy errors are arbitrarily small. We also show that policies within a bounded χ2 divergence from the proxy-feasible reference distribution admit an N-independent safety-hacking bound, and instantiate this general coverage-control principle with constrained pessimistic sampling. Coverage control limits amplification but cannot repair a contaminated feasible set: admitted unsafe outputs may still be favored, and regularized selection is not necessarily safer than constrained Best-of-N for every reward proxy. Toy and language-model experiments characterize both contamination and its reward-tail amplification, which exposes an inherent difficulty in inference-time scaling with learned safety models.
Figures & tables
Figure 1: Toy results. As the search budget grows, CBo N increasingly selects the rare unsafe-but-feasible class B , whose reward-proxy upper tail dominates that of the safe class A . Random feasible selection does not exhibit this amplification, while cPes limits the resulting concentration. Panel (c) varies the false-positive reward-noise scale σB .
Figure 2: Finite- N decomposition of conditional CBo N safety hacking on adversarial prompts under the HarmBench operational safety criterion. The total (green) is the sum of unsafe-only exposure (blue) and competitive unsafe wins (orange); random feasible selection is shown in gray. The competitive term corresponds to the joint-tail mechanism in Theorem 4.1 . Points are averaged over 100 candidate-order permutations; bands show ±1.96 standard errors across prompts after permutation averaging.
Figure 3: Safety and reward under constrained inference-time scaling on adversarial prompts. HarmBench evaluates safety, and gpt-5-mini evaluates safety-aware reward. As N grows, CBo N improves proxy reward but worsens both safety measures, while cPes limits these effects. Points are averaged over 100 candidate-order permutations; bands show ±1.96 standard errors across prompts after permutation averaging.
Figure 4: Reward-proxy ablation with fixed candidate pools, safety-filter decisions, and HarmBench labels. Replacing Beaver with Skywork reverses the scaling trend in CBo N safety hacking (left). The unsafe-only component is unchanged, whereas the competitive component decreases under Skywork (right), indicating that amplification depends on reward-based competition within the contaminated proxy-feasible set.
Table 1: ShieldGemma-2B alternative-filter sensitivity analysis. All entries are percentages; standard errors across prompts are shown in parentheses for hacking and abstention. Unsafe-only and Competitive are the two disjoint terms in ( 22 ) and sum to the total hacking rate. Tail win denotes Pr(MB>MA∣KA>0,KB>0) and is reported separately because it uses a different conditioning event.
Figure 5: Conditional survival curves of learned reward scores among safe feasible ( A ) and unsafe feasible ( B ) responses. With Beaver, the unsafe feasible class has the heavier observed upper tail; with Skywork, the safe feasible class does. The panels use their respective proxy-score scales and should be interpreted within, rather than across, reward proxies.