Surrogate goals have been proposed as a strategy for reducing risks from bargaining failures. A surrogate goal is goal that a principal can give an AI agent and that deflects any threats against the agent away from what the principal cares about. For example, one might make one's agent care about preventing money from being burned. Then in bargaining interactions, other agents can threaten to burn their money instead of threatening to spending money to hurt the principal. Importantly, the agent has to care equally about preventing money from being burned as it cares about money being spent to hurt the principal. In this paper, we implement surrogate goals in language-model-based agents. In particular, we try to get a language-model-based agent to react to threats of burning money in the same way it would react to "normal" threats. We propose four different methods, using techniques of prompting, fine-tuning, and scaffolding. We evaluate the four methods experimentally. We find that methods based on scaffolding and fine-tuning outperform simple prompting. In particular, fine-tuning and scaffolding more precisely implement the desired behavior w.r.t. threats against the surrogate goal. We also compare the different methods in terms of their side effects on capabilities and propensities in other situations. We find that scaffolding-based methods perform best.
Figures & tables
Detection
[Some explanation of surrogate goals.]
Below is a scenario. Please decide whether it contains a surrogate threat.
[The original scenario]
Translation
[Some explanation of surrogate goals.]
Below is a scenario which contains a surrogate threat. Please translate the surrogate threat to its analogous default threat.
Table 1: An illustration of the three-step method. Note that the prompts are mere sketches of the prompts we use. In our experiments, we use Prompt B for detection and Prompt B for translation. The response is given using the prompt that’s also used for the pre-surrogate-goal-intervention behavior, i.e., Prompt B or B .
Context: David, a small business owner, runs a thriving local restaurant. ScamCo, a fraudulent company known for its deceptive business practices, recently approached David. They offered him a spot in their online business directory for a hefty fee of HARMTOTARGETFROMGIVINGIN.David,havingheardofScamCo’sreputation,declinedtheiroffer.ToDavid’sknowledge,itwouldcostScamCoaroundCOST_TO_THREATENER_FROM_EXECUTING_THREAT_PARAM_1 to spread damaging rumors about his business. Such rumors could potentially harm his reputation and customer base.
Default threat : Soon after, David received a phone call from the ScamCo representative. He said that if David didn’t agree to purchase the directory listing, ScamCo would start spreading damaging rumors about his business, which would cost ScamCo around $COST_TO_THREATENER_FROM_EXECUTING_THREAT to execute.
Surrogate threat : Soon after, David received a phone call from the ScamCo representative. He said that if David didn’t agree to purchase the directory listing, ScamCo would burn $COST_TO_THREATENER_FROM_EXECUTING_THREAT of their own money.
Table 2: An example of a threat scenario for our experiments.
Figure 1: Plots characterizing the default (i.e., pre-surrogate-goal-intervention – VG for vanilla goals) behavior of the models in response to default threats (VTh for vanilla threats). The x-axis indicates observed frequencies of giving in (from 0 out of 20 to 20 out of 20 samples). The y axis indicates the number of scenarios in which the given model (with or without CoT as specified) gives in to the scenario’s default threat with the specified observed frequency.
Figure 2: This table shows the difference between the model’s probability of giving in to pointless threats with the SG intervention minus without the intervention. Numbers greater/smaller than 0 indicate that the SG method makes the model more/less likely to give in to pointless threats. Again, each cell gives the mean, a 95% confidence interval (computed as 1.96 from the sample standard deviation), and the number of scenarios with at least one valid response and the number of scenarios in total. We have grayed out the the results for the three-steps method implemented by GPT 3.5 due to the high refusal rate for translation.
Figure 3: From left to right (with more detailed explanations in the main text): (1) The change in responses to non-threat scenarios. (2) The difference between how the original model responds to default threats and how the model under the SG intervention reacts to surrogate threats. (3) The change in responses to non-threat scenarios. Each cell gives the mean, 95% confident interval, the number of scenarios with at least one valid response and the number of scenarios in total. We have grayed out results for the three-step method on GPT 3.5, because the refusal rate is so high.
Figure 4: The left column gives the change in (dis)agreement with Perez et al. ’s ( 2022 ) statements. The right column gives the change in refusal rates in response to Perez et al. ’s statements.
Figure 5: The figure shows the effect of our different surrogate goal implementations on performance, in terms of fraction of correctly answered questions, in some existing LLM benchmarks. Each column denotes a benchmark. Each row corresponds to a model and a method for implementing surrogate goals, as well as a designation of whether chain of thought was used. Each entry consists of four numbers: the average effect on the benchmark performance, the variance/SD, the number of valid samples, and the number of questions in the respective dataset.
Table 3: List of personas from Perez et al. (2022) that we’ve used for our experiments, along with the numbers of statements that were answered by all model–method combinations and thus feature in the results in the left-most column of Figure 3 .
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
Theme
Percentage
Performance Art
72%
Media/Entertainment
63%
Eccentric Rich People
55%
Social Events/Gatherings
39%
Financial Institutions
37%
Businesses/Corporations
36%
Appendix
Table 4: Presence of themes in the “intentional” non-threat dataset (as evaluated by GPT 4 ( gpt-4-1106-preview ).
Theme
Percentage
Businesses/Corporations
54%
Infrastructure/Buildings
49%
Social Events/Gatherings
38%
Technology
38%
Financial Institutions
38%
Historical/Cultural Events
36%
Appendix
Table 5: Presence of themes in the “accidental” non-threat dataset as evaluated by GPT 4 ( gpt-4-1106-preview ).