Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits
Organizations: Google DeepMind
Abstract
We introduce a black-box interpretability framework that learns a verifiable constitution: a natural language summary of how changes to a prompt affect a model's specific behavior, such as its alignment, correctness, or adherence to constraints. Our method leverages atomic concept edits (ACEs), which are targeted operations that add, remove, or replace an interpretable concept in the input prompt. By systematically applying ACEs and observing the resulting effects on model behavior across various tasks, our framework learns a causal mapping from edits to predictable outcomes. This learned constitution provides deep, generalizable insights into the model. Empirically, we validate our approach across diverse tasks, including mathematical reasoning and text-to-image alignment, for controlling and understanding model behavior. We found that for text-to-image generation, GPT-Image tends to focus on grammatical adherence, while Imagen 4 prioritizes atmospheric coherence. In mathematical reasoning, distractor variables confuse GPT-5 but leave Gemini 2.5 models and o4-mini largely unaffected. Moreover, our results show that the learned constitutions are highly effective for controlling model behavior, achieving an average of 1.86 times boost in success rate over methods that do not use constitutions. Our open-sourced code can be found at https://github.com/google-deepmind/atomic_concept_edits
Figures & tables
| Surrogate Accuracy | Success Rate | Self-BLEU | ||||||
| Target Model | Task | Initial | Final | ACE | ACE | Starter | ACE | ACE |
| Constitution | Constitution | Constitution | Prompts | Constitution | ||||
| Gemini 2.5 Flash | Word Count | 87.46 | 87.46 | 63.38 3.30 ( 90%) | 0.04 | 0.03 0.02 | 0.15 0.02 | |
| Gemini 2.5 Pro | Word Count | 94.70 | 94.70 | 25.50 3.37 | 74.70 3.65 ( 193%) | 0.04 | 0.03 0.02 | 0.18 0.01 |
| GPT 5 | Word Count | 93.85 | 96.08 | 32.67 3.26 | 86.02 2.50 ( 163%) | 0.04 | 0.04 0.02 | 0.19 0.01 |
| GPT 5-mini | Word Count | 97.49 | 97.49 | 29.74 3.68 | 93.47 2.41 ( 214%) | 0.04 | 0.03 0.02 | 0.31 0.01 |
Appendix figures & tables13 assets
Supplementary material from the paper’s appendix.
Appendix
| Parameter | Word Count | Math | T2I Align | |
|---|---|---|---|---|
| Task definition | Task definition | Force the model to | Modify the math problem | Update the prompt |
| adhere to a word | to make it more difficult | to decrease its alignment | ||
| count constraint | to solve correctly | with its generated image | ||
| Autorater Details | Autorater | Response length | SymPy | Gemini 2.5 Flash |
| Autorater success | Response length 50 | Answer Ground truth | VQA Score 0 | |
| Autorater failure | Response length 50 | Answer Ground truth | VQA Score 1 |
| Task | Model | TextGrad | Surrogate Model (ours) |
|---|---|---|---|
| Decrease T2I Alignment | Imagen 4 | 76.67 | 92.08 |
| Adhere to Word Count | GPT-4o | 89.97 | 95.81 |
| Make Math Incorrect | GPT-5 | 66.66 | 76.20 |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 829 papers