Code as Explanations: Executable and Verifiable LLM Explanations
Abstract
As large language models (LLMs) increasingly make consequential decisions and solve problems alongside humans, we must understand how models arrive at their conclusions. Understanding how an LLM behaves requires observing it across multiple inputs, yet free-text explanations such as chain-of-thought (CoT) are generated per instance and cannot be easily verified on new inputs. We propose {code explanations to address the mismatch in how LLM explanations are generated versus how they are evaluated: a program explains LLM behavior across a dataset of inputs, and its execution enables automatic verification. We define Consistency metric that measures the fraction of inputs on which the program reproduces the LLM's outputs, and use it to evaluate and select candidate programs. Across three open-source code-capable LLMs and eight tasks, we find that LLMs can produce highly consistent code explanations in several settings. Code explanations are also 6 to 100 shorter than CoT, and better align with model behavior in 9 out of 24 settings. However, consistency is significantly lower on tasks with more complex linguistic components, where generated programs reproduce model outputs no more frequently than an uninformative explanation baseline, and these programs often rely on longer hard-coded patterns. In experiments with Claude Opus 5.5, providing the model with consistency feedback and code execution improves explanation consistency across tasks, but consistency remains significantly lower on more complex tasks. Our findings establish executable code as a novel, verifiable, and compact approach to explaining LLM behavior and highlight verification feedback as a promising direction for improving explanation consistency. The code for this paper is at https://github.com/rajesh-lab/code-as-explanations.
Figures & tables
| Category | Task | Description |
| Structured | Web of lies | Determine if a person lies based on a set of statements |
| logic | Navigate | Track movements on grid to decide if agent returns to the start |
| Logic + simple | Hyperbaton | Identify the sentence with the correct adjective order |
| language | Object counting | Count the number of objects satisfying a given description |
| Logic + com- | Formal fallacies | Determine whether an argument is deductively valid or invalid |
| plex language | Disambiguation QA | Identify the antecedent of a pronoun |
| Category | Task | Code snippet |
| Structured logic | Navigate | if "forward" in words: if direction == 0: y_pos += steps |
| Logic + simple language | Hyperbaton | def categorize_adjective(adj): if adj in ["awful", "repulsive", "obnoxious", "terrible", "good", "nice", "lovely"]: return "opinion" |
| Logic + complex language | Formal fallacies | premises = [premise.replace(’ is ’, ’is’).replace(’ or ’, ’or’).replace(’ and ’, ’and’).replace(’ every ’, ’every_’).replace(’ whoever ’, ’whoever_’).replace(’ being a ’, ’being_a_’).replace(’ an ’, ’an_’).replace(’ a ’, ’a_’) for premise in premises] premises = [premise.replace(’not ’, ’not_’) for premise in premises] |
| Linguistic reasoning | Snarks | if any(indicator in statement for indicator in [’ARE U INSANE’, ’Have my down vote’, ’naive’, ’terrible support’, ’pragmatic’, ’more expensive’, ’less fun’, ’liability’, ’enjoyable’, ’best’, ’reddit’, ’fun’, ’sexist’, ’win you points’, ’lose you points’, ’True’, ’rational’, ’threaten’, ’pleaded’, ’highest’, ’lowest’, ’rich’, ’loans’, ’suspicious’, ’legit’, ’perfect’]): sarcastic_option = label |
Appendix figures & tables6 assets
Supplementary material from the paper’s appendix.
Appendix
| Role | Content |
| System | You are given a task along with N example input-output pairs that you previously answered. Your task is to explain your underlying algorithm. The function must correctly handle all tokens that appear in the inputs. Do not ignore any part of the input string. |
| User | Task description: T; Q: x_1, A: y_1; ... ; Q: x_N, A: y_N |
| Write a Python function that reproduces the outputs from the inputs. Think about how to write an algorithm step by step. Make sure that your code does not raise any error when running with the input strings. Do not show example usage. Output the final Python code in the following format: |
LLM Interpretability
Large language model (LLM) interpretability studies how internal representations and computations produce outputs, and how they can be explained or controlled.
See all 837 papers