cs.LGNov 10, 2026

Code as Explanations: Executable and Verifiable LLM Explanations

Authors: Nhi Nguyen, Shauli Ravfogel, Rajesh Ranganath

Abstract

As large language models (LLMs) increasingly make consequential decisions and solve problems alongside humans, we must understand how models arrive at their conclusions. Understanding how an LLM behaves requires observing it across multiple inputs, yet free-text explanations such as chain-of-thought (CoT) are generated per instance and cannot be easily verified on new inputs. We propose {code explanations to address the mismatch in how LLM explanations are generated versus how they are evaluated: a program explains LLM behavior across a dataset of inputs, and its execution enables automatic verification. We define Consistency metric that measures the fraction of inputs on which the program reproduces the LLM's outputs, and use it to evaluate and select candidate programs. Across three open-source code-capable LLMs and eight tasks, we find that LLMs can produce highly consistent code explanations in several settings. Code explanations are also 6×\times to 100×\times shorter than CoT, and better align with model behavior in 9 out of 24 settings. However, consistency is significantly lower on tasks with more complex linguistic components, where generated programs reproduce model outputs no more frequently than an uninformative explanation baseline, and these programs often rely on longer hard-coded patterns. In experiments with Claude Opus 5.5, providing the model with consistency feedback and code execution improves explanation consistency across tasks, but consistency remains significantly lower on more complex tasks. Our findings establish executable code as a novel, verifiable, and compact approach to explaining LLM behavior and highlight verification feedback as a promising direction for improving explanation consistency. The code for this paper is at https://github.com/rajesh-lab/code-as-explanations.

Figures & tables

Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. Do Language Models Trust Their Own Justifications? A Study on Functional Consistency

    Oct 23, 2026Alisson Rosa Pereira, Bruno Iochins GrisciFaithfulness of Language Model ExplanationsLLM Evaluation

  2. Enhancing Assessment of Self-Consistency in LLM Explanations using Perturbation Strength

    Sep 25, 2026Phuong Q. Le, Kemal Kurniawan, Jey Han LauSelf-Consistency in Language ModelsLanguage Model Robustness

  3. An Empirical Study of Counterfactual Self-Explanations in LLMs

    Sep 15, 2026Giannis Kalyvas, Giorgos Filandrianos, Orfeas Menis Mastromichalakis +2Counterfactual ExplanationsFaithfulness of Language Model Explanations