cs.CLOct 30, 2026

Circuit Oracle: Automating Attribution Graph Analysis via Natural-Language Queries

Authors: Shariar Kabir, Hong Kiat Tan, Swastik Agrawal, Sai V R Chereddy, Sriram Balasubramanian

Organizations: Independent Researcher · University of California, Los Angeles, CA, USA · The University of Maryland, College Park, MD, USA

Abstract

Attribution graphs, an emerging tool in mechanistic interpretability, use transcoders to decompose language model computations into sparse interpretable features connected by causal edges. However, turning a graph into a safety-relevant insight requires hours of manual analysis by experts. We introduce Circuit Oracle, an off-the-shelf LLM equipped with a multi-agent harness and task-specific authored skills, which answers natural-language questions about a target model by reading the attribution graph through tool calls. We evaluate it on three safety-relevant proxy tasks: detecting spurious features in probe circuits, eliciting hidden knowledge from taboo-finetuned models, and suppression jailbreaking via feature-level interventions. Beyond task scores, the oracle produces pruned subgraphs and context-specific feature explanations that we assess qualitatively. Results are mixed. On spurious-feature detection, Circuit Oracle reaches 78.2 ±\pm 2.4% accuracy, outperforming single-layer feature-ranking baselines. On secret elicitation, trained self-explainers like Activation Oracles remain the stronger baseline. On refusal suppression over 50 prompts, its single committed intervention is slightly below a diff-in-means baseline, and only exceeds it when allowed the best of five attempts. Results are stable across five runs and across model choices for the agents. We also release the pruned subgraphs and context-specific feature readings that revise existing global labels, and invite future work to develop evaluations that can measure their usefulness directly.

Figures & tables

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. ADAG: Automatically Describing Attribution Graphs

    Apr 8, 2026Aryaman Arora, Zhengxuan Wu, Jacob Steinhardt +1Mechanistic InterpretabilityCircuits in Language Models

  2. LLMs Can Annotate Attribution Graphs

    Jul 28, 2026Ameen Patel, Max Zhang, Nathan HuLLM InterpretabilityLLM-Assisted Annotation

  3. SafeSeek: Universal Attribution of Safety Circuits in Language Models

    Mar 24, 2026Miao Yu, Siyuan Fu, Moayad Aloqaily +6LLM InterpretabilityCircuits in Language Models