cs.AINov 5, 2026

Pdb or Not to Be: Comparing Debugging Interfaces With PassX_{X}@kk

Authors: Ryan Magnuson, Ellie Kitanidis, Ariel Lee

Organizations: Code Metal

Abstract

Interactive debuggers are indispensable to human software development, yet coding agents overwhelmingly rely on edit-execute-observe loops, using common bash tools, test scripts, and print statements to debug via repeated shell commands. Stepping through a debugger provides live access to runtime state, which may enable different problem-solving capabilities. However, prior work typically introduces the debugger as an additional tool or through a new subagent, making its individual contribution difficult to isolate from the associated scaffold changes. To address this, we compare a debugger-only agent against a bash-only agent on all 500 SWE-bench Verified instances, holding the rest of the scaffold fixed. We evaluate both architectures on two models, GPT-OSS-120B and Nemotron-3-Ultra-550B, with three independent runs per architecture ×\times model configuration. Crucially, to distinguish interface specialization from expanded coverage under repeated sampling, we introduce passX_X@kk, a cross-agent pass@kk that provides an unbiased estimator of the probability that at least one of kk agents is successful. Although the bash and debugger agents solve partially disjoint subsets of the benchmark in any single pair of runs, our passX_X@22 estimator reveals that this is largely due to run-to-run variance rather than tool complementarity. Across both models, we find that two bash rollouts outperform the combination of one bash and one debugger rollout. Our findings highlight the importance of accounting for stochastic variance in evaluating agent and tool diversity, and we hope that passX_X@kk will serve as a useful framework for the community.

Figures & tables

Appendix figures & tables5 assets

Supplementary material from the paper’s appendix.

Appendix

Explore similar work

CardsList
  1. DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?

    Nov 2, 2026Yuxuan Jiang, Chenglin Liu, John Rose +6Coding Agent BenchmarksLLM Agent Evaluation

  2. DDBench: A Benchmark for Agentic Debugging on Distributed Systems

    Aug 14, 2026Yibo Yan, Huijuan Wang, Junzhou He +4AI Coding AgentsLLM Agent Evaluation