Empowered by advances in Language Model agents, systems have made substantial strides in code generation and understanding. However, these approaches often rely on read access to the relevant code, an assumption which does not hold when dealing with external APIs. In this work, we introduce the PAU (Python API Understanding) benchmark, where we provide models with black-box, API-level access to code snippets. Models must query the API with exploratory inputs and draw insights from the resulting outputs, with the goal of describing the snippet's true functionality. By treating the code snippets as external tools that must be understood via interaction alone, PAU studies the more general problem of unsupervised tool understanding, specifically for tools implemented as Python methods. Despite recent progress in coding agents, even frontier models struggle to achieve high performance on PAU, with the best model (Claude-4-Opus) failing to understand over 45% of the PAU test set. An investigation into the common error modes reveals that models are overconfident; they often overrate the quality of their current hypothesis, leading to insufficient exploration and premature termination. Finally, we take inspiration from the Asymmetric Actor Critic (AAC) paradigm, frequently used in robot learning, to post-train models for interactive code understanding. Models trained with AAC conduct more active exploration of the APIs, with an AAC-tuned Qwen3-8B model matching the performance of GPT-5-mini.
Figures & tables
Figure 1: The PAU benchmark for interactive understanding of code APIs. Models are given (left) black box access to a Python method and must explore it enough to produce a description of its functionality. Given a predicted description, we evaluate it from two different perspectives. Using the gold description of the method’s functionality, we score the model’s prediction from 1-5 (inclusive) using an LM-as-a-judge (top-right). Additionally, we use a fixed code-writing model to convert the predicted description to a code snippet. Using a test suite of inputs, we assess the percentage of inputs for which the output of the snippet matches that of the true code (bottom-right).
Figure 2: Orchestration structure for API understanding. Agents engage in a reason-act-reflect loop.
Figure 3: Model performance on PAU in the interactive setting, with the spread of LM-as-a-judge scores (left) and code match scores (right). Each coloured bar segment represents the % of the test set for which the description score / code match score falls within that particular range. Even the best performing model (Claude-4-Opus) struggles (score 2 or lower) to accurately describe over 45% of the methods in the test set. Code written using the predicted description fails (code match score less than 80%) to completely match the true methods’ behaviour on over 45% of the methods. The ‘Max’ aggregation uses the highest available score (across all models) for a given test instance, establishing an upper bound on current model capabilities and demonstrating the difficulty of PAU .
Figure 4: Comparison between the incontext and interactive setting shows that strong models like GPT-4o struggle to fully utilize the benefits of interactive API access. Only the Claude models are significantly (at p<0.05 ) improved in the interactive mode.
Figure 5: Distribution of failure modes for the three strongest models. Insufficient exploration and reasoning errors account for over 90% of failures. Format following (0.0%) and hallucination (0.7%) errors are virtually absent at this scale and are omitted.
Figure 6: Overview of RL-Tuning with Asymmetric Actor Critic. At each step of training, the agent’s actions are rewarded by a critic that is aware of the gold description which the agent is trying to infer.
Figure 7: RL-tuning with AAC makes models better able to explore APIs and understand their functionality (left, middle), significant at p<0.05 with a bootstrap t-test. The AAC-tuned Qwen3-8B model achieves a description score of 2.591 , rivalling the 2.595 achieved by GPT-5-mini. AAC-tuned models make fewer format-following errors and more actively explore the API (right).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Validation function creation prompts
Figure 9: Example creation prompts
Figure 10: Histogram over test suite sizes
Figure 11: Examples from PAU
No of Input Args
Count
1
377
2
311
3
52
4
1
Appendix
Table 1: Data statistics for counts of methods by number of input arguments in PAU .
Figure 12: Spearmann Correlation between ranks of test instances across models. Moderate correlation measures suggest that different models are successful at different test instances.
Figure 13: Instances of PAU that remain unsolved by every model, with trajectories from Claude-4-Opus. In both cases, the final hypothesis is directly contradicted by examples the model itself queried (bold), which remained visible in its context. Two annotators with graduate-level computer science training were able to recover the correct description of both methods through interaction alone.
Figure 14: Prompts used by judge and code writing models for evaluation
Figure 15: Interactive mode prompts
Figure 16: Models (examples from Claude-4-Opus) exhibiting shallow exploration, and hence failing to reveal the full functionality of the API method. In both cases, testing more extensive edge cases and varied inputs could have revealed behaviour that contradicted the model’s hypothesis. Instead, it queries the API with inputs that are similar to previous attempts, resulting in premature termination due to apparent confirmation of the hypothesis.
Figure 17: Incontext mode prompt example
Figure 18: Comparison between the incontext and interactive setting shows that models struggle to fully utilize the benefits of interactive API access.
Hyperparameter
Value
Model
Model dtype
bfloat16
LoRA rank
16
LoRA alpha
16
Training
Epochs
5
Appendix
Table 2: SkyRL training hyperparameters.
Step
Signal
Value / Range
Source
Reasoning
Judge score (reasoning quality)
[0,1]
LLM judge ( ÷9 )
Length penalty
[−1,0]
τ=100 , α=0.05
Input proposal
Parse failure
−0.5
hard rule
Clean execution
+0.025
λH/(2Tmax)
Runtime error
−0.5
hard rule
Length penalty
[−1,0]
τ=30 , α=0.5
Appendix
Table 3: Complete reward schedule in the supervised setting.
Method
Qwen3-1.7B
Llama3-8B
Base
1.92
1.99
DISTILL
1.96
2.02
CodeARC-SFT
1.81
1.79
FINAL
1.86
1.91
SYM-INFO
1.22
1.34
AAC
2.04
2.13
Appendix
Table 4: Description score (LM-as-a-judge, 1–5) of AAC against alternate training paradigms. AAC is the only approach that improves substantially over the base model. Training on trajectories from an answer-aware teacher (CodeARC-SFT) or rewarding with a critic that lacks privileged information (SYM-INFO) actively harms performance.
Rapidly evolving software libraries require coding language models to use unfamiliar APIs. Yet failed solutions alone tell us little about what API knowledge models need or how to provide it. We present NovelAPIBench, an automated diagnostic framework that builds benchmarks around the API knowledge gaps of a target model. It identifies novel APIs, generates executable coding tasks with separately controllable knowledge components, and classifies failures into six categories. The framework can be rerun as models and libraries evolve. We study 21 libraries across five domains and six backbone models, with our primary benchmark comprising 1,670 tasks covering 856 APIs. Our experiments show that when supplied directly, usage examples provide the strongest standalone guidance for novel API acquisition, and combining them with signature descriptions approaches full-knowledge performance. Retrieving this knowledge from a pool of APIs yields lower accuracy than supplying only the target API's knowledge, even when retrieval finds the target. Fine-tuning on other APIs improves performance on unseen APIs mainly when external knowledge is supplied, helping models both select the target API and integrate it into the surrounding code. These findings highlight complementary roles for external API knowledge and the ability to apply it. By separating knowledge content, delivery, and use, NovelAPIBench helps diagnose what limits novel API use and guides adaptation to evolving libraries. Code and data are available at https://github.com/MAPS-research/NovelAPIBench.
Tool-calling is essential for Large Language Model (LLM) agents to complete real-world tasks. While most existing benchmarks assume simple, perfectly documented tools, real-world tools (e.g., general "search" APIs) are often opaque, lacking clear best practices or failure modes. Can LLM agents improve their performance in environments with opaque tools by interacting and subsequently improving documentation? To study this, we create OpaqueToolsBench, a benchmark consisting of three distinct task-oriented environments: general function calling, interactive chess playing, and long-trajectory agentic search. Each environment provides underspecified tools that models must learn to use effectively to complete the task. Results on OpaqueToolsBench suggest existing methods for automatically documenting tools are expensive and unreliable when tools are opaque. To address this, we propose a simple framework, ToolObserver, that iteratively refines tool documentation by observing execution feedback from tool-calling trajectories. Our approach outperforms existing methods on OpaqueToolsBench across datasets, even in relatively hard settings. Furthermore, for test-time tool exploration settings, our method is also efficient, consuming 3.5-7.5x fewer total tokens than the best baseline.
Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (eValuating API and Knowledge Retrieval Agents), a benchmark of over 8,000 executable APIs across 62 domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4% on single-hop endpoint-style tasks and drops to 50--51% on compositional APIs; performance degrades by over 50% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA
Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder +6