Empowered by advances in Language Model agents, systems have made substantial strides in code generation and understanding. However, these approaches often rely on read access to the relevant code, an assumption which does not hold when dealing with external APIs. In this work, we introduce the PAU (Python API Understanding) benchmark, where we provide models with black-box, API-level access to code snippets. Models must query the API with exploratory inputs and draw insights from the resulting outputs, with the goal of describing the snippet's true functionality. By treating the code snippets as external tools that must be understood via interaction alone, PAU studies the more general problem of unsupervised tool understanding, specifically for tools implemented as Python methods. Despite recent progress in coding agents, even frontier models struggle to achieve high performance on PAU, with the best model (Claude-4-Opus) failing to understand over 45% of the PAU test set. An investigation into the common error modes reveals that models are overconfident; they often overrate the quality of their current hypothesis, leading to insufficient exploration and premature termination. Finally, we take inspiration from the Asymmetric Actor Critic (AAC) paradigm, frequently used in robot learning, to post-train models for interactive code understanding. Models trained with AAC conduct more active exploration of the APIs, with an AAC-tuned Qwen3-8B model matching the performance of GPT-5-mini.
Figures & tables
Figure 1: The PAU benchmark for interactive understanding of code APIs. Models are given (left) black box access to a Python method and must explore it enough to produce a description of its functionality. Given a predicted description, we evaluate it from two different perspectives. Using the gold description of the method’s functionality, we score the model’s prediction from 1-5 (inclusive) using an LM-as-a-judge (top-right). Additionally, we use a fixed code-writing model to convert the predicted description to a code snippet. Using a test suite of inputs, we assess the percentage of inputs for which the output of the snippet matches that of the true code (bottom-right).
Figure 2: Orchestration structure for API understanding. Agents engage in a reason-act-reflect loop.
Figure 3: Model performance on PAU in the interactive setting, with the spread of LM-as-a-judge scores (left) and code match scores (right). Each coloured bar segment represents the % of the test set for which the description score / code match score falls within that particular range. Even the best performing model (Claude-4-Opus) struggles (score 2 or lower) to accurately describe over 45% of the methods in the test set. Code written using the predicted description fails (code match score less than 80%) to completely match the true methods’ behaviour on over 45% of the methods. The ‘Max’ aggregation uses the highest available score (across all models) for a given test instance, establishing an upper bound on current model capabilities and demonstrating the difficulty of PAU .
Figure 4: Comparison between the incontext and interactive setting shows that strong models like GPT-4o struggle to fully utilize the benefits of interactive API access. Only the Claude models are significantly (at p<0.05 ) improved in the interactive mode.
Figure 5: Distribution of failure modes for the three strongest models. Insufficient exploration and reasoning errors account for over 90% of failures. Format following (0.0%) and hallucination (0.7%) errors are virtually absent at this scale and are omitted.
Figure 6: Overview of RL-Tuning with Asymmetric Actor Critic. At each step of training, the agent’s actions are rewarded by a critic that is aware of the gold description which the agent is trying to infer.
Figure 7: RL-tuning with AAC makes models better able to explore APIs and understand their functionality (left, middle), significant at p<0.05 with a bootstrap t-test. The AAC-tuned Qwen3-8B model achieves a description score of 2.591 , rivalling the 2.595 achieved by GPT-5-mini. AAC-tuned models make fewer format-following errors and more actively explore the API (right).
Appendix figures & tables15 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 8: Validation function creation prompts
Figure 9: Example creation prompts
Figure 10: Histogram over test suite sizes
Figure 11: Examples from PAU
No of Input Args
Count
1
377
2
311
3
52
4
1
Appendix
Table 1: Data statistics for counts of methods by number of input arguments in PAU .
Figure 12: Spearmann Correlation between ranks of test instances across models. Moderate correlation measures suggest that different models are successful at different test instances.
Figure 13: Instances of PAU that remain unsolved by every model, with trajectories from Claude-4-Opus. In both cases, the final hypothesis is directly contradicted by examples the model itself queried (bold), which remained visible in its context. Two annotators with graduate-level computer science training were able to recover the correct description of both methods through interaction alone.
Figure 14: Prompts used by judge and code writing models for evaluation
Figure 15: Interactive mode prompts
Figure 16: Models (examples from Claude-4-Opus) exhibiting shallow exploration, and hence failing to reveal the full functionality of the API method. In both cases, testing more extensive edge cases and varied inputs could have revealed behaviour that contradicted the model’s hypothesis. Instead, it queries the API with inputs that are similar to previous attempts, resulting in premature termination due to apparent confirmation of the hypothesis.
Figure 17: Incontext mode prompt example
Figure 18: Comparison between the incontext and interactive setting shows that models struggle to fully utilize the benefits of interactive API access.
Hyperparameter
Value
Model
Model dtype
bfloat16
LoRA rank
16
LoRA alpha
16
Training
Epochs
5
Appendix
Table 2: SkyRL training hyperparameters.
Step
Signal
Value / Range
Source
Reasoning
Judge score (reasoning quality)
[0,1]
LLM judge ( ÷9 )
Length penalty
[−1,0]
τ=100 , α=0.05
Input proposal
Parse failure
−0.5
hard rule
Clean execution
+0.025
λH/(2Tmax)
Runtime error
−0.5
hard rule
Length penalty
[−1,0]
τ=30 , α=0.5
Appendix
Table 3: Complete reward schedule in the supervised setting.
Method
Qwen3-1.7B
Llama3-8B
Base
1.92
1.99
DISTILL
1.96
2.02
CodeARC-SFT
1.81
1.79
FINAL
1.86
1.91
SYM-INFO
1.22
1.34
AAC
2.04
2.13
Appendix
Table 4: Description score (LM-as-a-judge, 1–5) of AAC against alternate training paradigms. AAC is the only approach that improves substantially over the base model. Training on trajectories from an answer-aware teacher (CodeARC-SFT) or rewarding with a critic that lacks privileged information (SYM-INFO) actively harms performance.