From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python

Jinhua Wang; Biswa Sengupta

cs.SEApr 13, 2026

From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python

Authors: Jinhua Wang, Biswa Sengupta

Organizations: LLM Suite Team, JP Morgan Chase & Co.

Abstract

Cross-language migration of large software systems is a persistent engineering challenge, particularly when the source codebase evolves rapidly. We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust codebase (648K LOC, 65 crates) into Python (41K LOC, 28 modules), with public agent benchmarks as the objective function driving iterative refinement. Our subject system is Codex CLI, a production AI coding agent. We demonstrate that: (1) the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), and achieves 42.5% on Terminal-Bench versus Rust's 47.5%, confirming near-parity on real-world agentic tasks; (2) benchmark-driven debugging, revealing API protocol mismatches, environment pollution, a silent WebSocket failure mode, and an API 400 crash, is more effective than static testing alone; (3) the architecture supports continuous upstream synchronisation via an LLM-assisted diff-translate-test loop; and (4) the Python port has evolved into a capability superset with 30 feature-flagged extensions (multi-agent orchestration, semantic memory, guardian safety, cost tracking) absent from Rust, while preserving strict parity mode for comparison. Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost, while the benchmark-as-objective-function methodology provides a principled framework for growing a cross-language port from parity into an extended platform.

Figures & tables

Panel of Figure 1 — Fig. 1 : Three-tier architecture comparison. Left : the six Rust crates of codex-rs . Center : the Python port ( codex.* ), with one module per Rust crate; arrows denote 1:1 correspondence. Right : codex.enhancements — a Python-only superset of 30 flag-gated capabilities absent from Rust, layered incrementally above the port.

Category	Flags	Default
Multi-agent	MULTI_AGENT , MULTI_AGENT_V2 , MULTI_AGENT_ORCHESTRATION	On
Safety	GUARDIAN	On
Tool extensions	FILE_TOOLS , FILE_EDIT , WEB_FETCH , NOTEBOOK_EDIT , WORKTREE_TOOLS	Mixed
Memory	MEMORY_SYSTEM , TYPED_MEMORY , SEMANTIC_MEMORY , AUTO_MEMORY	Mixed
Context mgmt	MULTI_STRATEGY_COMPACTION , FORKED_COMPACTION	Mixed
Productivity	SKILLS , CRON_TOOL , VOICE_MODE	Mixed

TABLE I : Python-Specific Enhancement Module Categories

Iteration	Fix Applied	Accuracy
v0 (baseline)	Original Rust CLI	47.5%
v1	Naive tmux adapter	0.0%
v2	Full agent runner (Chat Completions)	31.3%
v3	Responses API function tool fix	35.0%
v4	Conversation history fix	35.0%
v5	Venv isolation + ripgrep install	45.0%

TABLE II : Terminal-Bench accuracy across translation iterations

Rust Pattern	Python Equivalent
Result<T, E>	Exceptions ( raise / try )
Option<T>	Optional[T]
enum (algebraic)	@dataclass + Union[...]
enum (simple)	enum.Enum
struct	@dataclass(frozen=True)
impl Trait	Protocol / ABC

TABLE III : Rust-to-Python Idiom Mapping

Panel of Figure 5 — Fig. 2 : Continuous upstream synchronisation pipeline. Benchmark regression triggers re-translation of the failing module.

Module	Py Files	Py LOC	Rs LOC	Ratio
agent	82	12,829	181,903	14.2x
analytics	2	319	989	3.1x
app_server	6	3,452	54,244	15.7x
auth	10	1,896	5,118	2.7x
backend_client	3	682	913	1.3x
cli	7	2,098	4,822	2.3x

TABLE IV : Lines of Code Comparison by Module

Fig. 3 : Lines of code by module (log scale). Python consistently requires fewer lines across all 28 modules.

Fig. 4 : Cyclomatic complexity distribution by module. The majority of functions (90%) achieve rank A (complexity 1–5).

Fig. 5 : Code density comparison. Python achieves higher information density per file.

Module	Py Tests	Rs Tests	Py KLOC	Tests/KLOC
agent	661	2519	15.3	43.1
analytics	12	10	0.3	36.7
app_server	131	448	3.7	35.4
auth	77	67	2.1	36.7
backend_client	12	8	0.7	17.2
cli	142	68	2.3	62.7

TABLE V : Test Parity: Python vs Rust

Fig. 6 : Test function counts by module: Python vs Rust.

Metric	Mean	Std	N
Startup Time (ms)	53.9	2.1	20
Import Time ( $\mu$ s)	57.0	0.0	1
Peak Memory (MB)	30.3	0.1	5
Importable Modules	328.0	0.0	1

TABLE VI : Runtime Performance Metrics (Python)

Tool Orchestration
Task	Mean (ms)	P50 (ms)
Orchestrator (skip approval)	0.030	0.028
Orchestrator (with approval)	0.029	0.028
Tool Registry (10 tools)	0.033	0.030
Shell Execution
Shell Handler ( echo )	3.319	3.289

TABLE VII : Harness Benchmark Results: Real Subsystem Operations

Fig. 7 : Harness benchmark latencies (log scale). The red dashed line marks typical LLM API round-trip time (

\sim

1–10s). All local operations complete 2–6 orders of magnitude faster.

Metric	Python	Rust	Ratio
Source Files	329	1,448	4.4x
Classes / Structs+Enums	1,590	2,675	1.7x
Top-level Functions	781	9,323	11.9x
Methods	1,828	11,202	6.1x
Async Methods	459	0	—
Properties	140	0	—

TABLE VIII : API Surface Comparison

Fig. 8 : API surface comparison (log scale). Python provides equivalent functionality with fewer, higher-level abstractions.

Fig. 9 : Cross-module import dependency heatmap. Darker cells indicate stronger coupling.

Metric	Value
Total Commits	4957
Migration Period Commits	146
Lines Added (migration)	137,331
Lines Deleted (migration)	912,510
Net Line Change	-775,179
Test-related Commits	55

TABLE IX : Migration Effort Summary

Task	Tools	Success
Create Python File	1	100
Shell Echo	1	100
Multi-step File Creation	2	100
Directory Listing	1	100
Git Status	1	100
Complex Pipeline	3	100

TABLE X : End-to-End Agent Task Evaluation

Fig. 10 : Agent evaluation task latencies. All tasks complete in under 25ms. Green bars indicate 100% success rate.

Fig. 11 : LLM pipeline operation latencies (log scale). All local operations complete orders of magnitude faster than LLM API calls.

Agent	Resolved	Accuracy	Unique Passes
Original Codex CLI (Rust)	38 / 80	47.5%	11
codex-python (ours)	34 / 80	42.5%	9
Both solved	25	—	—

TABLE XI : Terminal-Bench head-to-head results (80 tasks, GPT-5.4)

Agent	Tasks	Patches	Resolved	Rate
codex-python (ours)
astropy	22	22	12	54.5%
django	58	58	47	81.0%
Total	80	80	59	73.8%
Original Codex CLI (Rust)
Total	80	80	56	70.0%

TABLE XII : SWE-bench Verified head-to-head results (80 tasks, GPT-5.4, 4 workers)

Appendix figures & tables1 asset

Supplementary material from the paper’s appendix.

Appendix

Sub-package	Flag	Default	Description
agents/	MULTI_AGENT	On	Hierarchical child-agent spawning with inherited context
	MULTI_AGENT_V2	On	Improved coordination protocol with result aggregation
	MULTI_AGENT_ORCHESTRATION	On	Parallel agent dispatch with safety bounds (max depth 5, max 100 agents)
guardian	GUARDIAN	On	Automated LLM-based risk assessment before tool execution
bridge/	IDE_BRIDGE	Off	WebSocket bridge for IDE integration (VS Code, JetBrains)
bridge/	LSP_INTEGRATION	Off	Language Server Protocol support for code intelligence

Appendix

TABLE XIII : Complete list of Python enhancement flags

Code Translation

Code translation converts programs between languages, frameworks, and representations while aiming to preserve behavior and sometimes improve safety or performance.

See all 41 papers

CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora
Apr 20, 2026Shangyu Li, Juyong Jiang, Meibo Ren +7Code Translation RL for Language Models
The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · Monash University +1
Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing
Jun 2, 2026Mehmet Utku ColakAI Coding Agents LLM Prompting
Department of Computer Engineering, Istanbul Technical University · Istanbul, Turkey
Fine-Tuning Qwen3-27B for C-to-Rust Code Translation: A Three-Stage Curriculum of Pretraining, Debugging-Aware SFT, and Task-Specific SFT
Aug 13, 2026Pu Zhao, Changdi Yang, Yixiao Chen +4Code Translation Code Generation
Northeastern University · EmbodyX Inc · Aibao LLC

Apr 20, 2026J/K move · Enter open · S save

CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora

Shangyu Li, Juyong Jiang, Meibo Ren, Sizhe Zhong+6

The Hong Kong University of Science and Technology · The Hong Kong University of Science and Technology (Guangzhou) · Monash University+1

Code Translation RL for Language Models LLM Fine-Tuning Code Language Models

Open paper AskPDF arXiv

Transpilation, or code translation, aims to convert source code from one programming language (PL) to another. It is beneficial for many downstream applications, from modernizing large legacy codebases to augmenting data for low-resource PLs. Recent large language model (LLM)-based approaches have demonstrated immense potential for code translation. Among these approaches, training-based methods are particularly important because LLMs currently do not effectively adapt to domain-specific settings that suffer from a lack of knowledge without targeted training. This limitation is evident in transpilation tasks involving low-resource PLs. However, existing training-based approaches rely on a pairwise transpilation paradigm, making it impractical to support a diverse range of PLs. This limitation is particularly prominent for low-resource PLs due to a scarcity of training data. Furthermore, these methods suffer from suboptimal reinforcement learning (RL) reward formulations. To address these limitations, we propose CodePivot, a training framework that leverages Python as an intermediate representation (IR), augmented by a novel RL reward mechanism, Aggressive-Partial-Functional reward, to bootstrap the model's multilingual transpilation ability without requiring parallel corpora. Experiments involving 10 PLs show that the resulting 7B model, trained on Python-to-Others tasks, consistently improves performance across both general and low-resource PL-related transpilation tasks. It outperforms substantially larger mainstream models with hundreds of billions more parameters, such as Deepseek-R1 and Qwen3-235B-A22B-Instruct-2507, on Python-to-Others tasks and Others-to-All tasks, respectively. In addition, it outperforms its counterpart trained directly on Any-to-Any tasks on general transpilation tasks. The code and data are available at https://github.com/lishangyu-hkust/CodePivot.

Abstract

Code Translation

Explore similar work

CodePivot: Bootstrapping Multilingual Transpilation in LLMs via Reinforcement Learning without Parallel Corpora

Cross-Lingual Token Arbitrage: Optimizing Code Agent Context Windows via Local LLM Preprocessing

Fine-Tuning Qwen3-27B for C-to-Rust Code Translation: A Three-Stage Curriculum of Pretraining, Debugging-Aware SFT, and Task-Specific SFT