From Translation to Superset: Benchmark-Driven Evolution of a Production AI Agent from Rust to Python
Organizations: LLM Suite Team, JP Morgan Chase & Co.
Abstract
Cross-language migration of large software systems is a persistent engineering challenge, particularly when the source codebase evolves rapidly. We present a methodology for LLM-assisted continuous code translation in which a large language model translates a production Rust codebase (648K LOC, 65 crates) into Python (41K LOC, 28 modules), with public agent benchmarks as the objective function driving iterative refinement. Our subject system is Codex CLI, a production AI coding agent. We demonstrate that: (1) the Python port resolves 59/80 SWE-bench Verified tasks (73.8%) versus Rust's 56/80 (70.0%), and achieves 42.5% on Terminal-Bench versus Rust's 47.5%, confirming near-parity on real-world agentic tasks; (2) benchmark-driven debugging, revealing API protocol mismatches, environment pollution, a silent WebSocket failure mode, and an API 400 crash, is more effective than static testing alone; (3) the architecture supports continuous upstream synchronisation via an LLM-assisted diff-translate-test loop; and (4) the Python port has evolved into a capability superset with 30 feature-flagged extensions (multi-agent orchestration, semantic memory, guardian safety, cost tracking) absent from Rust, while preserving strict parity mode for comparison. Our evaluation shows that for LLM-based agents where API latency dominates, Python's expressiveness yields a 15.9x code reduction with negligible performance cost, while the benchmark-as-objective-function methodology provides a principled framework for growing a cross-language port from parity into an extended platform.
Figures & tables
| Category | Flags | Default |
|---|---|---|
| Multi-agent | MULTI_AGENT , MULTI_AGENT_V2 , MULTI_AGENT_ORCHESTRATION | On |
| Safety | GUARDIAN | On |
| Tool extensions | FILE_TOOLS , FILE_EDIT , WEB_FETCH , NOTEBOOK_EDIT , WORKTREE_TOOLS | Mixed |
| Memory | MEMORY_SYSTEM , TYPED_MEMORY , SEMANTIC_MEMORY , AUTO_MEMORY | Mixed |
| Context mgmt | MULTI_STRATEGY_COMPACTION , FORKED_COMPACTION | Mixed |
| Productivity | SKILLS , CRON_TOOL , VOICE_MODE | Mixed |
| Iteration | Fix Applied | Accuracy |
|---|---|---|
| v0 (baseline) | Original Rust CLI | 47.5% |
| v1 | Naive tmux adapter | 0.0% |
| v2 | Full agent runner (Chat Completions) | 31.3% |
| v3 | Responses API function tool fix | 35.0% |
| v4 | Conversation history fix | 35.0% |
| v5 | Venv isolation + ripgrep install | 45.0% |
| Rust Pattern | Python Equivalent |
|---|---|
| Result<T, E> | Exceptions ( raise / try ) |
| Option<T> | Optional[T] |
| enum (algebraic) | @dataclass + Union[...] |
| enum (simple) | enum.Enum |
| struct | @dataclass(frozen=True) |
| impl Trait | Protocol / ABC |
| Module | Py Files | Py LOC | Rs LOC | Ratio |
|---|---|---|---|---|
| agent | 82 | 12,829 | 181,903 | 14.2x |
| analytics | 2 | 319 | 989 | 3.1x |
| app_server | 6 | 3,452 | 54,244 | 15.7x |
| auth | 10 | 1,896 | 5,118 | 2.7x |
| backend_client | 3 | 682 | 913 | 1.3x |
| cli | 7 | 2,098 | 4,822 | 2.3x |
| Module | Py Tests | Rs Tests | Py KLOC | Tests/KLOC |
|---|---|---|---|---|
| agent | 661 | 2519 | 15.3 | 43.1 |
| analytics | 12 | 10 | 0.3 | 36.7 |
| app_server | 131 | 448 | 3.7 | 35.4 |
| auth | 77 | 67 | 2.1 | 36.7 |
| backend_client | 12 | 8 | 0.7 | 17.2 |
| cli | 142 | 68 | 2.3 | 62.7 |
| Metric | Mean | Std | N |
|---|---|---|---|
| Startup Time (ms) | 53.9 | 2.1 | 20 |
| Import Time ( s) | 57.0 | 0.0 | 1 |
| Peak Memory (MB) | 30.3 | 0.1 | 5 |
| Importable Modules | 328.0 | 0.0 | 1 |
| Task | Mean (ms) | P50 (ms) |
|---|---|---|
| Tool Orchestration | ||
| Orchestrator (skip approval) | 0.030 | 0.028 |
| Orchestrator (with approval) | 0.029 | 0.028 |
| Tool Registry (10 tools) | 0.033 | 0.030 |
| Shell Execution | ||
| Shell Handler ( echo ) | 3.319 | 3.289 |
| Metric | Python | Rust | Ratio |
|---|---|---|---|
| Source Files | 329 | 1,448 | 4.4x |
| Classes / Structs+Enums | 1,590 | 2,675 | 1.7x |
| Top-level Functions | 781 | 9,323 | 11.9x |
| Methods | 1,828 | 11,202 | 6.1x |
| Async Methods | 459 | 0 | — |
| Properties | 140 | 0 | — |
| Metric | Value |
|---|---|
| Total Commits | 4957 |
| Migration Period Commits | 146 |
| Lines Added (migration) | 137,331 |
| Lines Deleted (migration) | 912,510 |
| Net Line Change | -775,179 |
| Test-related Commits | 55 |
| Task | Tools | Success | Mean (ms) | P50 (ms) |
|---|---|---|---|---|
| Create Python File | 1 | 100 | ||
| Shell Echo | 1 | 100 | ||
| Multi-step File Creation | 2 | 100 | ||
| Directory Listing | 1 | 100 | ||
| Git Status | 1 | 100 | ||
| Complex Pipeline | 3 | 100 |
| Agent | Resolved | Accuracy | Unique Passes |
| Original Codex CLI (Rust) | 38 / 80 | 47.5% | 11 |
| codex-python (ours) | 34 / 80 | 42.5% | 9 |
| Both solved | 25 | — | — |
| Agent | Tasks | Patches | Resolved | Rate |
| codex-python (ours) | ||||
| astropy | 22 | 22 | 12 | 54.5% |
| django | 58 | 58 | 47 | 81.0% |
| Total | 80 | 80 | 59 | 73.8% |
| Original Codex CLI (Rust) | ||||
| Total | 80 | 80 | 56 | 70.0% |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Sub-package | Flag | Default | Description |
| agents/ | MULTI_AGENT | On | Hierarchical child-agent spawning with inherited context |
| MULTI_AGENT_V2 | On | Improved coordination protocol with result aggregation | |
| MULTI_AGENT_ORCHESTRATION | On | Parallel agent dispatch with safety bounds (max depth 5, max 100 agents) | |
| guardian | GUARDIAN | On | Automated LLM-based risk assessment before tool execution |
| bridge/ | IDE_BRIDGE | Off | WebSocket bridge for IDE integration (VS Code, JetBrains) |
| LSP_INTEGRATION | Off | Language Server Protocol support for code intelligence |
Code Translation
Code translation converts programs between languages, frameworks, and representations while aiming to preserve behavior and sometimes improve safety or performance.
See all 41 papers