Organizations: North Carolina State University Raleigh, NC, USA · Federal University of Cear´a Itapaj´e, Cear´a, Brazil
A coding agent can migrate a machine learning workload, report success, and still change how the model trains. Rising compute costs, limited accelerator capacity, and vendor dependence are pushing organizations toward alternatives to graphics processing units (GPUs); a key question is whether agent-generated migrations to tensor processing units (TPUs) can be trusted and at what cost. We build a dataset of 148 PyTorch ML research projects and use Claude Code to migrate ten of them from NVIDIA GPUs to Google Cloud TPUs. We independently assess whether each migration runs, preserves numerical behavior, executes on the intended hardware, maintains the project's goal, and delivers acceptable performance. We also examine failures, human effort, and costs. Eight projects meet our project-goal criterion, and migrations require 2.4 human interventions on average. Of 70 execution attempts, 53 fail (76%), and 26 of these failures (49%) raise no error. The most frequent cause is repeated compilation and other execution inefficiencies, while interactions with third-party libraries can silently change the computation: in one four-chip migration, the agent reports success despite a change in gradient clipping that reduces accuracy from 86.53% to 84.89%. Execution costs vary widely: in one workload, TorchTPU, a PyTorch execution backend for TPUs, trains 1.86 times faster and 25% cheaper per run than the GPU. For practitioners, these findings motivate independent reference runs, verification under the intended hardware configuration, and workload-specific acceptance and stopping criteria that account for both computational and human costs.
| Metric | Level | Churn | Agent | Time (min) | Cost ($) | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| # | Project | Challenge | Sub-experiment | Metric | Paper | GPU | TPU | Att. | FF | Human | 1st | Claim. | Meas. | Silent | (files, / ) | $ | h | GPU/TPU | GPU/TPU |
| 1 | MELoRA [ 40 ] | CUDA | RoBERTa-b./MRPC | Acc. | 90.93 | 89.46 | 88.73 | 10 | 0 | 0 (0) | none | L3 | L4 ‡ | 3 | 3, 93/ 1 | 6.89 | 0.97 | 35.9/12.0 | 0.51/0.24 |
| 2 | UniPELT [ 41 ] | Dyn. shape | BERT-b./STS-B | Spear. | 87.10 | 86.58 | 86.15 | 5 | 1 | 1 (0) | none | L3 | L3 | 2 | 11, 124/ 15 | 5.44 | 1.04 | 13.6/8.5 ∗ | 0.19/0.17 ∗ |
| 3 | ZO-Bench [ 42 ] | CUDA | OPT-1.3B/SST-2 | Acc. | 92.20 | 92.78 | 92.09 | 10 | 1 | 3 (2) | none | L3 | L3 | 3 | 5, 196/ 19 | 22.55 | 4.27 | 229.5/338.7 | 3.27/6.77 |
| 4 | BiLLM [ 43 ] | Lin. alg. | OPT-1.3B/WikiText2 | PPL | 69.97 | 77.08 | – | 8 | 1 | 2 (0) | none | – | none | 3 | 9, 116/ 28 | 12.20 | 3.02 | 17.5/– | 0.25/– |
| 5 | CoFi [ 44 ] | Dyn. shape | BERT-b./MRPC | Acc. | 82.60 | 82.60 | 79.17 | 8 | 2 | 3 (1) | none | L2 | L2 | 5 | 10, 298/ 62 | 16.96 | 5.27 | 55.8/80.9 | 0.79/1.62 |
| Category | Cause and diagnostic symptoms | Count |
|---|---|---|
| C1 Environment/dependencies | Incompatible pinned libraries or changed data/model locations; import, API, and package errors. | 7 |
| C2 CUDA/device assumptions | GPU-specific calls ( nvidia-smi , --cuda ), CPU-resident tensors, or optimizer created before device transfer. | 8 |
| C3 Library interactions | HF Trainer, PEFT, or accelerate alter parameter or checkpoint handling; stale parameters, incorrect checkpoint formats, or restoration errors. | 5 |
| C4 Lazy-tensor execution | Dynamic shapes, host synchronization, or graph growth; repeated recompilation, slow steps, or increasing host memory without errors. | 13 |
| C5 Operators/numerics | Unsupported operators, CPU fallbacks, rejected mixed precision, or numerical differences (e.g., sigmoid tail). | 7 |
| C6 Memory capacity | Workload exceeds 16 GB HBM; out-of-memory errors. | 3 |
Software Engineering Agents are AI systems that use code, tools, and feedback to perform software development tasks.
See all 349 papers