ORCA: Evaluating LLMs on Data Science Code Translation

Xiaolong Li; Jinyang Li; Bowen Qin; Ge Qu; Nan Huo; Xiaohan Xu; Shipei Lin; Reynold Cheng

cs.AISep 25, 2026

ORCA: Evaluating LLMs on Data Science Code Translation

Authors: Xiaolong Li, Jinyang Li, Bowen Qin, Ge Qu, Nan Huo, Xiaohan Xu, Shipei Lin, Reynold Cheng

Organizations: The University of Hong Kong · National University of Singapore · The Chinese University of Hong Kong, Shenzhen

Abstract

Data Science Code Translation (DSCT) is the process of converting code between data science libraries while preserving functional equivalence and enabling interoperability across data science ecosystems. While Large Language Models (LLMs) have demonstrated considerable progress in Data Science Code Generation (DSCG), their performance in DSCT remains insufficiently studied. To address this gap, we introduce ORCA, a comprehensive benchmark with two complementary settings: ORCA-MAIN, which comprises 1,600 carefully curated grounding-level tasks across 3 representative domains: Data Querying, Data Manipulation, and Deep Learning; and ORCA-PROJECT, which contains 200 translation tasks over complete data science projects across 7 data science task types. Each task is accompanied by annotated reference translations and test cases for validating functional equivalence. We further incorporate a multi-stage quality verification process that thoroughly verifies task correctness and test case robustness. Experimental results demonstrate challenges in DSCT, with even frontier LLMs showing limited performance. Specifically, Claude-Opus-4.6 achieves a success rate of 56.92% on ORCA-MAIN and 33.67% on ORCA-PROJECT, indicating considerable room for improvement in DSCT. We also observe a clear directional preference in DSCT, where translation is consistently easier when the source code expresses the task through more explicit, fine-grained operations. Motivated by this, we propose an intent-augmented method, in which the model first infers source-code intent and then uses it as additional context for translation, achieving average absolute success-rate gains of 4.80% and 5.33% on ORCA-MAIN and ORCA-PROJECT, respectively.

Figures & tables

Figure 1: ORCA evaluates DSCT by providing a model with source code $\mathcal{S}$ , associated data $\mathcal{D}$ , and translation requirement $\mathcal{R}$ , requiring translated target code $\mathcal{T}$ that is verified by execution-based test cases. ORCA supports both grounding-level tasks and project-level tasks.

Figure 2: Overview of the ORCA construction pipeline: data collection, task construction, and quality assurance for grounding-level and project-level DSCT.

Figure 3: Distribution summary of ORCA-Main operation categories and ORCA-Project task types.

Statistic	Main	Project
Total Tasks	1,600	200
Avg. Source-Code Tokens	988	8439
# Distinct Test Cases	5598	2406
# DS Libraries	5	21
# Distinct APIs	496	615
# Databases	31	26

Table 1: Data Statistics

Popular Open-Source Models
Model	Data Querying	Data Manipulation	Deep Learning	Overall
Qwen2.5-Coder-7B	$11.67_{\pm 0.93}$	$17.22_{\pm 1.18}$	$23.33_{\pm 1.84}$	$16.67_{\pm 0.73}$
Llama-3.1-8B	$7.22_{\pm 0.92}$	$11.33_{\pm 1.09}$	$18.50_{\pm 1.39}$	$11.58_{\pm 0.64}$
Ministral-14B	$18.11_{\pm 0.98}$	$28.33_{\pm 1.36}$	$19.33_{\pm 1.23}$	$22.25_{\pm 0.70}$
Phi-4	$27.78_{\pm 1.34}$	$25.56_{\pm 0.92}$	$25.67_{\pm 1.18}$	$26.42_{\pm 0.67}$
Gemma-3-27B	$20.78_{\pm 1.07}$	$28.33_{\pm 1.30}$	$22.00_{\pm 1.39}$	$23.92_{\pm 0.72}$

Table 2: Performance of various LLMs on ORCA-Main . Best results in each category are marked in bold.

Figure 4: ORCA-Main average SR across translation directions.

Figure 5: ORCA-Project baseline performance.

Figure 6: An example of Intent-Augmented translation from PyTorch to TensorFlow .

	ORCA-Main			ORCA-Project
Model	Baseline	CoT-Intent	Intent	Baseline	CoT-Intent	Intent
Qwen2.5-Coder-32B	$30.88_{\pm 0.66}$	$29.50_{\pm 0.86}$	$\mathbf{38.71_{\pm 0.61}}$	$4.67_{\pm 1.04}$	$3.33_{\pm 1.15}$	$\mathbf{7.00_{\pm 1.32}}$
Llama-3.1-70B	$26.21_{\pm 0.69}$	$25.50_{\pm 0.72}$	$\mathbf{33.04_{\pm 0.63}}$	$2.00_{\pm 0.87}$	$1.33_{\pm 0.76}$	$\mathbf{4.00_{\pm 1.32}}$
DeepSeek-V3	$46.96_{\pm 0.65}$	$\mathbf{48.50_{\pm 0.66}}$	$47.38_{\pm 0.75}$	$18.00_{\pm 2.00}$	$23.00_{\pm 1.73}$	$\mathbf{24.00_{\pm 2.00}}$
Kimi-K2.5	$51.12_{\pm 0.65}$	$52.79_{\pm 0.66}$	$\mathbf{56.04_{\pm 0.55}}$	$24.00_{\pm 2.18}$	$28.67_{\pm 1.89}$	$\mathbf{31.33_{\pm 2.08}}$
Claude-Opus-4.6	$56.92_{\pm 0.59}$	$\mathbf{62.71_{\pm 0.55}}$	$60.92_{\pm 0.46}$	$33.67_{\pm 2.57}$	$\mathbf{45.00_{\pm 2.00}}$	$42.67_{\pm 2.25}$

Table 3: Performance comparison across Baseline, CoT-Intent, and Intent methods on ORCA-Main and ORCA-Project . Best results in each setting are marked in bold.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	Level	# Instances	# DS Libraries	Avg. Test Cases / Task	Avg. Lines	Data Source
Data Science Code Generation
DSP [ 6 ]	Snippet	1,119	8	2.1	4.5	Notebooks
DS-1000 [ 25 ]	Snippet	1,000	7	1.6	3.6	Stack Overflow
DA-Code [ 18 ]	File	500	10	—	85	Kaggle / GitHub
InfiAgent-DABench [ 16 ]	File	603	4	—	—	GitHub (CSV)
DSCodeBench [ 23 ]	Snippet	1,000	10	200	22.5	GitHub

Appendix

Table 4: Comparison of ORCA with existing benchmarks. † Mixed refers to Stack Overflow, BIRD, and LiveSQLBench.

Library	Version	Library	Version
pandas	2.1.4	numpy	1.26.4
polars	0.20.31	pyarrow	22.0.0
torch	2.8.0	tensorflow	2.15.0
xgboost	2.0.3	lightgbm	4.2.0
scikit-learn	1.7.2	catboost	1.2.2
mord	0.7	sqlalchemy	2.0.45

Appendix

Table 5: Library versions in ORCA.

Figure 7: Screenshot of the annotator instruction manual used in the tutorial.

Source Code	Baseline	CoT-Intent	Intent
DQ ⬇ -- (... 18 lines above ...) WITH TransplantData AS (...), CenterAverages AS (...) SELECT ... -- (... 12 lines omitted ...) Core challenge: multi-CTE + JSON + strict output schema.	⬇ # (... 20 lines above ...) final_df = mismatch_cases . merge (...) final_df = final_df . sort_values (...) final_df = final_df . assign (... round (...)) # round # (... 8 lines omitted ...) Failed : output-format mismatch.	⬇ # (... 18 lines above ...) def safe_json_extract (...): ... center_averages = (...). groupby (...). mean () result_df = (...). assign (... Decimal (...)) # (... 6 lines omitted ...) Passed	⬇ # (... 15 lines above ...) result_df = result_df . rename (...) result_df [ ’avg_risk...’ ] = ... round (4) # avg_risk result_df = result_df . sort_values (...) # (... 7 lines omitted ...) Failed : format/order mismatch.
DM ⬇ # (... 22 lines above ...) daily_stats = groupby (...). resample ( ’D’ ). agg (...) rolling_corr = groupby (...). rolling (...). corr (...) merged_df = reset_daily_stats . merge (...) # (... 10 lines omitted ...) Core challenge: grouped time-index alignment.	⬇ # (... 14 lines above ...) rolling_corr = np . full ((..., 31), np . nan ) ... rolling_corr [ group , ts //24] = corr # ts//24 # (... 6 lines omitted ...) Failed : fixed day-axis causes index overflow.	⬇ # (... 14 lines above ...) daily_stats = np . zeros (..., dtype = np . float64 ) result = np . zeros (..., dtype = np . float64 ) result = np . round ( result , 4) # (... 6 lines omitted ...) Passed	⬇ # (... 11 lines above ...) result = np . zeros (( GROUPS , 6)) rolling_corr = np . corrcoef (...) result = np . round ( result , 4) # (... 7 lines omitted ...) Passed
DL ⬇ # (... 16 lines above ...) class ResidualBlock ( nn . Module ): ... self . shortcut = nn . Sequential (...) class BatchSizeAware ResNet (...): ... # (... 10 lines omitted ...) Core challenge: shortcut + training-mode behavior.	⬇ # (... 9 lines above ...) self . shortcut = layers . Sequential (...) self . shortcut = layers . Sequential ([...]) # layers.Sequential # (... 5 lines omitted ...) Failed : invalid API path.	⬇ # (... 20 lines above ...) def call ( self , x ): out = self . bn2 ( out ) # bn2(out) out += self . shortcut ( identity ) # (... 15 lines omitted ...) Failed : training-state unresolved.	⬇ # (... 12 lines above ...) class ResidualBlock ( tf . keras . Model ): self . shortcut = tf . keras . Sequential ([...]) def call ( self , x , training = None ): out = self . bn1 ( out , training = training ) # (... 8 lines omitted ...) Passed

Appendix

Table 6: Qualitative comparison of baseline, CoT-Intent, and Intent-Augmented translations on representative DQ, DM, and DL examples.

Model	Pass@1		Pass@k
Model	SR (%)	Cost (USD)	SR (%)	Cost (USD)
Gemma-3-27B	23.94	$0.45	25.25	$1.47
DeepSeek-R1	48.12	$2.66	50.62	$8.01

Appendix

Table 7: Performance and inference cost on ORCA-Main

Model	My $\rightarrow$ PG	PG $\rightarrow$ My	$\|\Delta\|$	My $\rightarrow$ SQ	SQ $\rightarrow$ My	$\|\Delta\|$	PG $\rightarrow$ SQ	SQ $\rightarrow$ PG	$\|\Delta\|$	$\overline{\|\Delta\|}$
	MySQL $\leftrightarrow$ PostgreSQL			MySQL $\leftrightarrow$ SQLite			PostgreSQL $\leftrightarrow$ SQLite
Qwen2.5-Coder-7B	52.2	45.0	7.20	71.2	53.8	17.40	57.4	52.8	4.60	9.73
Llama-3.1-8B	47.6	46.0	1.60	62.2	47.6	14.60	53.2	47.8	5.40	7.20
Ministral-14B	51.4	46.0	5.40	60.2	47.2	13.00	52.2	49.4	2.80	7.07
Phi-4	49.8	43.8	6.00	60.6	52.2	8.40	48.4	51.8	3.40	5.93
Gemma-3-27B	52.0	48.6	3.40	68.0	53.2	14.80	57.6	52.8	4.80	7.67

Appendix

Table 8: SQL dialect-to-dialect translation performance on BIRD Mini Dev. Each cell is Success Rate (%).

|\Delta|

columns report the absolute gap between the two directions of each language pair;

\overline{|\Delta|}

is the per-model average over the three pairs. The bottom row reports the macro-average across all listed models.

Popular Open-Source Models
Model	Overall
Phi-4	$0.00_{\pm 0.00}$
Qwen2.5-Coder-32B	$4.67_{\pm 1.04}$
Llama-3.1-70B	$2.00_{\pm 0.87}$
GLM-4.7	$15.67_{\pm 1.61}$
DeepSeek-V3	$18.00_{\pm 2.00}$

Appendix

Table 9: Baseline performance of various LLMs on ORCA-Project . Results are reported as Success Rate (%), with mean

\pm

standard deviation over 3 runs. Best results are marked in bold.

Bin	#Tasks	Avg. Comp.	Opus-4.6	Sonnet-4.5	DS-V3	Kimi-K2.5	Avg.
Low	66	29.45	46.97	39.39	22.73	31.82	35.23
Medium	66	79.33	27.27	28.79	16.67	24.24	24.24
High	68	237.79	26.47	16.18	14.71	16.18	18.38
High–Low Change	–	–	-20.50	-23.21	-8.02	-15.64	-16.85

Appendix

Table 10: SR by data-complexity bin across all ORCA-Project tasks. Values are reported as mean SR (%). The last row reports the High–Low change, computed as the High-bin SR minus the Low-bin SR, with negative values indicating performance degradation.

Bin	# Tasks	Avg. Comp.	Opus-4.6	Sonnet-4.5	DS-V3	Kimi-K2.5	Avg.
Low	20	35.60	50.00	35.00	20.00	40.00	36.25
Medium	20	78.70	35.00	30.00	18.00	35.00	29.50
High	22	200.82	22.73	13.64	13.64	13.64	15.91
High–Low Change	–	–	-27.27	-21.36	-6.36	-26.36	-20.34

Appendix

Table 11: SR by data-complexity bin within Classification tasks. Values are reported as mean SR (%). The last row reports the High–Low change, computed as the High-bin SR minus the Low-bin SR, with negative values indicating performance degradation.

Popular Open-Source Models
	Data Querying			Data Manipulation			Deep Learning
Model	Pd $\rightarrow$ PG	PG $\rightarrow$ Pd	$\Delta$	NP $\rightarrow$ Pd	Pd $\rightarrow$ NP	$\Delta$	TF $\rightarrow$ PT	PT $\rightarrow$ TF	$\Delta$
Qwen2.5-Coder-7B	15.54	7.80	+7.74	28.34	6.10	+22.24	27.74	18.92	+8.82
Llama-3.1-8B	9.30	5.14	+4.16	20.35	2.31	+18.04	22.77	14.23	+8.54
Ministral-14B	26.86	9.36	+17.50	42.05	14.61	+27.44	23.96	14.70	+9.26
Phi-4	35.57	19.99	+15.58	37.64	13.48	+24.16	31.93	19.41	+12.52

Appendix

Table 12: Bidirectional performance on ORCA-Main across DQ, DM, and DL subsets. Each direction cell reports mean SR (%) over three runs.

\Delta

is computed as the first listed direction minus the second direction within each domain. Best SR results in each direction column within the proprietary-model section are marked in bold. Pd: Pandas, PG: PostgreSQL, NP: NumPy, TF: TensorFlow, PT: PyTorch.

Translation direction	Direct	Multi-step or indirect
PostgreSQL $\rightarrow$ Pandas	2,521 (70.48%)	1,056 (29.52%)
Pandas $\rightarrow$ PostgreSQL	2,746 (99.35%)	18 (0.65%)
NumPy $\rightarrow$ Pandas	1,449 (65.80%)	753 (34.20%)
Pandas $\rightarrow$ NumPy	550 (42.05%)	758 (57.95%)
PyTorch $\rightarrow$ TensorFlow	2,665 (99.22%)	21 (0.78%)
TensorFlow $\rightarrow$ PyTorch	2,546 (96.66%)	88 (3.34%)

Appendix

Table 13: Operation-level correspondence across all ORCA-Main tasks. Counts refer to source-operation occurrences rather than unique API names or tasks.

ORCA-Main			ORCA-Project
Core operations/functions	Tasks	Share	Files translated	Tasks	Share
1	41	2.6%	3	16	8.0%
2–10	530	33.1%	4	127	63.5%
11–22	519	32.4%	5	51	25.5%
$\geq$ 23	510	31.9%	6	6	3.0%

Appendix

Table 14: Structural granularity of ORCA-Main and ORCA-Project .

Domain	Simple	Moderate	Difficult
Data Querying	180 / 45.21	234 / 40.68	186 / 33.13
Data Manipulation	189 / 39.35	233 / 37.50	178 / 34.59
Deep Learning	124 / 54.23	153 / 26.67	123 / 17.33
Overall	493 / 45.23	620 / 36.03	487 / 29.67

Appendix

Table 15: Task counts and aggregate SR across all 16 evaluated models by ORCA-Main structural-complexity group. Each cell reports #tasks / SR (%).

Domain	Direction	GPT-5.5	Claude-Opus-4.6
DQ	PostgreSQL $\rightarrow$ Pandas	34/50 (68%)	34/50 (68%)
DQ	Pandas $\rightarrow$ PostgreSQL	35/50 (70%)	22/50 (44%)
DM	NumPy $\rightarrow$ Pandas	29/50 (58%)	27/50 (54%)
DM	Pandas $\rightarrow$ NumPy	16/50 (32%)	11/50 (22%)
DL	PyTorch $\rightarrow$ TensorFlow	16/50 (32%)	14/50 (28%)
DL	TensorFlow $\rightarrow$ PyTorch	28/50 (56%)	22/50 (44%)

Appendix

Table 16: GPT-5.5 and Claude-Opus-4.6 on representative ORCA subsets. Cells report passed tasks / total tasks (SR).

Model	Prompt	DQ	DM	DL	Overall
GPT-5.5	Baseline	69	45	44	52.67
	Detailed-Criteria	71 (+2)	56 (+11)	35 (-9)	54.00 (+1.33)
	Library-Version	68 (-1)	48 (+3)	37 (-7)	51.00 (-1.67)
Claude-Opus-4.6	Baseline	56	38	36	43.33
	Detailed-Criteria	58 (+2)	42 (+4)	39 (+3)	46.33 (+3.00)
	Library-Version	56 (0)	40 (+2)	39 (+3)	45.00 (+1.67)

Appendix

Table 17: Prompt ablations on the 300-task ORCA-Main subset. Each cell reports SR (%); parenthesized values are changes from the corresponding baseline.

Domain	Direction	Direct SR	Agent SR	$\Delta$
DQ	PostgreSQL $\rightarrow$ Pandas	68	70	+2
DQ	Pandas $\rightarrow$ PostgreSQL	44	58	+14
DM	NumPy $\rightarrow$ Pandas	54	56	+2
DM	Pandas $\rightarrow$ NumPy	22	24	+2
DL	PyTorch $\rightarrow$ TensorFlow	28	34	+6
DL	TensorFlow $\rightarrow$ PyTorch	44	52	+8

Appendix

Table 18: Direct prompting and a ReAct-style agent on ORCA-Main .

Subset	Baseline SR	Claude Code SR	Baseline cost	Claude Code cost
Data Querying	56.0	69.0	–	–
Data Manipulation	38.0	45.0	–	–
Deep Learning	36.0	44.0	–	–
ORCA-Main overall	43.33	52.67	$6.72	$202.88
ORCA-Project	17.0	26.0	$46.84	$164.89
Total cost	–	–	$53.56	$367.77

Appendix

Table 19: Claude Code performance and cost on representative ORCA subsets.

Failure pattern	Share	Definition
Data or interface mismatch	58.1%	Incorrect columns, schema, container, public interface, shape, axis, layout, dtype, precision, or null behavior.
Logic or behavior mismatch	22.9%	Executable code with incorrect query or algorithmic logic, train/eval behavior, randomness, or gradient behavior.
Incomplete or invalid implementation	19.0%	Missing imports, helpers, or definitions; invalid target-library APIs; or failed execution.

Appendix

Table 20: Failure patterns across the 3 prompt settings.

Failure pattern	Claude Code	Baseline
Data or interface mismatch	85/142 (59.9%)	99/170 (58.2%)
Logic or behavior mismatch	32/142 (22.5%)	44/170 (25.9%)
Incomplete or invalid implementation	25/142 (17.6%)	27/170 (15.9%)

Appendix

Table 21: Claude Code and baseline failure patterns on ORCA-Main . Counts are over failed tasks in each setting.

Failure pattern	Claude Code	ReAct baseline
Cross-stage interface	35/74 (47.3%)	35/83 (42.2%)
Local data semantics	5/74 (6.8%)	7/83 (8.4%)
Output or project contract	30/74 (40.5%)	39/83 (47.0%)
Modeling, implementation, or execution	4/74 (5.4%)	2/83 (2.4%)

Appendix

Table 22: Claude Code and ReAct-baseline failure patterns on the 100-task ORCA-Project subset. Counts are over failed tasks in each setting.

Failure pattern	Share	Definition
Cross-stage interface	38%	Incompatible schema, dtype, shape, ordering, preprocessing state, train–inference alignment, or I/O between an identifiable producer and consumer.
Local data semantics	38%	Incorrect query, transformation, split, leakage decision, feature computation, or numerical operation within 1 stage.
Output or project contract	13%	Violation of a required output schema, formatting, cardinality, file, function, class, signature, entry point, or return format.
Modeling, implementation, or execution	11%	Errors in model construction, training, prediction, loss or metric behavior, completeness, API usage, or execution.

Appendix

Table 23: Systematic failure breakdown for the project-level analysis in Section 5.4.

Cross-stage interface		Local data semantics
Subtype	Share	Subtype	Share
Column or schema	72.44%	Field/nested-value extraction	56.39%
Train–inference alignment	50.67%	Completeness/missing values	37.44%
Dtype	39.11%	Parsing/type conversion	32.60%
Feature or value ordering	24.44%	Query/filter/join/aggregation	30.84%
Serialization or stage I/O	16.44%	Split/sampling/leakage	25.99%

Appendix

Table 24: Subtype breakdowns for cross-stage and local-data failures.

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Code Translation

Code translation converts programs between languages, frameworks, and representations while aiming to preserve behavior and sometimes improve safety or performance.

See all 42 papers

DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science
Feb 27, 2026Fan Shu, Yite Wang, Ruofan Wu +4LLM Evaluation Instruction Following
University of Houston · Snowflake AI Research
SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation
Jul 21, 2026Weifeng Sun, Ye Fan, Yuchen Chen +6Scientific Code Generation Code Generation Evaluation
Singapore Management University · Nanjing University · SUN YAT-SEN University +1
Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation
Oct 20, 2026Yali Du, Yunji Zhang, Ming LiCode Translation Hallucination in Language Models
National Key Laboratory for Novel Software Technology, and the School of Artificial Intelligence, Nanjing University, China

Feb 27, 2026J/K move · Enter open · S save

DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

Fan Shu, Yite Wang, Ruofan Wu, Boyi Liu+3

University of Houston · Snowflake AI Research

LLM Evaluation Instruction Following LLM Fine-Tuning

Open paper AskPDF arXiv

The fast-growing demands in using Large Language Models (LLMs) to tackle complex multi-step data science tasks create an emergent need for accurate benchmarking. There are two major gaps in existing benchmarks: (i) the lack of standardized, process-aware evaluation that captures instruction adherence and process fidelity, and (ii) the scarcity of accurately labeled training data. To bridge these gaps, we introduce DARE-bench, a benchmark designed for machine learning modeling and data science instruction following. Unlike many existing benchmarks that rely on human- or model-based judges, all tasks in DARE-bench have verifiable ground truth, ensuring objective and reproducible evaluation. To cover a broad range of tasks and support agentic tools, DARE-bench consists of 6,300 Kaggle-derived tasks and provides both large-scale training data and evaluation sets. Extensive evaluations show that even highly capable models such as gpt-o4-mini struggle to achieve good performance, especially in machine learning modeling tasks. Using DARE-bench training tasks for fine-tuning can substantially improve model performance. For example, supervised fine-tuning boosts Qwen3-32B's accuracy by 1.83x and reinforcement learning boosts Qwen3-4B's accuracy by more than 8x. These significant improvements verify the importance of DARE-bench both as an accurate evaluation benchmark and critical training data.

Abstract

Code Translation

Explore similar work

DARE-bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

SciCodePile: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

Hallucinations of Large Language Models in Cross-Language Code Translation: Phenomena and Mitigation