Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at https://github.com/cjhCoder7/CangjieBench.
Figures & tables
Figure 1: Overview of the CangjieBench framework: dataset construction, evaluation tasks, and evaluation sandbox.
Problem ID
Problem Name
Exclusion Reason
ClassEval_14
BookManagementDB
Relies on sqlite3 for database operations
ClassEval_25
CookiesUtil
Relies on json for data serialisation
ClassEval_28
DatabaseProcessor
Relies on sqlite3 for database operations
ClassEval_34
DocFileHandler
Relies on docx for Word document processing
ClassEval_38
ExcelProcessor
Relies on openpyxl for spreadsheet manipulation
ClassEval_44
HtmlUtil
Relies on bs4 and gensim for HTML/NLP processing
Table 1: ClassEval tasks excluded from CangjieBench and the Python-specific library responsible for the exclusion.
Figure 2: The four method families evaluated on CangjieBench. Direct generation is the baseline. The other three families add Cangjie-specific context in different ways.
\multirow 2.5* Method
\multirow 2.5* Subject
HumanEval
ClassEval
Avg.
Pass@1
Compile
Pass@1
Compile
Pass@1
Compile
T2C
C2C
T2C
C2C
T2C
C2C
T2C
C2C
T2C
C2C
T2C
C2C
\multirow 6* Direct
DeepSeek-V3
\cellcolor gray!122.4
\cellcolor gray!352.4
\cellcolor gray!123.0
\cellcolor gray!353.0
\cellcolor gray!121.2
\cellcolor gray!351.2
\cellcolor gray!121.2
\cellcolor gray!351.2
\cellcolor gray!122.0
\cellcolor gray!352.0
\cellcolor gray!122.4
\cellcolor gray!352.4
ERNIE-4.5
\cellcolor gray!123.0
\cellcolor gray!354.3
\cellcolor gray!124.3
\cellcolor gray!354.9
\cellcolor gray!120.0
\cellcolor gray!351.2
\cellcolor gray!120.0
\cellcolor gray!351.2
\cellcolor gray!122.0
\cellcolor gray!353.2
\cellcolor gray!122.8
\cellcolor gray!353.6
Kimi-K2
\cellcolor gray!12 20.7
\cellcolor gray!35 21.3
\cellcolor gray!12 23.8
\cellcolor gray!35 23.8
\cellcolor gray!12 4.8
\cellcolor gray!35 8.3
\cellcolor gray!12 7.1
\cellcolor gray!35 8.3
\cellcolor gray!12 15.3
\cellcolor gray!35 16.9
\cellcolor gray!12 18.1
\cellcolor gray!35 18.5
Qwen3
\cellcolor gray!123.7
\cellcolor gray!353.7
\cellcolor gray!124.3
\cellcolor gray!354.3
\cellcolor gray!121.2
\cellcolor gray!352.4
\cellcolor gray!121.2
\cellcolor gray!352.4
\cellcolor gray!122.8
\cellcolor gray!353.2
\cellcolor gray!123.2
\cellcolor gray!353.6
Table 2: Pass@1 (%) and Compile rate (%) on CangjieBench. \colorbox gray!12Light gray columns indicate Text-to-Code (T2C), and \colorbox gray!35dark gray columns indicate Code-to-Code (C2C). The best result per method is in bold , and the overall best across all methods is \color red red and underlined . Avg. averages over HumanEval (164 tasks) and ClassEval (84 tasks) weighted by task count.
Figure 3: Per-task token usage of Qwen3-Coder across the five methods on (a) Text-to-Code and (b) Code-to-Code. Total tokens are on a log scale, and each bar is split into Input (Prompt) and Output (Completion) percentages, with the absolute total printed above.
Figure 4: Pass@1 per 1,000 input tokens for Kimi-K2, Qwen3-Coder, and GPT-5 on (a) Text-to-Code and (b) Code-to-Code. Pass@1 and input tokens are task-count-weighted over HumanEval (164) and ClassEval (84), matching Table 2 . Higher is better.
Figure 5: Δ Pass@1 (C2C − T2C, percentage points), task-count-weighted over HumanEval and ClassEval. Negative (red) means a Python reference hurts on Code-to-Code, and positive (blue) means it helps. Top: prompt-based methods × base models. Bottom: each of the four agent configurations.
\multirow 2* Failure category
Direct
Syntax- Constrained
RAG (Code)
RAG (Docs)
Agent
Avg.
T2C
C2C
T2C
C2C
T2C
C2C
T2C
C2C
T2C
C2C
T2C
C2C
Syntax (foreign)
83.9
82.0
30.0
27.2
56.6
59.1
60.9
61.0
46.1
50.4
58.8
58.7
API/library
12.4
14.0
34.6
42.4
32.5
30.4
26.4
26.6
30.0
30.0
26.0
27.5
Type/mutability
2.6
3.2
16.3
16.8
8.2
8.7
4.6
6.8
13.3
13.7
7.9
8.9
Duplicate def.
0.0
0.1
8.9
10.2
0.0
0.0
4.5
3.7
0.0
0.0
2.6
2.7
Runtime exc.
0.4
0.5
0.7
0.8
0.6
0.7
0.4
0.5
1.2
1.0
0.6
0.7
Table 3: Failure distribution per method and modality, primary annotator glm-5.1. Cells show the share of failed samples in each category. Avg. aggregates failed samples across methods within each modality. The last row gives the total failure count for each column.
Figure 6: Inter-annotator confusion matrix between glm-5.1 (rows) and kimi-k2.6 (columns) for (a) T2C and (b) C2C. Each cell shows the row-normalised percentage and the absolute count.
\multirow 3* Backbone
\multirow 3* Agent harness
HumanEval
ClassEval
Syntax
Agent
Syntax
Agent
Pass@10
Tokens
Pass@1
Tokens
Pass@10
Tokens
Pass@1
Tokens
Kimi-K2
iFlow CLI
84.1 \textcolor green!60!black+45.1
27.1k
39.0
24.2k
28.6 \textcolor green!60!black+10.7
36.6k
17.9
27.5k
Qwen3-Coder
iFlow CLI
64.0 \textcolor green!60!black+34.1
28.1k
29.9
36.1k
19.0 \textcolor green!60!black+14.2
42.2k
4.8
36.2k
Qwen3-Coder
Qwen Code CLI
64.0 \textcolor green!60!black+39.0
28.1k
25.0
224.8k
19.0 \textcolor green!60!black+11.9
42.2k
7.1
212.9k
GPT-5
Codex CLI
96.3 \textcolor green!60!black+11.5
46.4k
84.8
391.7k
42.9 \textcolor green!60!black+10.8
82.8k
32.1
618.2k
Table 4: Repeated syntax-guided sampling against single-run agents on Text-to-Code. Syntax reports task-level Pass@10 from ten Syntax-Constrained samples per task, and Agent reports Pass@1 from one CLI run. \textcolor green!60!black+N.N shows the pp gain of Syntax Pass@10 over Agent Pass@1. Tokens is the input-plus-output sum per task. Same backbone within each row.
Figure 7: Two representative RAG failures. On all_prefixes , RAG-Docs returns ArrayList documentation when the task needs Cangjie string slicing, and the model writes substring (an API Cangjie does not expose), which the compiler rejects. On sum_product , RAG-Code returns syntactically valid but task-irrelevant snippets, and the model still emits while i < n without parentheses.