Code large language models (Code LLMs) have made significant progress in code generation by translating natural language descriptions into functional code; however, real-world applications often demand stricter adherence to detailed requirements such as coding style, line count, and structural constraints, beyond mere correctness. To address this, the paper introduces forward and backward constraints generation to improve the instruction-following capabilities of Code LLMs in controlled code generation, ensuring outputs align more closely with human-defined guidelines. The authors further present IFEvalCode, a multilingual benchmark comprising 1.6K test samples across seven programming languages (Python, Java, JavaScript, TypeScript, Shell, C++, and C#), with each sample featuring both Chinese and English queries. Unlike existing benchmarks, IFEvalCode decouples evaluation into two metrics: correctness (Corr.) and instruction-following (Instr.), enabling a more nuanced assessment. Experiments on over 40 LLMs reveal that closed-source models outperform open-source ones in controllable code generation and highlight a significant gap between the models' ability to generate correct code versus code that precisely follows instructions.
Figures & tables
Figure 1: Comparision between general code generation and Controllable code generation for 8 programming language and 2 natural language.
Statistics
Number (Zh/En)
Problems
810/810
Python
103/103
Java
102/102
Cpp
100/100
Typescript
100/100
Javascript
100/100
Table 1: Benchmark statistics.
Benchmark
#Languages
#Domains
Size (Zh/En)
Difficulty
Human Check
Check Code Controllability
HumanEval ( Chen et al., 2021a )
Python
3
164 (En)
✓
✗
MBPP ( Austin et al., 2021 )
Python
Algorithm
974 (En)
✓
✗
BigCodeBench ( Zhuo et al., 2024 )
Python
7
2000+ (En)
✗
✗
CodeArena ( Yang et al., 2024a )
44
7
397+ (En)
✗
✗
FullStackBench ( Liu et al., 2024d )
16
11
3374 (En)
✓
✗
LiveCodeBench ( Jain et al., 2024 )
Python
3
450+ (En)
✗
✗
Table 2: Comparison between IFEvalCode and other code benchmarks. Different from the existing benchmarks, IFEvalCode effectively provides a new comprehensive view of controlled code generation by adding the ‘ check_instruction ‘ part into the test samples.
Figure 3: Examples of the verifiable instructions with ‘ check_instruction ’ in IFEvalCode.
Figure 4: Construction of the benchmark IFEvalCode and instruction corpus IFEvalCode-Instruct.
Models
Params
Python
Java
Cpp
C-sharp
Typescript
Javascript
Php
Shell
Avg.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Closed-source LLMs
Claude-3.5-Sonnet
48.6
26.7
33.3
10.8
19.0
9.0
12.6
2.9
24.0
25.0
48.0
32.0
41.0
20.0
28.0
34.0
31.9
20.0
Claude-3.7-Sonnet
47.6
31.4
37.3
7.8
29.0
6.0
16.5
3.9
29.0
28.0
54.0
34.0
59.0
27.0
36.0
43.0
38.5
22.6
GPT-4o
45.7
36.2
25.5
5.9
22.0
5.0
15.5
3.9
30.0
25.0
47.0
26.0
51.0
18.0
32.0
44.0
33.6
20.5
GPT-4o-mini
49.5
30.5
27.5
10.8
10.0
5.0
7.8
5.8
29.0
25.0
47.0
27.0
42.0
22.0
23.0
44.0
29.5
21.2
Table 3: English evaluation of English queries in IFEvalCode of 8 programming languages. ‘Corr.’ denotes the correctness of the model-generated response, and ‘Instr.’ denotes the instruction-following accuracy of the response. The underlined values are the maximum in each column within each colored subgroup.
Models
Params
Python
Java
Cpp
C-sharp
Typescript
Javascript
Php
Shell
Avg.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Corr.
Instr.
Closed-source LLMs
Claude-3.5-Sonnet
49.5
33.3
38.2
11.8
23.0
7.0
14.6
2.9
20.0
29.0
45.0
32.0
47.0
29.0
27.0
36.0
33.1
22.6
Claude-3.7-Sonnet
47.6
28.6
36.3
7.8
28.0
7.0
12.6
3.9
30.0
30.0
51.0
30.0
61.0
22.0
37.0
44.0
37.9
21.6
GPT-4o
42.9
39.0
24.5
7.8
22.0
7.0
14.6
3.9
27.0
26.0
46.0
23.0
51.0
27.0
30.0
42.0
32.2
22.0
GPT-4o-mini
45.7
28.6
21.6
9.8
19.0
3.0
12.6
2.9
20.0
25.0
40.0
28.0
43.0
25.0
20.0
31.0
27.8
19.1
Table 4: Chinese evaluation results in IFEvalCode of 8 programming languages. ‘Corr..’ denotes the correctness of the model-generated response, and ‘Instr.’ denotes the instruction-following accuracy of the response. The underlined fonts denote the best performance in the same parameter range.
ID
Method
Corr. avg
Instr. avg
①
Qwen2.5-Coder-32B
29.9
20.7
②
① + Ds1
34.5
26.4
③
② + Ds2
38.2
30.4
④
③ + Iter2
40.3
34.2
⑤
④ + Iter3
40.5
33.2
Table 5: Ablation study.
Figure 5: Figure 5(a) and 5(b) plot the trends with the number of constraints increaing
Figure 6: The Venn diagram of the numbers of the correct samples for each programming language.