Cross-lingual code generation is critical in enterprise environments where multiple programming languages coexist. However, fine-tuning large language models (LLMs) individually for each language is computationally prohibitive. This paper investigates whether parameter-efficient fine-tuning methods and optimizer enhancements can improve cross-lingual transfer from Python to languages like Java. We fine-tune the Code Llama 7B model using low-rank adaptation (LoRA) to optimize a small subset of parameters and compare Adam and Sophia optimizers, while exploring a novel Fourier-based regularization technique. Our contributions include: (1)demonstrating that LoRA fine-tuning on a small, high-quality dataset (MBPP) can exceed the pass@1 performance of the more broadly fine-tuned Code Llama-Python-7B model (40.1% vs. 38.4%); (2) showing that while Sophia achieves faster convergence than Adam, final pass@1 scores show marginal differences; and (3) presenting evidence that Fourier-based regularization during fine-tuning significantly improves cross-lingual transfer, achieving 42.1% pass@1 on Java tasks compared to the 34.2% baseline. These findings suggest that combining LoRA, optimized training methods, and frequency-domain regularization can efficiently adapt single-language LLMs to perform well across multiple programming languages.
Figures & tables
Figure 1: Python HumanEval Pass@1 Benchmark Comparison showing that my LoRA fine-tuned model (40.1%) outperforms Code Llama-Python-7B (38.4%) and the base model (33.5%).
Figure 2: Java: Corrections of prior failures
Figure 3: Java: Improvement of earlier code
Figure 4: Fourier Transform regularization substantially improves cross-lingual generalization, showing a significant increase in Java MultiPL-E Pass@1 from baseline (34.2%) to my method (42.1%).
Figure 5: Frequency distribution of Java code generation performance using Fourier Transform regularization, showing most models achieving scores between 30-40% pass@1, with my best configuration reaching 42.1%.
Figure 6: Python: Corrections of prior failures
Figure 7: Training loss comparison between AdamW and Sophia optimizers. Sophia demonstrates more stable convergence and ultimately reaches a lower final loss.
Parameter
Value
LoRA Rank ( r )
8
LoRA Alpha ( α )
16
LoRA Dropout
0.05
Target Modules
q_proj, v_proj, down_proj, up_proj
Batch Size
4
Learning Rate
2e-4
Table 1: LoRA and Training Configuration
Figure 8: Parameter exploration for Fourier Transform regularization on Java MultiPL-E, showing the relationship between regularization strength, temperature, and model performance.
Optimizer
Validation Loss
Training Time
Memory Usage
AdamW
1.2437
40.3 min
3670.9 MB
Sophia
1.1504
41.3 min
3684.7 MB
Improvement
+7.5%
-2.5%
-0.4%
Table 2: Performance comparison between AdamW and Sophia optimizers
Model Variant
Fourier λ
Pass@1
LoRA (MLP only)
0.02
42.1%
LoRA (Comprehensive)
0.01
38.4%
LoRA (Standard)
0.001
35.4%
Baseline
-
34.2%
Table 3: Effect of Fourier regularization on Java cross-lingual performance
Figure 9: Temperature evaluation results showing that higher temperatures (0.8-1.0) and very low temperatures (0.0-0.2) produced better results than mid-range values.
Figure 10: Beam size evaluation showing performance improvements with larger beam sizes up to 10, after which results plateaued.
Figure 11: Impact of top-p sampling values on model performance, with higher p values generally yielding better results.
Figure 12: Validation loss comparison showing Sophia consistently maintained lower validation loss throughout training.
Figure 13: Perplexity comparison showing Sophia achieved lower perplexity more consistently than AdamW.
Figure 14: Gradient norm comparison showing Sophia maintained smaller, more stable gradient norms compared to AdamW’s larger fluctuations.
Figure 15: Composite performance metrics comparing AdamW and Sophia optimizers, showing Sophia’s overall advantage in training efficiency.
Figure 16: Effect of beam size on cross-lingual transfer performance, showing consistent improvements with larger beam sizes.
Figure 17: 3D visualization of model performance across beam size, temperature, and regularization strength parameters, highlighting the multidimensional nature of hyperparameter optimization.
Figure 18: Performance distribution across different beam sizes for the unmerged LoRA implementation with Fourier regularization.
Figure 19: Performance comparison across different Fourier regularization parameter combinations, revealing clear patterns in effectiveness.
Figure 20: Comprehensive visualization of performance across beam size, temperature, and other hyperparameters, showing the robust performance of my approach.
Figure 21: Three-dimensional prediction surface showing the relationship between key hyperparameters and model performance.