Accuracy Variant HumanEval-X LeetCode Total C++ Java Python C++ Java Python Model: First baseline 0 % 0 % 0 % 0 % 0 % 0 % instruction 0 % 0 % 0 % 0 % 0 % 0 % both-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % cpp-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % python-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % both-triplet-ep:1 0 % 0 % 0 % 0 % 0 % 0 % cpp-triplet-ep:1 0 % 0 % 0 % 0 % 0 % 0 % python-triplet-ep:15 0 % 0 % 0 % 0 % 0 % 0 % Model: General baseline 0 % 0 % 0 % 0 % 0 % 0 % instruction 0 % 0 % 0 % 0 % 0 % 0 % both-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % cpp-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % python-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % both-triplet-ep:1 0 % 0 % 0 % 0 % 0 % 0 % cpp-triplet-ep:1 0 % 0 % 0 % 0 % 0 % 0 % python-triplet-ep:20 0 % 0 % 0 % 0 % 0 % 0 % Model: Tuned baseline 0 % 0 % 0 % 0 % 0 % 0 % instruction 0 % 0 % 0 % 0 % 0 % 0 % both-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % cpp-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % python-infonce-ep:1 0 % 0 % 0 % 0 % 0 % 0 % both-triplet-ep:20 0 % 0 % 0 % 0 % 0 % 0 % cpp-triplet-ep:20 0 % 0 % 0 % 0 % 0 % 0 % python-triplet-ep:20 0 % 0 % 0 % 0 % 0 % 0 % For the code-first model, all the models, both baseline and prompt-tuned, exhibited poor performance. Variants prompt-tuned using the InfoNCE loss function performed particularly poorly, with two variants producing zero correct solutions. Overall, results were bad across training datasets and benchmark datasets, although worse for the LeetCode dataset, e.g., the highest accuracy for LeetCode was 13.5%. The mean accuracy was 21.6% for the baseline, 23.1% for the instruction variant (a 1.5 percentage-point gain), and 21.3% for the best-performing prompt-tuned model, using the Triplet loss function with the Python training set at epoch 15. The best prompt-tuned model only surpassed the baseline for Java code generation, with 8.5% and 3.9% improvements in accuracy for HumanEval-X and LeetCode, respectively. It is interesting to note that the performance of the model variants improved for Java even though they were only trained on Python and C++ examples. For the general-purpose model, Table 5 shows that every combination using InfoNCE or Triplet loss improved accuracy compared to the baseline. The baseline achieved 40.1% correct solutions and the instruction variant 45.5%. Notwithstanding, the accuracy of the prompt-tuned variants ranged from 46.5% to 47.4%, with the highest mean accuracy of 47.4% achieved by the variant using the Python training set and Triplet loss at epoch 1. Comparing InfoNCE and Triplet loss shows that Triplet performs slightly better, but the difference is minimal, with only 0.9%. At the level of individual benchmarks, differences among loss functions and training sets are small. Examining the training sets reveals that the combined training set (Both) yields the best Triplet result at 47.4%, while InfoNCE on the combined set (Both) reaches 46.5%; this small gap indicates that training set choice had little impact for the general purpose model. Most variants reach peak performance early: five variants perform best at epoch 1, and the Python Triplet variant at epoch 1 achieves 47.4% compared to 47.0% at epoch 20, a difference of only 0.4%, suggesting that it was possible to achieve significant benefits from a small amount of contrastive prompt tuning . As observed for the code-first model, the performance of the variants when generating Java code improved compared to the baseline in spite of Java examples not being part of the training set. Similarly, variants prompt-tuned only with Python examples improved their accuracy when generating C++ code. This indicates that the learned representations generalize across languages . The code-tuned model and its variants also benefit from contrastive prompt tuning, as each variant yields higher mean accuracy than the mean baseline accuracy of 38.4% and the mean instruction accuracy of 43.2%. The best variant uses Triplet loss with the C ++ training set at epoch 20, achieving 52.0% correct solutions. All variants improve by at least 5.7% on mean accuracy over the baseline and 0.9% on the instruction variant. A notable factor is that the baseline has zero correct solutions and instruction has 29.9% accuracy for HumanEval-X Java, which strongly influences the overall result. Examining benchmark-level performance shows that all variants underperform the baseline on HumanEval-X Python and LeetCode Python by at least 3.0 and 2.0% respectively. This is unsurprising, as the baseline version was fine-tuned mostly on Python code 10 10 10 https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf . In contrast, multiple variants surpass the baseline, considering both datasets, in Java and C ++ . When compared against the instruction variant, all variants surpass it on HumanEval-X Python and Java, with one exception. In contrast, the instruction variant on HumanEval-X C ++ has the highest accuracy for that language. On LeetCode, the instruction variant exhibits the best results for C ++ and Java, 25% and 17.3%, respectively, a tie with the Triplet loss variant trained on the C ++ training set. Comparing InfoNCE and Triplet loss indicates that Triplet yields higher overall accuracy: Triplet variants range from 46.1% to 52.0% correct solutions, which is 1.0 to 6.9% higher than the best InfoNCE variant at epoch 1 (45.1%). This makes Triplet the superior choice for code correctness for this model. For InfoNCE, the best results occur at epoch 1 and decline with more epochs, whereas Triplet improves with longer prompt tuning. Since longer tuning also consumes more energy, InfoNCE may be preferable when energy cost is a priority but Triplet is better for accuracy. Considering the three training sets for InfoNCE, overall accuracy differences are small except on specific subsets: on HumanEval-X Python, the combined train set (Both) outperforms the C ++ training set by about 7.9%, while on HumanEval-X Java, the C ++ training set outperforms the combined training set (Both) by roughly 3.0%. For Triplet loss, the choice of training set matters more: Tuning with the Python training set results in about 5.9% lower overall accuracy than the C ++ set, mostly caused by an 18.9% drop on HumanEval-X Java. The combined training set (Both) achieves the highest Java accuracy ( 60.4%), but is behind the C ++ training set on other cases by 3–9%. This contrasts with the code-first model’s behavior and shows that optimal training set selection varies by model variant . 6. Energy efficiency of the generated code This section presents energy consumption measurements of the generated code, for baseline models and their variants (best, instruction). We report only on results that reach statistical significance compared to the baseline. Table presents the results as the mean percentage of the energy and execution time consumed by programs generated by a baseline model. We only perform comparisons if both the baseline model and a variant (best or instruction) were able to successfully generate code. Therefore, the number of comparisons for best and instruction in the same row may differ. The use of mean percentages avoids big discrepancies that can arise in this scenario. Furthermore, we argue that the performance of an incorrect program does not matter. An alternative would be to report only on the performance of the common sub-set of correctly generated programs, but this would greatly reduce the amount of reported data. Across all three model variants, training tends to have a more consistent impact on energy efficiency and execution time than instruction. Differences between best-trained and instruction variants are often large. For example, the first model shows an energy efficiency of 58.5% for HumanEval-X C ++ in the best variant versus 174.5% with instruction. Figure presents examples of programs generated for a programming problem from HumanEval-X. The inefficient program on the left-hand side was generated by the baseline code-first model whereas the one on the right-hand side is a more efficient solution produced by the best prompt-tuned variant. The efficient version includes early exits for known cases and skips even numbers, reducing loop iterations by half. In addition, for LeetCode Java (code-first and code-tuned) and LeetCode C ++ (code-first) no program generated through the instruction approach exhibited performance that differed from the baseline with statistical significance. For the best variant, this happened in just one scenario, LeetCode Python (code-first). These results suggest that while training could improve energy efficiency, the results are uncertain , considering the small sample size of this study.