cs.CLSep 30, 2026
SaveUniBuc at SemEval-2024 Task 2: Tailored Prompting with Solar for Clinical NLI
Organizations: Interdisciplinary School of Doctoral Studies, ♡HLT Research Center University of Bucharest, Romania · Faculty of Mathematics and Computer Science
Abstract
This paper describes the approach of the UniBuc team in tackling the SemEval 2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials. We used SOLAR Instruct, without any fine-tuning, while focusing on input manipulation and tailored prompting. By customizing prompts for individual CTR sections, in both zero-shot and few-shots settings, we managed to achieve a consistency score of 0.72, ranking 14th in the leaderboard. Our thorough error analysis revealed that our model has a tendency to take shortcuts and rely on simple heuristics, especially when dealing with semantic-preserving changes.
Figures & tables
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Section | F1 |
|---|---|
| Eligibility (Single) | 0.6637 |
| Eligibility (Comparison) | 0.5274 |
| Intervention (Single) | 0.7306 |
| Intervention (Comparison) | 0.7751 |
| Results (Single) | 0.7141 |
| Results (Comparison) | 0.7525 |
Table 2: Initial results for the train set (first 50 examples)
| Section | F1 |
|---|---|
| Eligibility (Single) | 0.8178 |
| Eligibility (Comparison) | 0.8285 |
| Intervention (Single) | 0.7678 |
| Intervention (Comparison) | 0.6000 |
| Results (Single) | 0.7368 |
| Results (Comparison) | 0.7749 |
Table 3: Initial results for the development set
| Section (CTR type) | Prompt |
|---|---|
| Eligibility (Single) | Instruction: You are given clinical trial criteria and a statement that may or may not be contradictory. Regarding the inclusion and exclusion criteria, is the statement correct? Respond only with Yes or No. ## Criteria: {premise}. ## Statement: {hypothesis}. ## Response (Yes or No): |
| Eligibility (Comparison) | Instruction: You are given clinical trial criteria for a primary and a secondary trial, and a statement. Regarding the inclusion and exclusion criteria, is the statement correct for each trial? Respond only with Yes or No. ## Criteria: {premise}. ## Statement: {hypothesis}. ## Response (Yes or No): |
| Intervention (Single) | Instruction: You are given a CTR and a statement. Can the statement be deduced from the CTR? Focus on the interventions. Respond only with Yes or No. ##CTR: {premise}. ##Statement: {hypothesis}. ##Response (Yes or No): |
| Intervention (Comparison) | (same prompt template as single CTR for interventions) |
| Results (Single) | Instruction: You are given the results of a CTR and a statement. Can the statement be deduced from the CTR in terms of number of participants, measures and results? Respond only with Yes or No. ##CTR: {premise}. ##Statement: {hypothesis}. ##Response (Yes or No): |
| Results (Comparison) | Instruction: You are given the results of a CTR and a statement. Can the statement be deduced from the CTR? Respond only with Yes or No. ##CTR: {premise}. ##Statement: {hypothesis}. ##Response (Yes or No): |
Table 4: Evaluation templates for each CTR section
| Section (CTR type) | Prompt |
|---|---|
| Eligibility (Single) | Instruction: You are given the eligibility criteria for a clinical trial report. You must summarize the report focusing on inclusion and exclusion criteria. Report: {premise}. Use short sentences. Summary: |
| Eligibility (Comparison) | (same prompt template as single CTR for eligibility) |
| Intervention (Single) | Instruction: You are given the intervention information for a clinical trial report. Each report contains 1-2 cohorts, which receive different treatments, or have different characteristics. You must summarize the report focusing on the type, dosage, frequency, and duration of treatments being studied. Report: {premise}. Use short sentences to group by cohort. Summary: |
| Intervention (Comparison) | Instruction: You are given the intervention information for a clinical trial report. Each report contains 1-2 cohorts, which receive different treatments, or have different characteristics. You must summarize the report focusing on the type, dosage, frequency, and duration of treatments being studied. Report: {premise}. Use short sentences to group by cohort and group by primary trial and secondary trial. Summary: |
| Results (Single) | Instruction: You are given the results of a CTR and a statement. Extract all the relevant information from the CTR that is related to the statement. Report: {premise}. Statement: {hypothesis}. Answer: |
| Results (Comparison) | Instruction: You are given the results of two clinical trials. Each trial contains 1-2 cohorts, which receive different treatments, or have different characteristics. You must summarize the report for each trial, focusing on number of participants, outcome measures, units, results. Report: {premise}. Use short sentences and keep all numeric values. Summary: |
Table 5: Summarization templates for each CTR section. The results section (single CTR) is the only one for which summaries depend on the hypothesis due to lack of time to rerun the summaries for the test set.