Balancing Chain-of-Thought and Answer Supervision for Small Language Models on Medical Question Answering

Le Zhou; Xiao Luo; Chinmay Chandra; Zhan Zhang

cs.CLNov 2, 2026

Balancing Chain-of-Thought and Answer Supervision for Small Language Models on Medical Question Answering

Authors: Le Zhou, Xiao Luo, Chinmay Chandra, Zhan Zhang

Organizations: Pace University, New York City, USA · Southern Methodist University, Dallas, USA

Abstract

Small language models (<=9B parameters) can support local medical question answering (QA) while reducing privacy and latency concerns, but effective fine-tuning remains challenging. This paper compares two chain-of-thought (CoT) supervised fine-tuning methods: Uniform-across-Sequence (UaS), which applies equal loss weighting to the full CoT-and-answer sequence, and Segment-weighted (SwS), which separately weights CoT and answer losses. We evaluate six general- and medical-domain models on multiple-choice QA (MedQA and EMS-MCQA) and free-text QA (Med-O1). Results show that neither method is universally best: on multiple-choice QA (MedQA and EMS-MCQA), SwS generally performs better, with general-purpose models benefiting most; notably, Qwen3.5-4B improves by up to 44.8% over its zero-shot baseline on MedQA. In contrast, medical-domain models show smaller or even negative gains on MCQs. On free-text QA (Med-O1), UaS performs best for nearly all models and provides consistent improvements across both model types, with the 3B Llama-3.2 model approaching the performance of much larger 7B medical-domain models. We also find that excessively long CoT generation is associated with lower performance in some models on either multiple-choice or free-text QA tasks, suggesting that limiting the CoT length during training may benefit selected models. Overall, the results show that CoT loss weighting should be matched to the task format and the knowledge already encoded in the base model, rather than using a single fine-tuning strategy for all clinical QA tasks.

Figures & tables

Fig. 1 : Overview of the training and evaluation pipeline. (A) Training: MCQ (MedQA/EMS-MCQA) and free-text (Med-O1) examples are converted into a shared CoT-and-answer SFT target, supervised under either Uniform (UaS) or Segment-weighted (SwS) loss. (B) Evaluation: generated responses are parsed to output CoT and answer via the Answer: delimiter and evaluated by accuracy (MCQ) or Token F1/BLEU/ROUGE/BERTScore (free-text).

Dataset	Answer Type	Ave. CoT Length (# of Words)	Ave. Answer Length (# of Words)	Training	Test
MedQA	Multiple Choice	311.5	3.5	7,080	600
EMS-MCQA	Multiple Choice	51.2	5.7	2,600	600
Med-O1	Free Text	312.5	94.4	13,793	600

TABLE I : Datasets and Statistics

General-purpose Models
Models	MedQA			EMS-MCQA
Models	Base	UaS	SwS ( $\lambda$ )	Base	UaS	SwS ( $\lambda$ )
Llama-3.2-3B	58.0	63.5	64.7* (0.6)	66.0	62.0	66.2 (1.0)
Qwen3.5-4B	33.0	65.5	77.8* (0.8)	33.2	53.8	62.5* (0.8)
Qwen3.5-9B	35.5	74.2	74.5* (0.2)	32.7	48.2	57.3* (0.8)
Medical-domain Models

TABLE II : Performance Comparison on MCQ Datasets: MedQA and EMS-MCQA

General-purpose models
	Token F1			BLEU-2			ROUGE-L			BERTScore F1
Models	Base	UaS	SwS ( $\lambda$ )	Base	UaS	SwS ( $\lambda$ )	Base	UaS	SwS ( $\lambda$ )	Base	UaS	SwS ( $\lambda$ )
Llama-3.2-3B	39.6	48.7*	47.6 (0.4)	12.7	28.1*	27.1 (0.2)	26.4	34.4*	33.9 (0.2)	87.9	89.5*	89.4 (0.4)
Qwen3.5-4B	24.6	29.1*	28.0 (0.2)	6.3	10.5*	9.7 (0.2)	12.9	16.8*	16.3 (0.2)	83.7	85.1*	85.0 (0.2)
Qwen3.5-9B	26.3	29.7	30.4* (1.0)	8.2	11.4*	11.4* (0.4)	14.9	17.5	18.5* (1.0)	84.2	85.3*	85.3* (0.4)
Medical-domain models

TABLE III : Performance Comparison on Med-O1 Dataset

Category	#Q	Qwen3.5-4B			Meerkat-8B
		Base	UaS	SwS (0.8)	Base	UaS	SwS (0.4)
Neurology / Psychiatry	83	33.7	61.4	80.7	76.4	76.4	76.4
Endocrinology / Metabolism	67	23.9	65.7	82.1	83.1	81.4	81.4
Pediatrics	65	36.9	70.8	78.5	85.2	78.7	82.0
Cardiovascular	60	31.7	68.3	73.3	75.9	61.1	63.0
Infectious Diseases	58	37.9	67.2	82.8	75.6	71.1	71.1

TABLE IV : MedQA sub-domain accuracy for Qwen3.5-4B and Meerkat-8B.

Fig. 2 : Effect of

\lambda

on SwS performance for the top-performing models on MedQA and EMS-MCQA.

Fig. 3 : MCQ accuracy by generated-to-reference CoT length bin, shown under each model’s best-performing setting: Llama-3.2-3B/EMS-MCQA (SwS,

\lambda

=1.0), Qwen3.5-4B/MedQA (SwS,

\lambda

=0.8), Meerkat-8B/MedQA (Base), and Lingshu-7B/EMS-MCQA (SwS,

\lambda

=0.8). Percentages in parentheses on the x-axis labels indicate the share of the test dataset falling into each bin.

Fig. 4 : Med-O1 ROUGE-L by generated-to-reference CoT length bin, shown under each model’s best-performing setting (UaS): Llama-3.2-3B (left) and Meerkat-8B (right). Percentages in parentheses on the x-axis labels indicate the share of the test dataset falling into each bin.

Fine-Tuning

Fine-tuning adapts pretrained models to new tasks, domains, and behaviors by updating some or all of their parameters.

See all 255 papers

Apr 26, 2026

Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale

Practitioners deploying small open-weight large language models (LLMs) for medical question answering face a recurring design choice: invest in a domain-fine-tuned model, or keep a general-purpose model and inject domain knowledge at inference time via retrieval-augmented generation (RAG). We isolate this trade-off by holding model size, prompt template, decoding temperature, retrieval pipeline, and evaluation protocol fixed, and varying only (i) whether the model has been domain-adapted (Gemma 3 4B vs. MedGemma 4B, both 4-bit quantized and served via Ollama) and (ii) whether retrieved passages from a medical knowledge corpus are inserted into the prompt. We evaluate all four cells of this 2x2 design on the full MedQA-USMLE 4-option test split (1,273 questions) with three repetitions per question (15,276 LLM calls). Domain fine-tuning yields a +6.8 percentage-point gain in majority-vote accuracy over the general 4B baseline (53.3% vs. 46.4%, McNemar p < 10^-4). RAG over MedMCQA explanations does not produce a statistically significant gain in either model, and in the domain-tuned model the point estimate is slightly negative (-1.9 pp, p = 0.16). At this scale and on this benchmark, domain knowledge encoded in weights dominates domain knowledge supplied in context. We release the full experiment code and JSONL traces to support replication.

Avi-ad Avraam Buskila

Department of Information Science and Applied Artificial Intelligence, Bar-Ilan University, Ramat-Gan, Israel

Retrieval-Augmented Generation Healthcare LLM Fine-Tuning Question Answering Medical QA RAG Evaluation

Ask PDF

Feb 23, 2026

To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

Objective: To improve the efficiency of medical question answering (MedQA) with large language models (LLMs) by avoiding unnecessary reasoning while maintaining accuracy. Methods: We propose Selective Chain-of-Thought (Selective CoT), an inference-time strategy that first predicts whether a question requires reasoning and generates a rationale only when needed. Two open-source LLMs (Llama-3.1-8B and Qwen-2.5-7B) were evaluated on four biomedical QA benchmarks-HeadQA, MedQA-USMLE, MedMCQA, and PubMedQA. Metrics included accuracy, total generated tokens, and inference time. Results: Selective CoT reduced inference time by 13-45% and token usage by 8-47% with minimal accuracy loss (

\leq

4%). In some model-task pairs, it achieved both higher accuracy and greater efficiency than standard CoT. Compared with fixed-length CoT, Selective CoT reached similar or superior accuracy at substantially lower computational cost. Discussion: Selective CoT dynamically balances reasoning depth and efficiency by invoking explicit reasoning only when beneficial, reducing redundancy on recall-type questions while preserving interpretability. Conclusion: Selective CoT provides a simple, model-agnostic, and cost-effective approach for medical QA, aligning reasoning effort with question complexity to enhance real-world deployability of LLM-based clinical systems.

Zaifu Zhan, Min Zeng, Shuang Zhou +6

Department of Electrical and Computer Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States · Division of Computational Health Sciences, Department of Surgery, University of Minnesota, 420 Delaware St SE, 55455, Minneapolis, MN, United States · Department of Computer Science and Engineering, University of Minnesota, 200 Union St SE, 55455, Minneapolis, MN, United States +1

CoT Reasoning Efficient Language Model Reasoning LLM Inference Efficiency LLM Reasoning Biomedical QA

Ask PDF

Mar 26, 2026

When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models

Large Language Models (LLMs) are increasingly deployed in medical settings, yet their sensitivity to prompt formatting remains poorly characterized. We evaluate MedGemma (4B and 27B parameters) on MedMCQA (4,183 questions) and PubMedQA (1,000 questions) across a broad suite of robustness tests. Our experiments reveal several concerning findings. Chain-of-Thought (CoT) prompting decreases accuracy by 5.7% compared to direct answering. Few-shot examples degrade performance by 11.9% while increasing position bias from 0.14 to 0.47. Shuffling answer options causes the model to change predictions 59.1% of the time, with accuracy dropping up to 27.4 percentage points. Front-truncating context to 50% causes accuracy to plummet below the no-context baseline, yet back-truncation preserves 97% of full-context accuracy. We further show that cloze scoring (selecting the highest log-probability option token) achieves 51.8% (4B) and 64.5% (27B), surpassing all prompting strategies and revealing that models "know" more than their generated text shows. Permutation voting recovers 4 percentage points over single-ordering inference. These results demonstrate that prompt engineering techniques validated on general-purpose models do not transfer to domain-specific medical LLMs, and that reliable alternatives exist.

Binesh Sadanandan, Vahid Behzadan

SAIL Lab, University of New Haven, West Haven, CT 06516, USA

Prompt Sensitivity LLM Prompting Language Model Robustness Medical LLMs Clinical Language Model Evaluation Medical QA

Ask PDF

Abstract

Fine-Tuning

Explore similar work

Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale

To Reason or Not to: Selective Chain-of-Thought in Medical Question Answering

When Chain-of-Thought Backfires: Evaluating Prompt Sensitivity in Medical Language Models