Evaluating Large Language Models on Historical Health Crisis Knowledge in Resource-Limited Settings: A Hybrid Multi-Metric Study

Mohammed Rakibul Hasan

cs.CLMar 20, 2026

Evaluating Large Language Models on Historical Health Crisis Knowledge in Resource-Limited Settings: A Hybrid Multi-Metric Study

Authors: Mohammed Rakibul Hasan

Organizations: Department of Electrical & Computer Engineering, North South University, Dhaka, Bangladesh

Abstract

Large Language Models (LLMs) offer significant potential for delivering health information. However, their reliability in low-resource contexts remains uncertain. This study evaluates GPT-4, Gemini Pro, Llama~3, and Mistral-7B on health crisis-related enquiries concerning COVID-19, dengue, the Nipah virus, and Chikungunya in the low-resource context of Bangladesh. We constructed a question--answer dataset from authoritative sources and assessed model outputs through semantic similarity, expert-model cross-evaluation, and Natural Language Inference (NLI). Findings highlight both the strengths and limitations of LLMs in representing epidemiological history and health crisis knowledge, underscoring their promise and risks for informing policy in resource-constrained environments.

Figures & tables

	GPT-4	Gemini Pro	Llama-3	Mistral-7B
GPT-4	1.000	0.697	0.683	0.742
Gemini Pro	0.697	1.000	0.671	0.765
Llama-3	0.683	0.671	1.000	0.729
Mistral-7B	0.742	0.765	0.729	1.000

Table 1: Global inter-model mean cosine similarity matrix computed over all 100 benchmark questions. Values on the diagonal (self-similarity) are 1.000 by definition and are excluded from analysis.

Figure 2: Inter-model semantic similarity heatmap. Each cell shows the mean cosine similarity between the row and column model across all 100 questions, computed from all-MiniLM-L6-v2 embeddings (384-dimensional). The diagonal is masked. Colour scale: 0.50 (light) to 1.00 (dark blue). Mistral-7B–Gemini Pro is the highest-similarity pair ( $0.765$ ); Llama-3–Gemini Pro is the lowest ( $0.671$ ).

Figure 3: Divergence category distributions. Left (Fig. 3a): Horizontal bar chart of overall category counts across all 100 questions—82 Needs Review (orange), 18 High Divergence / Hallucination Likely (red), 0 Highly Consistent (green)—with percentage labels. Right (Fig. 3b): Stacked bar chart by disease domain showing that all 18 high-divergence questions belong exclusively to the COVID-19 domain ( $72\%$ ), while Dengue, Chikungunya, and Nipah Virus contain only Needs Review questions.

Figure 4: Per-question mean pairwise cosine similarity across all 100 benchmark questions, colour-coded by disease domain (Dengue: red, COVID-19: blue, Chikungunya: green, Nipah Virus: purple). Horizontal dashed lines mark the Highly Consistent ( $\geq 0.85$ , upper) and Needs Review ( $\geq 0.65$ , lower) thresholds. Vertical dotted lines delimit domain boundaries at questions 26, 51, and 76. All points below the lower threshold (questions 26–50) originate from the COVID-19 domain.

Figure 5: NLI label distribution by model as percentage of total sentences per model. Each model contributes 100 sentences. Bars are grouped by model on the x-axis; colours indicate label: Entailment (green), Neutral (orange), Contradiction (red). The legend is positioned outside the plot to the right. Note Gemini Pro’s uniquely bimodal pattern: highest entailment (22%) and highest contradiction (43%) simultaneously.

Model	NLI Mean	NLI Std	Entailment	Contradiction	Ref-Sim	HFS	HRS
GPT-4	2.24	0.976	0.00	0.38	0.7012	0.4091	0.5256
Gemini Pro	2.58	1.565	0.22	0.43	0.6723	0.4046	0.5528
Llama-3	2.50	0.870	0.00	0.25	0.7171	0.4532	0.5348
Mistral-7B	2.52	0.904	0.01	0.25	0.7514	0.5068	0.4786

Table 2: NLI response-level averages and hybrid evaluation scores by model. HFS = Hybrid Factual Score (0–1, higher is better); HRS = Hallucination Risk Score (0–1, lower is better). NLI mean ranges from 1.0 (all Contradiction ) to 5.0 (all Entailment ). NLI std reflects within-model variance across 100 sentences. Bold marks best per column.

Figure 6: Model comparison on composite hybrid metrics. Left (Fig. 6a): Mean Hybrid Factual Score (HFS, higher is better) with $\pm 1$ standard deviation error bars. Value annotations are placed above each bar. Mistral-7B leads ( $0.507$ ); Gemini Pro is lowest ( $0.405$ ). Right (Fig. 6b): Mean Hallucination Risk Score (HRS, lower is better) with the same error bars. Mistral-7B is lowest ( $0.479$ ); Gemini Pro is highest ( $0.553$ ). Model colours: GPT-4 blue, Gemini Pro green, Llama-3 orange, Mistral-7B purple.

Figure 7: Domain-wise mean Hybrid Factual Score (HFS) by model, displayed as grouped bars (four models per domain). The grey dashed horizontal line marks the overall mean HFS across all 400 responses. The legend is positioned outside the plot to the right. COVID-19 consistently produces the lowest HFS values across all four models, corroborating the elevated divergence rates in Section 4.2 . Dengue and Chikungunya yield the highest domain-level scores.

Model A	Model B	Cohen’s $d$	Magnitude
GPT-4	Mistral-7B	$-0.560$	Medium
Gemini Pro	Mistral-7B	$-0.494$	Small
Llama-3	Mistral-7B	$-0.341$	Small
GPT-4	Llama-3	$-0.258$	Small
Gemini Pro	Llama-3	$-0.238$	Small
GPT-4	Gemini Pro	$+0.021$	Small

Table 3: Pairwise Cohen’s

d

effect sizes for HFS.

Figure 8: Lower-triangular Pearson correlation matrix of eight evaluation metrics. Deep green cells indicate strong positive correlation; deep red indicates strong negative correlation; near-white indicates near-zero correlation. Key values: NLI Score–HFS: $r=0.906$ ; Ref-Response Sim–HFS: $r=0.735$ ; HRS–HFS: $r=-0.602$ ; Contradiction Rate–Judge Hallucination: $r=0.015$ (near-zero, confirming non-redundancy of the two hallucination paradigms).

LLM Reliability

Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.

See all 787 papers

Evaluating Health Misinformation in Low-Resource Languages: Integrating Small Language Models with a Culturally-Sensitive Responsible NLP Framework (Bangla as a Case Study)
Jul 14, 2026Farnaz Farid, Raihan Alam, Al Al-Areqi +5Small Language Models Automated Fact-Checking
School of Social Sciences, Western Sydney University · Microsoft · School of Computer, Data and Mathematical Sciences, Western Sydney University +4
Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare
Mar 1, 2026Shravani HariprasadClinical Language Model Evaluation Prompt Sensitivity
Independent Researcher
Toward Global Large Language Models in Medicine
Jan 5, 2026Rui Yang, Huitao Li, Weihao Xuan +47Medical LLMs Multilingual Language Model Evaluation

Jul 14, 2026J/K move · Enter open · S save

Evaluating Health Misinformation in Low-Resource Languages: Integrating Small Language Models with a Culturally-Sensitive Responsible NLP Framework (Bangla as a Case Study)

Farnaz Farid, Raihan Alam, Al Al-Areqi, Farhad Ahamed+4

School of Social Sciences, Western Sydney University · Microsoft · School of Computer, Data and Mathematical Sciences, Western Sydney University+4

Small Language Models Automated Fact-Checking Low-Resource Language Processing Health Misinformation Detection Responsible AI

Open paper AskPDF arXiv

Artificial Intelligence (AI) technologies, while serving as a foundational enabler for modern social media and digital health services, exert a bivalent effect by simultaneously acting as a combatant against and a spread vector for misinformation. A prevalent challenge in mitigating this issue arises in non-English contexts and low socioeconomic classes, where limited data hinders the training of AI models for effective detection. Consequently, culturally and linguistically diverse (CALD) communities struggle to access trustworthy health information through AI-driven tools. Current AI tools underperform due to a lack of training data and are largely unable to consider language nuances and traditions in non-English contexts. This research addresses these gaps by proposing a CALD-friendly AI-based health misinformation detector and providing a dashboard for medical professionals to analyse this misinformation, a critical step toward mitigating a growing concern among CALD populations. To this end, we conduct a series of experiments using a Bangla-translated health misinformation dataset to evaluate the performance of various Small Language Models (SLMs). SLMs are particularly relevant in this context given the frequent underperformance of Large Language Models (LLMs), which often stems from insufficient domain-specific knowledge and the prohibitive costs of resource-intensive fine-tuning. The results demonstrate that Phi-4 is the superior model, achieving an ideal balance between precision and recall in claim extraction. Then, to mitigate the limitations of SLMs, we design and test a novel health misinformation detection framework grounded in Responsible Natural Language Processing (NLP), which incorporates cultural sensitivity, potential for harm, and communication quality, thereby providing a holistic lens for evaluating misinformation in low-resource languages.

Abstract

LLM Reliability

Explore similar work

Evaluating Health Misinformation in Low-Resource Languages: Integrating Small Language Models with a Culturally-Sensitive Responsible NLP Framework (Bangla as a Case Study)

Prompt Sensitivity and Answer Consistency of Small Open-Source Language Models for Clinical Question Answering in Low-Resource Healthcare

Toward Global Large Language Models in Medicine