PAINT: A Framework for Multilingual Pain Representation from Medical Licensing Exams to Real-World Clinical Texts
Authors: Boya Zhang, Yuanchao Ma, Jiageng Wu, Krittaphas Chaisutyakorn, Rodrigo Gameiro, Ahram Han, Chien-Yu Huang, Nikhil Jaiswal, Marianna Leite, Miriam Mora-Mau, Jaime Pineda-Moreno, Siena Placino, Ryohei Yamamoto, Yugang Jia, Jie Yang, Leo Anthony Celi, Douglas Teodoro
Organizations: University of Geneva · Massachusetts Institute of Technology · McGill University · Harvard Medical School · Brigham and Women’s Hospital · Harvard T.H. Chan School of Public Health · Mahidol University · Seoul National University Hospital · National Taiwan University · Universidade de São Paulo · Santa Marcelina College of Medicine · Universidad Peruana Cayetano Heredia · HM Hospitales · St. Luke’s Medical Center College of Medicine · Fukushima Medical University · Beth Israel Deaconess Medical Center
Pain is one of the most common and clinically important symptoms, reflecting disease onset, progression, and treatment response. However, pain is difficult to accurately capture and interpret from medical text, as its expression varies across languages, documentation types, and patient demographics, and is often embedded in terminology, negation, temporality, and contextual descriptions. Existing pain-related studies remain largely monolingual or limited to narrow clinical settings. In this study, we introduce PAINT, a multilingual framework for studying pain representation across medical exams and real-world clinical texts. PAINT includes nine pain-focused dataset partitions spanning English, Spanish, Portuguese, and Japanese, with clinician annotations in all four languages. We curated a structured schema covering pain relevance, pain presence, anatomical location, pain descriptors, and demographic information. To capture multilingual pain representations, we leverage large language models (LLMs) and adopt an ensemble strategy to improve performance. Ten clinicians across four languages validate the labels: the ensemble reaches macro-F1 of 0.91-1.00 on pain relevance and pain presence and 92-100% accuracy on demographics, with inter-annotator κ of 0.82-1.00. We release the WorldMedQA-Pain ensemble labels, the extraction pipeline, validation scripts, and clinician annotation sheets for publicly licensed corpora, providing a foundation for pain-centered clinical NLP and for studying pain documentation across languages and healthcare contexts.
Figures & tables
Figure 1: Overview of PAINT: medical licensing examinations and clinical text in four languages, annotated with an ensemble of open-weight language models and validated by clinicians.
Lang
Pain mention in source
Reason
en
“She has no pain”
denial
es
“sin fiebre ni dolor” (no fever or pain)
denial
pt
“nódulo indolor” (painless nodule)
painless
ja
\jpfont 右陰囊の無痛性腫大 (painless swelling)
\jpfont 無痛性 ( painless )
Table 1: Examples where pain is mentioned but not asserted in the patient, the mention-without-presence case, agreed by all three models. The trace is an explicit denial or a painless -type derivative. Quotes are excerpts from the source text.
Brazil
Japan
Spain
UK
Model
Rel.
Pres.
Rel.
Pres.
Rel.
Pres.
Rel.
Pres.
Independent evaluation
Gemma-4-E4B
0.894
0.872
0.972
0.951
0.905
0.921
0.946
0.936
Qwen3-8B
0.879
0.873
0.948
0.956
0.913
0.917
0.931
0.932
DeepSeek-R1-Distill-Qwen-32B
0.870
0.872
0.936
0.965
0.889
0.892
0.935
0.949
LLaMA-3.1-8B
0.668
0.744
0.657
0.751
0.674
0.738
0.718
0.810
Table 2: Macro F1 for pain relevance and pain presence on WorldMedQA-Pain. “Rel.” and “Pres.” denote pain relevance and pain presence, respectively. Independent reports each model evaluated on the full corpus. Ensemble reports the ensemble, and GPT-5 for comparison, evaluated on the consensus subset, consisting of records on which all three selected models agree on both pain fields. Coverage is the fraction of the full corpus retained in the consensus subset. GPT-5 † is included as a proprietary reference.
Brazil
Japan
Spain
UK
Model
Rel.
Pres.
Rel.
Pres.
Rel.
Pres.
Rel.
Pres.
Gemma-4-E4B
0.68
0.46
0.81
0.50
0.69
0.62
0.79
0.56
Qwen3-8B
0.56
0.50
0.60
0.61
0.76
0.61
0.62
0.53
DeepSeek-R1-Distill-Qwen-32B
0.46
0.56
0.49
0.75
0.53
0.47
0.62
0.75
2-of-3 majority vote
0.60
0.53
0.70
0.71
0.81
0.68
0.75
0.73
GPT-5 †
0.80
0.64
0.98
0.73
0.85
0.68
0.83
0.63
Table 3: Pain macro-F1 on the non-consensus, or abstain, rows of WorldMedQA-Pain, for each model, the 2-of-3 majority vote, and GPT-5 † . All systems score below the consensus macro-F1 of 0.92 to 0.99 in Table 2 , most on pain-presence. Best per column in bold ; † proprietary reference.
Field
Task
Definition
Input
Output
Metric
Pain relevance
IE
Pain-related concepts
Text
True / False
Macro-F1
Pain presence
IE
Pain assertion in the patient
Text
True / False
Macro-F1
Age
IE
Patient’s age
Text
Number
Accuracy
Gender
IE
Patient’s gender
Text
Male / Female
Accuracy
Occupation
IE
Patient’s occupation
Text
Text
Accuracy
Table 4: The PAINT extraction pipeline, with each field’s task, definition, input, output, and metric. IE: information extraction.
Pain
Demographics
Dataset
Rel.
Pres.
Age
Gen.
Occ.
(a) Ensemble vs. clinician (F1 / acc)
MEDIQA
0.91
0.96
99
99
99
MIMIC-III
0.95
0.96
99
100
96
Cantemist
0.98
1.00
100
97
97
IFMIR
1.00
1.00
92
95
97
Table 5: Clinician validation on BRIDGE-Pain, with 100 records per clinician and a 50-record overlap per dataset. Panel (a): ensemble versus clinician annotations; macro-F1 is reported for pain relevance and pain presence, and accuracy for demographic extraction. Panel (b): agreement between the two clinicians on the overlapping subset; Cohen’s κ is reported for the binary pain fields and percentage agreement for demographic fields.
Split
Lang
n
PP-res.
Pain-pres.
Age
Gender
Ethnicity
Occupation
WorldMedQA-Pain (exam corpus, n=2,963 )
Brazil REVALIDA
pt
904
804 (88.9%)
271 (33.7%)
83.1%
84.6%
1.9%
9.0%
Japan NLE
ja
742
689 (92.9%)
112 (16.3%)
49.8%
49.9%
0.6%
5.2%
Spain MIR
es
1,049
969 (92.4%)
136 (14.0%)
44.3%
42.6%
0.4%
2.0%
UK PLAB
en
268
244 (91.0%)
90 (36.9%)
95.5%
98.4%
0.8%
2.1%
BRIDGE-Pain (clinical text, n=17,795 )
Table 6: Sample budget per split. n : extracted records. PP-res. : rows where all three models agree on both pain fields. Pain-pres. : pain-presence-positive rows within PP-resolved. Demographic columns give ensemble coverage within the PP-resolved rows. The three largest partitions are down-sampled to 5,000; Cantemist and MEDIQA are in full.
Brazil
Japan
Spain
UK
Language
pt
ja
es
en
Exam
REVALIDA
NLE
MIR
PLAB
Years
2011–23
2022–24
2018–23
2025
n items
904
742
1,049
268
Pain-rel. %
36.5
17.5
15.9
38.8
Pain-pres. %
30.0
15.1
13.0
33.6
Table 7: WorldMedQA-Pain per-partition statistics. Pain rates are from the three-model ensemble; clinician-annotated labels only validate it, see Appendix I and Table 2 . The Location column is the share of items with a non-empty pain-location.
Appendix figures & tables18 assets
Supplementary material from the paper’s appendix.
Appendix
L.
Frequent pain descriptor tokens (count across WorldMedQA-Pain + BRIDGE-Pain)
Table 8: Frequent pain descriptor tokens per language, across both corpora. Japanese descriptors fuse with the anatomical site or pain-stem; the other languages use an independent quality channel. Counts for BRIDGE-Pain are aggregate statistics only, as the underlying text is not redistributed.
Figure 2: Age composition per dataset. Each cell shows the bin’s sample size; colour encodes its share of the dataset, and rows sum to 100%. Total is the age-labelled count within the pain-consensus subset.
Figure 3: Pain extraction prompt, English. The same four-step template, hierarchical constraints, and FINAL_JSON schema are reused across all four languages; only the in-language lexicon and worked examples differ, see Figs. 5 – 7 .
Figure 4: Demographic extraction prompt, English. The same template and never-infer policy are reused in the native-language variants with locally-attested surface forms substituted in.
Figure 5: Japanese pain-prompt substitutions: Step 1 lexicon and Step 2–4 worked examples. The rest of the template, Fig. 3 , is unchanged.
Table 9: BRIDGE-Pain validation pooled across the five datasets, on records where the two clinicians agree, about 750 each. Pain fields give the ensemble macro-F1 against clinician annotation; demographic fields give clinician-confirmed accuracy with a Wilson 95% confidence interval. Per-dataset values are in Table 5 .
Split
Age
Gen.
Occ.
Clinician-confirmed accuracy (%)
UK PLAB
100
100
100
Spain MIR
100
96
100
Japan NLE
100
100
100
Brazil REVALIDA
92
98
96
Inter-annotator agreement (%)
Appendix
Table 10: WorldMedQA-Pain demographic validation, 30 records per partition with an overlap of 10 for agreement. Pain fields carry clinician-annotated labels, Table 2 , and are not re-annotated here. Accuracy is over the 50 unique records per partition. Cohen’s κ on the 10-record overlaps, where defined, is 0.82–1.00, lowest for Brazil REVALIDA gender (0.82) and UK PLAB age (0.89).
Dataset
Field
#cat
%
pe
κ
MEDIQA
age
6
100
0.81
1.00
gender
3
100
0.52
1.00
occupation
6
98
0.83
0.88
MIMIC-III
age
30
100
0.07
1.00
gender
2
100
0.50
1.00
occupation
21
98
0.30
0.97
Appendix
Table 11: Demographic inter-annotator agreement on the 50-record BRIDGE-Pain overlaps: number of distinct categories, percent agreement, expected chance agreement pe , and Cohen’s κ . κ is undefined (—) where the overlap contains a single category, so pe=1 ; it is near-uninformative where pe is close to 1 (IFMIR age). For IFMIR gender and occupation, both clinicians record every record in the 50-record overlap as null and κ is undefined on a single-category field. Where pe is small, as for age on MIMIC-III, Cantemist, and Brateca with 30–37 distinct values, perfect agreement is the condition under which κ is least inflated.
Dataset
Field
Clin. A
Clin. B
MEDIQA
pain-relevance
false
true
MEDIQA
pain-relevance
false
true
MEDIQA
occupation
null
registered nurse
MIMIC-III
pain-relevance
false
true
MIMIC-III
pain-relevance
false
true
MIMIC-III
pain-relevance
false
true
Appendix
Table 12: Every disagreement across all double-annotated overlaps ( n=17 of 1,370 judgements). MEDIQA pain-relevance disagreements arise on doctor–patient dialogue turns where intent must be judged; the MIMIC-III pain disagreements concern whether pain elicited during a neurological examination counts as a pain mention; the IFMIR case turns on whether \jpfont 気分不快 (malaise) is a pain term; Brateca pain-presence disagreements concern assertion strength. The Brateca age/gender pair is one 9,400-character record whose header states Age: 38 … Sex: explicitly; clinician B marked both fields as not stated, a reading slip on a long record rather than schema ambiguity. In the UK PLAB age case, clinician B confirmed the empty extraction and clinician A entered 0. No judgement was altered after the fact.
Field
Agree
Disagree
Marginals (A)
pain-relevance
50
0
38 true / 12 false
pain-presence
50
0
35 true / 15 false
gender
50
0
29 female / 21 male
age
50
0
35 distinct values
occupation
50
0
5 stated / 45 null
Appendix
Table 13: Cantemist confusion matrices on the 50-record overlap, all diagonal. Marginals are close to balanced for the binary fields ( pe=0.51 for gender, essentially the minimum for a binary field) and negligible for age, so the agreement is not an artefact of a degenerate label distribution. Cantemist consists of Spanish oncology case reports written to a rigid template in which age and sex are stated verbatim in the opening clause (“Mujer de 45 años sin alergias…”), leaving nothing to infer. Agreement between annotators does not mean agreement with the ensemble: both clinicians independently rejected three ensemble values (record 1147 pain-relevance false → true; 1217 gender female → male; 949 occupation “exfumadora” → none), which is why panel (a) of Table 5 is below 1.00 where panel (b) is not.
Column
Filled by
Content
<field>__ consensus
ensemble
silver label from the three-model consensus, pre-filled
<field>__ correct(TRUE/ FALSE)
annotator
whether the ensemble value is correct
<field>__ true_value_ if_false
annotator
the correct value, supplied when FALSE
Appendix
Table 14: Per-field columns of every annotation sheet.
Partition
Lang
Clin.
Sent
Returned
MEDIQA
en
P1
21 May
21 May
MEDIQA
en
P2
21 May
22 May
MIMIC-III
en
P3
21 May
22 May
MIMIC-III
en
P4
21 May
22 May
Cantemist
es
P5
9 Jul
10 Jul
Cantemist
es
P6
21 May
22 May
Appendix
Table 15: Date each annotation sheet was sent to and returned by each clinician, 2026. Upper block: BRIDGE-Pain; lower block: WorldMedQA-Pain.
We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine. HealMed contains 1,000 examples in each of nine languages, drawn from nine datasets and covering three task formats: MCQA, NLI and open-ended QA. The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions. Each translation was evaluated and revised by two experts fluent in English and the corresponding target language. On HealMed, performance declined most in low-resource languages, although the size of the gap varied markedly across languages and models. The strongest proprietary models were the most stable across languages, whereas many open-source and medically specialized models showed larger and less consistent gaps. Medical specialization alone did not ensure multilingual robustness. Furthermore, expert revision could either raise or lower measured performance, indicating that translation quality materially affects cross-language evaluation results.
Large Language Models (LLMs) are increasingly proposed for clinical decision support including multilingual diagnosis in low-resource settings. However, their reliability, calibration and safety characteristics remain insufficiently understood for structured, high-risk tasks. We present a system-level analysis of multilingual orthopedic diagnosis from free-text clinical notes in English, Hindi and Punjabi. We evaluate three modeling regimes: (i) task-aligned multilingual transformer encoders, (ii) a task-fine-tuned baseline (DistilBERT), and (iii) a domain-adaptive architecture tailored to orthopedic text (IndicBERT-HPA). These models are compared with zero-shot, instruction-tuned LLMs to assess suitability for structured diagnostic classification. Results indicate that while LLMs exhibit strong linguistic fluency, they show unstable calibration and reduced reliability under structured multilingual conditions, particularly in low-resource languages. These findings are specific to zero-shot evaluation and do not imply limitations of fine-tuned models. Domain-adaptive specialization substantially improves cross-lingual discrimination and confidence behavior. IndicBERT-HPA, with language-specific orthopedic adapter heads achieves consistently strong performance across six diagnostic categories and more predictable deployment characteristics than task-only adaptation. Building on these observations, we outline a conceptual deterministic agent-based validation framework for future implementation, formalizing evidence checks, language-sensitive validation and conservative human-in-the-loop gating. Reliable multilingual clinical decision support requires specialized architecture, explicit reliability analysis, and structured validation for safety-critical systems.
Danish Ali, Li Xiaojian, Sundas Iqbal +1
School of Computer Science, Wuhan University, China · School of Software, Nanjing University of Information Science and Technology (NUIST), China · Department of Orthopedic, Bahawal Victoria Hospital, Pakistan
Large Language Models (LLMs) with reasoning capabilities have recently demonstrated strong potential in medical Question Answering (QA). Existing approaches are largely English-focused and primarily rely on distillation from general-purpose LLMs, raising concerns about the reliability of their medical knowledge. In this work, we present a method to generate multilingual reasoning traces based on medical knowledge extracted from Wikipedia. We produce 500k traces in English, Italian, and Spanish, using a retrieval-augmented generation approach over medical information from Wikipedia. The traces are generated to solve medical questions drawn from MedQA and MedMCQA, which we extend to Italian and Spanish. We test our pipeline in both in-domain and out-of-domain settings across Medical QA benchmarks, and demonstrate that our reasoning traces improve performance both when utilized via in-context learning (few-shot) and supervised fine-tuning, yielding state-of-the-art results among 8B-parameter LLMs. We believe that these resources can support the development of more transparent clinical decision-support tools in multilingual settings. We release the full suite of resources: reasoning traces, translated QA datasets, Medical-Wikipedia, and fine-tuned models.
Pietro Ferrazzi, Aitor Soroa, Rodrigo Agerri
Fondazione Bruno Kessler, Trento, Italy · HiTZ Center - Ixa, University of the Basque Country EHU, Spain