Text-based telemedicine has become an increasingly used mode of care, requiring clinicians to deliver medical advice clearly and effectively in writing. As platforms increasingly rely on patient ratings and feedback, clinicians face growing pressure to maintain satisfaction scores, even though these evaluations often reflect communication quality more than clinical accuracy. We analyse patient satisfaction signals in Romanian text-based telemedicine. Using a sample of anonymised text-based telemedicine consultations, we model feedback as a binary outcome, treating thumbs-up responses as positive and grouping negative or absent feedback into the other class. We extract from doctor responses interpretable, predominantly language-agnostic features (e.g., length, structural characteristics, readability proxies), along with Romanian LIWC psycholinguistic features and politeness/hedging markers where available. We train a classifier with a time-based split and perform SHAP-based analyses, which indicate that metadata dominates prediction, functioning as a strong prior, while characteristics of the response text provide a smaller but actionable signal. In subgroup correlation analyses, politeness and hedging are consistently associated with positive patient feedback, whereas lexical diversity shows a negative association.
Figures & tables
Subset
Split
ROC-AUC
PR-AUC
F1
Precision
Recall
Simple Questions
Val
0.6780
0.6865
0.6206
0.6206
0.6209
Only Clarifications
Val
0.7153
0.5857
0.6367
0.6411
0.6493
All Questions
Val
0.7192
0.6313
0.6518
0.6542
0.6577
New Patients
Val
0.6479
0.6003
0.5804
0.6061
0.5963
Returning Patients
Val
0.7656
0.6533
0.6951
0.6968
0.6937
Simple Questions
Val High-conf
0.6951
0.6138
0.6456
0.7724
0.6416
Table 1: Feedback prediction performance across question subsets and patient groups (New vs. Returning), reported on the validation split and a high-confidence subset.
Figure 1: Mean absolute SHAP values for the CatBoost feedback classifier. Larger values indicate greater impact on predictions (magnitude only; SHAP sign is not shown). Quantities are binned and are not processed in absolute numbers.
Figure 2: Pearson correlations between selected doctor-response text features and positive feedback, aggregated over all interactions. Positive values indicate features that tend to co-occur with thumbs-up feedback, while negative values indicate features that tend to co-occur with thumbs-down or missing feedback. Quantities are binned and are not processed in absolute numbers.
Figure 3: Mean absolute SHAP values for the most important doctor text features, computed separately for each medical question specialty. Bars show the response-text features with the largest contribution magnitude to the feedback prediction model within each specialty. Values indicate feature importance magnitude only and do not show effect direction. Quantities are binned and are not processed in absolute numbers.
Text-based telemedicine increasingly relies on lightweight patient feedback, however, such feedback primarily reflects perceived communication quality rather than medical accuracy. We introduce an LM-guided counterfactual recommendation pipeline that discovers and refines interpretable communication features such as tone, personalization, actionability and completeness in addressing patient concerns, without interfering with the medical content. These features are used together with patient-doctor interaction metadata to estimate positive feedback. At inference time, the system searches over low-cost ordinal feature changes and recommends minimal communication changes predicted to increase the probability of positive feedback, while independent auditor models test whether these gains generalize beyond the selection model. Across interactions, recommendations yield a mean +6.41% gain in predicted positive feedback probability under independent auditors, and are non-negative for 93.31% of recommendations. These results suggest that small, interpretable communication changes can capture most predicted gains while preserving the doctor's control over medical reasoning and final wording.
Adrian Cosma, Nicoleta-Nina Basoc, Andrei Niculae +2
IDSIA, Dalle Molle Institute for Artificial Intelligence · National University of Science and Technology POLITEHNICA Bucharest
As large language models (LLMs) are increasingly used in clinical settings, it is critical to evaluate their reliability under realistic variation in clinical text. We study this question in clinical triage, comparing LLMs to practicing physicians under text perturbations that preserve the underlying clinical setting. We introduce a benchmark of over 6,000 clinical scenarios, 7,000 physician annotations, and 225,000 model responses. Using this benchmark, we make two key observations. First, LLMs are more likely than physicians to recommend unnecessary care at baseline, and this tendency increases under perturbed inputs. Further, we find that LLM recommendations are more sensitive to gender and tone perturbations than human recommendations. Together, these results demonstrate that LLMs can vary under clinically irrelevant textual changes, highlighting the need for deployment-oriented evaluations grounded in expert physician behavior.
Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science · Worcester Polytechnic Institute, Department of Computer Science
Patient-voiced clinical-triage benchmarks report high under-triage rates for consumer LLMs for constrained multiple-choice output, yet the same cases score differently with free-text. We ask whether output format changes the model's clinical representation or only the mapping from a preserved representation to an answer. Using sparse-autoencoder (SAE) features in Gemma 3 4B/12B IT and Qwen3-8B, we find the same medical features fire on the shared clinical narrative under both formats but go {silent} at the multiple-choice decision token in all the cases at every model. Three independent methods (natural-language autoencoder verbalization, decision-token logit attribution, and top-feature characterization) agree that scaffold and format features, but not medical features, drive the decision logits. Behaviorally, the multiple-choice penalty inverts under both structured and natural-language input, option-order shuffle rules out positional bias, and the gap is dominated by off-by-one decision (the model picks an adjacent acuity letter to the gold answer) rather than knowledge failure. Thus, the failure originates in the output format and not in the clinical representation.
David Fraile Navarro, Berardino Como, Jialei Sheng +2
Macquarie University, Sydney, Australia · Politecnico di Bari, Bari, Italy · NSW Health, Sydney, Australia +1