FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts

Sylvia Vassileva; Ivan Koychev; Svetla Boytcheva

cs.CLApr 7, 2026

FMI@SU ToxHabits: Evaluating LLMs Performance on Toxic Habit Extraction in Spanish Clinical Texts

Authors: Sylvia Vassileva, Ivan Koychev, Svetla Boytcheva

Organizations: Faculty of Mathematics and Informatics, Sofia University St. Kliment Ohridski, Sofia, Bulgaria · Graphwise, Sofia, Bulgaria

Abstract

The paper presents an approach for the recognition of toxic habits named entities in Spanish clinical texts. The approach was developed for the ToxHabits Shared Task. Our team participated in subtask 1, which aims to detect substance use and abuse mentions in clinical case reports and classify them in four categories (Tobacco, Alcohol, Cannabis, and Drug). We explored various methods of utilizing LLMs for the task, including zero-shot, few-shot, and prompt optimization, and found that GPT-4.1's few-shot prompting performed the best in our experiments. Our method achieved an F1 score of 0.65 on the test set, demonstrating a promising result for recognizing named entities in languages other than English.

Figures & tables

Characteristic	Train	Dev	Test
# Mentions	5,754	1,838	-
# Documents	900	300	300
# Sentences	28,971	8,939	8,626

Table 1: Dataset statistics for substask 1.

Dataset	Tobacco	Alcohol	Cannabis	Drug
Train set	640	948	599	3,567
Dev set	193	332	194	1,119

Table 2: Trigger mentions per type on train and dev sets.

Figure 1: The process for mention extraction using an LLM.

Model	Precision	Recall	F1
Dictionary Baseline	0.4936	0.8270	0.6182
Spanish Clinical RoBERTa	0.7885	0.7933	0.7909
GPT-4.1 zero-shot	0.4942	0.6860	0.5745
GPT-4.1 few-shot (3 examples)	0.5064	0.6766	0.5793
GPT-4.1 few-shot (5 examples)	0.5687	0.7181	0.6347
GPT-4.1 few-shot (7 examples)	0.5712	0.7247	0.6389

Table 3: Results of the models for the NER task on the dev set.

Model	GC	GCT
Dictionary Baseline	0.8374	0.8302
Spanish Clinical RoBERTa	0.8203	0.8109
GPT-4.1 zero-shot	0.8308	0.8126
GPT-4.1 few-shot (3 examples)	0.8076	0.7888
GPT-4.1 few-shot (5 examples)	0.7888	0.7716
GPT-4.1 few-shot (7 examples)	0.7960	0.7783

Table 4: Evaluation of the models on the dev set on metrics reflecting the overlap with the gold entities: GC - G old entity is C ontained in a predicted entity, GCT - G old entity is C ontained in a predicted entity and has the correct T ype.

Model	Precision	Recall	F1
Dictionary Baseline	0.46	0.87	0.6
GPT-4.1 zero-shot	0.49	0.68	0.57
GPT-4.1 few-shot	0.59	0.72	0.65
Dictionary+GPT-4.1 zero-shot	0.43	0.83	0.57
Dictionary+GPT-4.1 few-shot	0.44	0.83	0.57

Table 5: Results of the submitted models for the NER task on the test set.

Predicted phrase	Correct Phrase
tabaquismo activo	tabaquismo
fumaba mucho	fumaba
cocaína esnifada	cocaína
consumo abusivo de alcohol	alcohol
fuma hachís	hachís
ex fumador	fumador

Table 6: Examples of errors in the predicted vs the correct phrase.

Clinical Information Extraction

Clinical Information Extraction turns clinical notes, reports, and patient narratives into structured, searchable facts and relations.

See all 106 papers

Detoxify: A framework for abusive text transformation using LLMs
Jul 14, 2025Rohitash Chandra, Jiyong Choi, Jayesh SonawaneLLM Evaluation Text Detoxification
Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla
Prompt PerturbAItions: Mechanisms of LLM Response Inconsistencies
May 29, 2026Soorya Ram Shimgekar, Agam Goyal, Amruta Parulekar +6Prompt Sensitivity LLM Interpretability
University of Illinois Urbana-Champaign · Nimblemind
Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters
Aug 23, 2026Philipp Steigerwald, Eric Rudolph, Jens AlbrechtLanguage Model Ensembles
Technische Hochschule Nürnberg Georg Simon Ohm

Jul 14, 2025J/K move · Enter open · S save

Detoxify: A framework for abusive text transformation using LLMs

Rohitash Chandra, Jiyong Choi, Jayesh Sonawane

Transitional Artificial Intelligence Research Group, School of Mathematics and Statistics, UNSW, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla Institute, Sydney, Australia · Centre for Artificial Intelligence and Innovation, Pingla

LLM Evaluation Text Detoxification

Open paper AskPDF arXiv

Although Large Language Models (LLMs) have demonstrated significant advancements in natural language processing tasks, their effectiveness in the classification and transformation of abusive text into non-abusive versions remains an area for exploration. In this study, we present Detoxify: a framework that employs LLMs to transform abusive text (tweets and reviews) containing hate speech and profanity into non-abusive text while retaining the original intent. We evaluate the performance of four state-of-the-art LLMs, such as Gemini, GPT-4o, DeekSeek and Groq, on their ability to identify abusive text. We aim to transform and obtain a text that is clean of abusive and inappropriate content, but maintains a similar level of sentiment and semantics, i.e. the transformed text needs to maintain its message. Afterwards, we evaluate the raw and transformed datasets with sentiment analysis and semantic analysis. Our results show Groq provides vastly different results when compared with other LLMs. We have identified similarities between GPT-4o and DeepSeek. Groq stood out as the most distinct, as it often restructured sentences with excessive positive phrasing, with the original context lost or altered.

Abstract

Clinical Information Extraction

Explore similar work

Detoxify: A framework for abusive text transformation using LLMs

Prompt PerturbAItions: Mechanisms of LLM Response Inconsistencies

Nürnberg NLP @ GermEval Shared Task 2026: Harmful Content Detection in German Social Media through Error-Independent LLM Voters