Effects of Transcript Compression on LLM-based Medical Misinformation Detection in Japanese YouTube Videos

Yuya Wake; Sho Tsugawa; Toshiyuki Amagasa

cs.CLSep 25, 2026

Effects of Transcript Compression on LLM-based Medical Misinformation Detection in Japanese YouTube Videos

Authors: Yuya Wake, Sho Tsugawa, Toshiyuki Amagasa

Organizations: University of Tsukuba, 1-1-1 Tennodai, Tsukuba, Ibaraki 305-8573, Japan

Abstract

Large language models (LLMs) are increasingly used to assess long-form medical videos, but their effectiveness may depend on whether transcripts are provided in full or compressed through summarization, retrieval, or claim screening. This study examines how such transcript compression affects LLM-based veracity classification of Japanese medical YouTube videos. We compare four transcript input designs: full transcripts, LLM-generated summaries, RAPTOR-based retrievalaugmented generation (RAG), and Screening, which extracts candidate medical and health-related sentences. Using 74 long-form videos labeled as Real or Fake, we evaluate classification performance and analyze linguistic changes using J-LIWC, hedge expressions, and institutional or technical terms. The full-transcript Baseline achieved the best performance, whereas all compressed inputs increased false negatives, meaning that Fake videos were more likely to be misclassified as Real. Summary caused the largest performance drop, while Screening performed best among the compressed inputs but still omitted many medically relevant sentences. Linguistic analyses showed that these errors were not explained by a simple increase in certainty. Instead, Summary reduced affective, social, temporal, cognitive, and conversational cues, while Summary and RAG made institutional and technical terms more salient. These findings suggest that transcript compression can represent Fake videos as more coherent and authoritative inputs, thereby weakening cues needed for misinformation detection

Figures & tables

Condition	Accuracy	F1-Score	MCC	TN	FN	FP	TP
Baseline	0.905	0.891	0.524	4	1	6	63
Summary	0.541	0.608	0.206	8	32	2	32
RAG	0.770	0.800	0.365	7	14	3	50
Screening	0.824	0.838	0.391	6	9	4	55

Table 1: Classification performance of each input design

Condition	Feature	Compressed Rate	$\Delta$	p-value
Summary	certain	0.0124	0.00372	$3.52\times 10^{-7}$
Summary	tentat	0.0119	0.000646	0.00665
RAG	certain	0.0117	0.000108	1.000
RAG	tentat	0.00943	0.0000842	0.797
Screening	certain	0.0104	-0.000203	0.894
Screening	tentat	0.00970	0.000181	0.575

Table 2: Differences in certain and tentat across conditions

Condition	Feature	Compressed Rate	$\Delta$	p-value
Summary	affect (affective processes)	0.0345	0.00942	$9.85\times 10^{-9}$
	posemo (positive emotion)	0.0192	0.00691	$8.13\times 10^{-8}$
	cogproc (cognitive processes)	0.0755	0.00610	$9.85\times 10^{-9}$
	relativ (relativity)	0.0624	0.00531	$9.85\times 10^{-9}$
	drives (drives)	0.0551	0.00423	$8.13\times 10^{-8}$
	time (time orientation)	0.0328	0.00341	$8.75\times 10^{-8}$

Table 3: Other significant J-LIWC features with relatively large mean differences

Type	Hedge expressions
Formal hedges	おそらく( probably ), 可能性( possibility ), と考えられる( it is considered that ), とは限らない( not necessarily ), 場合もある( in some cases )
Conversational hedges	たぶん( maybe/probably ), なんか( kind of ), 気がする( I feel that ), かも( might ), かな( I wonder ), みたいだ( seems like )

Table 4: List of formal and conversational hedges, shown as Japanese expression (English gloss).

Condition	Feature	Compressed Rate	$\Delta$	p-value
Summary	formal hedge rate	0.00523	-0.00289	$1.01\times 10^{-7}$
	conversational hedge rate	0.00896	0.0374	$4.01\times 10^{-13}$
	total hedge rate	0.0142	0.0346	$3.13\times 10^{-12}$
RAG	formal hedge rate	0.00282	$-3.06\times 10^{-6}$	0.592
	conversational hedge rate	0.0311	0.00728	$1.17\times 10^{-9}$
	total hedge rate	0.0339	0.00728	$1.17\times 10^{-9}$

Table 5: Changes in formal and conversational hedge rates across conditions

Condition	Feature	Compressed Rate	$\Delta$	p-value
Summary	total rate	0.00719	-0.00239	0.0377
	論文 ( paper ) rate	0.0000987	0.000199	0.000372
	病院 ( hospital ) rate	0.000181	0.000167	0.0377
RAG	total rate	0.00663	-0.00203	0.0241

Table 6: Significant institutional and technical term rates across conditions

TP	FP	FN	Precision	Recall	F1-Score
192	55	653	0.777	0.227	0.352

Table 7: Overall extraction performance of the Screening Condition

Health Misinformation Detection

See all 9 papers

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution
Mar 5, 2026Qiao Jin, Yin Fang, Lauren He +12LLM Answer Verification Healthcare
Division of Intramural Research, National Library of Medicine, National Institutes of Health · Department of Computer Science, University of Virginia · Center for Cancer Research, National Cancer Institute, National Institutes of Health +3
Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit
Jun 2, 2026JooYoung Lee, Lin Tian, Angela Brillantes +2LLM Evaluation Zero-Shot Text Classification
University of Technology Sydney, Sydney, Australia.
CARE: A Conformal Safety Layer for Medical Summarization
Jun 8, 2026Suhana Bedi, Bridget Lin, Anson Y. Zhou +5Text Summarization Conformal Risk Control
Department of Biomedical Data Science, Stanford University · Department of Anesthesiology, Perioperative and Pain Medicine, Stanford University · Department of Computer Science, Stanford University +2

Mar 5, 2026J/K move · Enter open · S save

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

Qiao Jin, Yin Fang, Lauren He, Yifan Yang+11

Division of Intramural Research, National Library of Medicine, National Institutes of Health · Department of Computer Science, University of Virginia · Center for Cancer Research, National Cancer Institute, National Institutes of Health+3

LLM Answer Verification Healthcare Small Language Models LLM Hallucination Detection Medical LLMs Scientific Claim Verification

Open paper AskPDF arXiv

Assessing whether an article supports an assertion is essential for hallucination detection and claim verification. While large language models (LLMs) have the potential to automate this task, achieving strong performance requires frontier models such as GPT-5 that are prohibitively expensive to deploy at scale. To efficiently perform biomedical evidence attribution, we present Med-V1, a family of small language models with only three billion parameters. Trained on high-quality synthetic data newly developed in this study, Med-V1 substantially outperforms (+27.0% to +71.3%) its base models on five biomedical benchmarks unified into a verification format. Despite its smaller size, Med-V1 performs comparably to frontier LLMs such as GPT-5, along with high-quality explanations for its predictions. We use Med-V1 to conduct a first-of-its-kind use case study that quantifies hallucinations in LLM-generated answers under different citation instructions. Results show that the format instruction strongly affects citation validity and hallucination, with GPT-5 generating more claims but exhibiting hallucination rates similar to GPT-4o. Additionally, we present a second use case showing that Med-V1 can automatically identify high-stakes evidence misattributions in clinical practice guidelines, revealing potentially negative public health impacts that are otherwise challenging to identify at scale. Overall, Med-V1 provides an efficient and accurate lightweight alternative to frontier LLMs for practical, real-world biomedical evidence attribution. Med-V1 is available at https://github.com/NLM-DIR/Med-V1.

Abstract

Health Misinformation Detection

Explore similar work

Med-V1: Small Language Models for Zero-shot and Scalable Biomedical Evidence Attribution

Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit

CARE: A Conformal Safety Layer for Medical Summarization