Claim Automation using Large Language Model

Zhengda Mo; Zhiyu Quan; Eli O'Donohue; Kaiwen Zhong

cs.CLFeb 18, 2026

Claim Automation using Large Language Model

Authors: Zhengda Mo, Zhiyu Quan, Eli O'Donohue, Kaiwen Zhong

Organizations: Actuarial and Risk Management Sciences, University of Illinois at Urbana-Champaign · PCMI Corporation

Abstract

While Large Language Models (LLMs) have achieved strong performance on general-purpose language tasks, their deployment in regulated and data-sensitive domains, including insurance, remains limited. Leveraging millions of historical warranty claims, we propose a locally deployed governance-aware language modeling component that generates structured corrective-action recommendations from unstructured claim narratives. We fine-tune pretrained LLMs using Low-Rank Adaptation (LoRA), scoping the model to an initial decision module within the claim processing pipeline to speed up claim adjusters' decisions. We assess this module using a multi-dimensional evaluation framework that combines automated semantic similarity metrics with human evaluation, enabling a rigorous examination of both practical utility and predictive accuracy. Our results show that domain-specific fine-tuning substantially outperforms commercial general-purpose and prompt-based LLMs, with approximately 80% of the evaluated cases achieving near-identical matches to ground-truth corrective actions. Overall, this study provides both theoretical and empirical evidence to prove that domain-adaptive fine-tuning can align model output distributions more closely with real-world operational data, demonstrating its promise as a reliable and governable building block for insurance applications.

Figures & tables

Figure 1: Overview of the token-level generation architecture used for claim automation.

Figure 2: LoRA adaptation applies to a single projection matrix. The original weight matrix $W_{\mathrm{frozen}}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$ remains unchanged, while trainable matrices $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$ and $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$ introduce a low-rank update via $\Delta W=BA$ . The effective weight is $W_{\mathrm{frozen}}+\Delta W$ .

Figure 3: Self-attention module and FFN in transformer. Each matrix represented by a light blue parallelogram corresponds to a learnable projection matrix within the transformer block. These matrices are potential targets for LoRA adaptation, where low-rank updates can be applied individually to each matrix.

Metric	Family	What it measures
Edit distance similarity	Surface-level similarity	Character- or token-level similarity derived from edit distance between prediction and reference
Unigram precision	Surface-level similarity	1-gram overlap precision between prediction and reference
Bigram precision	Surface-level similarity	2-gram overlap precision between prediction and reference
Sentence BLEU	Surface-level similarity	Sentence-level n-gram overlap between prediction and reference (BLEU)
TF-IDF cosine similarity	Lexical–semantic similarity	Weighted lexical overlap in a bag-of-words vector space
BERT cosine similarity	Lexical–semantic similarity	Contextual sentence-level representation similarity

Table 1: Taxonomy of automated evaluation metrics used in this study

Score	Meaning	Description
1.0	Perfect Match	The predicted actions fully and precisely cover the ground-truth actions. No missing or incorrect actions.
0.8	Strong Match	The predicted actions mostly cover the ground truth, with only minor omissions or paraphrasing and no major errors.
0.6	Partial Match	The prediction covers some key actions but misses important parts. Still related and somewhat helpful.
0.4	Weak Match	Very limited overlap. Only a few actions (or fragments) are relevant to the ground truth.
0.2	Minimal Match	Prediction is largely irrelevant or incorrect, with almost no valid overlap with the ground truth.
0.0	No Match	Completely unrelated or invalid prediction, with no correspondence to the ground truth.

Table 2: Human evaluation scoring guidelines used in all annotation tasks.

Metric	Chatterjee $\xi$	Spearman $\rho$	Kendall $\tau$
BERT cosine similarity	0.692	0.733	0.607
LLM-as-a-Judge score	0.606	0.724	0.678
BLEURT (normalized)	0.576	0.703	0.566
TF-IDF cosine similarity	0.577	0.683	0.564
BERTScore (F1)	0.581	0.676	0.537
Edit distance similarity	0.530	0.655	0.526

Table 3: Correlation between different automated evaluation metrics and human judgments.

Model	Base model	Scale	Prompt	Fine-tuned	Training data
M1	DeepSeek-R1	8B	$\times$	$\times$	General-purpose data
M2	DeepSeek-R1	8B	$\checkmark$	$\times$	General-purpose data
M3	Qwen-Instruct	7B	$\checkmark$	$\times$	General-purpose data
M4	DeepSeek-R1	8B	$\times$	$\checkmark$	General-purpose + Domain-specific data

Table 4: Model configurations.

	Usability		Accuracy
Model	Format (%)	Validity (%)	Acc (Valid) (%)	Acc (HQ) (%)
M1: DeepSeek-R1	0.0	8.0	56.0	64.3
M2: DeepSeek-R1+ Prompt	6.5	29.5	64.4	71.2
M3: Qwen-Instruct+ Prompt	86.5	97.5	62.1	68.4
M4: DeepSeek-R1+ Fine-tune	100.0	100.0	81.5	92.0

Table 5: Main results on usability and accuracy.

Panel of Figure 4 — Figure 4: Semantic similarity distributions measured by BERT cosine similarity.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$n$	Input token length
$\bm{x}=(x_{1},x_{2},\ldots,x_{n})$	Input token sequence (after tokenization)
$T$	Output token length
$\bm{y}=(y_{1},y_{2},\ldots,y_{T})$	Output token sequence
$\bm{y}_{<t}=(y_{1},\ldots,y_{t-1})$	Previously generated output tokens until $t$
$\hat{\bm{y}}_{t}$	Predicted token distribution at step $t$

Appendix

Table 10

Model	Count	Mean	Std.	Median	Max	$p(\mathrm{Sim}_{\mathrm{BERT}}\geq\kappa)$
DeepSeek-R1 + Fine-tune	1500	0.869	0.159	0.929	1.000	0.791
Gemini-2.5-Flash	1500	0.799	0.160	0.857	0.977	0.685
Claude-Haiku-4.5	1500	0.757	0.170	0.803	0.974	0.575
GPT-4o-mini	1500	0.787	0.163	0.844	0.974	0.656
GPT-4.1	1500	0.749	0.163	0.793	0.977	0.561
GPT-5.2	1500	0.719	0.167	0.762	1.000	0.472

Appendix

Table 7: Summary statistics of BERT cosine similarity of DeepSeek-R1 + Fine-tune and SoTA commercial LLM on the full evaluation set (

N=1500

Model	Count	Mean	Std.	Median	Max	$p(\mathrm{Sim}_{\mathrm{BERT}}\geq\kappa)$
DeepSeek-R1 + Fine-tune	1334	0.899	0.132	0.946	1.000	0.863
Gemini-2.5-Flash	1334	0.817	0.146	0.867	0.977	0.733
Claude-Haiku-4.5	1334	0.776	0.159	0.819	0.970	0.624
GPT-4o-mini	1334	0.806	0.149	0.856	0.974	0.705
GPT-4.1	1334	0.767	0.152	0.809	0.977	0.605
GPT-5.2	1334	0.736	0.155	0.773	1.000	0.507

Appendix

Table 8: Summary statistics of BERT cosine similarity DeepSeek-R1 + Fine-tune and SoTA commercial LLM on the HQ subset (

N=1334

Figure 5: BERT cosine similarity: DeepSeek-R1 + Fine-tune versus Gemini-2.5-Flash on the full evaluation set and HQ subset.

Figure 6: BERT cosine similarity: DeepSeek-R1 + Fine-tune versus Claude Haiku 4.5 on the full evaluation set and HQ subset.

Figure 7: BERT cosine similarity: DeepSeek-R1 + Fine-tune model versus GPT-4.1 on the full evaluation set and HQ subset.

Figure 8: BERT cosine similarity: DeepSeek-R1 + Fine-tune model versus GPT-4o-mini on the full evaluation set and HQ subset.

Figure 9: BERT cosine similarity: DeepSeek-R1 + Fine-tune versus GPT-5.2 on the full valid set and HQ subset.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain
Oct 7, 2026Linqi Zhang, Chong Qi, Yan Cheng +4AI-Assisted Decision Making LLM Evaluation
School of Computer Science and Technology, Fudan University · School of Integrated Circuits, Nanjing University · School of Economics, Fudan University
An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs
Mar 15, 2026Qian Zhu, Xinnan Guo, Jingjing Huo +5Domain Adaptation for LLMs LLM Alignment
Ant Group
Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation
Mar 8, 2026David Beauchemin, Richard KhouryLLM Evaluation LLM Reliability
Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Quebec City, Quebec, Canada

Oct 7, 2026J/K move · Enter open · S save

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain

Linqi Zhang, Chong Qi, Yan Cheng, Wanqing Cao+3

School of Computer Science and Technology, Fudan University · School of Integrated Circuits, Nanjing University · School of Economics, Fudan University

AI-Assisted Decision Making LLM Evaluation LLM Reliability LLM Decision-Making

Open paper AskPDF arXiv

Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks. Insurance claim adjudication is one such task, requiring models to connect case evidence, insurance rules, intermediate judgments, and payout calculations across a structured decision process. We introduce InsClaimBench, an end-to-end benchmark for evaluating insurance claim adjudication across the decision chain. Grounded in real claim materials and structured insurance rules, InsClaimBench contains 3,780 cases in 375 case families across auto, property, and health insurance, comprising 86,656 atomic rule judgments. It evaluates each claim from atomic rules through adjudication modules to payout decisions and amounts, with controlled factual variants testing whether required changes are correctly propagated across levels. Evaluation of six LLMs reveals a progressive loss of reliability along the decision chain. Payout-decision accuracy ranges from 74.23--80.19%, while joint decision--amount accuracy drops to 47.54--73.15%. Strong local performance also fails to ensure case-level correctness: atomic-rule accuracy reaches 95.48%, whereas rule-vector exact match peaks at only 36.90%, and the most frequent module errors are not necessarily those most associated with final-decision failure. Under factual changes, these inconsistencies further become propagation failures: module updates are less reliable than rule updates, correct local judgments can still yield incorrect payouts, and correct payouts can conceal intermediate errors. These results show that reliable claim adjudication requires consistent composition and propagation across the decision chain.

Abstract

LLM Fine-Tuning

Explore similar work

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain

An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs

Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation