Claim Automation using Large Language Model

Zhengda Mo; Zhiyu Quan; Eli O'Donohue; Kaiwen Zhong

cs.CLFeb 18, 2026

Claim Automation using Large Language Model

Authors: Zhengda Mo, Zhiyu Quan, Eli O'Donohue, Kaiwen Zhong

Organizations: Actuarial and Risk Management Sciences, University of Illinois at Urbana-Champaign · PCMI Corporation

Abstract

While Large Language Models (LLMs) have achieved strong performance on general-purpose language tasks, their deployment in regulated and data-sensitive domains, including insurance, remains limited. Leveraging millions of historical warranty claims, we propose a locally deployed governance-aware language modeling component that generates structured corrective-action recommendations from unstructured claim narratives. We fine-tune pretrained LLMs using Low-Rank Adaptation (LoRA), scoping the model to an initial decision module within the claim processing pipeline to speed up claim adjusters' decisions. We assess this module using a multi-dimensional evaluation framework that combines automated semantic similarity metrics with human evaluation, enabling a rigorous examination of both practical utility and predictive accuracy. Our results show that domain-specific fine-tuning substantially outperforms commercial general-purpose and prompt-based LLMs, with approximately 80% of the evaluated cases achieving near-identical matches to ground-truth corrective actions. Overall, this study provides both theoretical and empirical evidence to prove that domain-adaptive fine-tuning can align model output distributions more closely with real-world operational data, demonstrating its promise as a reliable and governable building block for insurance applications.

Figures & tables

Figure 1: Overview of the token-level generation architecture used for claim automation.

Figure 2: LoRA adaptation applies to a single projection matrix. The original weight matrix $W_{\mathrm{frozen}}\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$ remains unchanged, while trainable matrices $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$ and $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$ introduce a low-rank update via $\Delta W=BA$ . The effective weight is $W_{\mathrm{frozen}}+\Delta W$ .

Figure 3: Self-attention module and FFN in transformer. Each matrix represented by a light blue parallelogram corresponds to a learnable projection matrix within the transformer block. These matrices are potential targets for LoRA adaptation, where low-rank updates can be applied individually to each matrix.

Metric	Family	What it measures
Edit distance similarity	Surface-level similarity	Character- or token-level similarity derived from edit distance between prediction and reference
Unigram precision	Surface-level similarity	1-gram overlap precision between prediction and reference
Bigram precision	Surface-level similarity	2-gram overlap precision between prediction and reference
Sentence BLEU	Surface-level similarity	Sentence-level n-gram overlap between prediction and reference (BLEU)
TF-IDF cosine similarity	Lexical–semantic similarity	Weighted lexical overlap in a bag-of-words vector space
BERT cosine similarity	Lexical–semantic similarity	Contextual sentence-level representation similarity

Table 1: Taxonomy of automated evaluation metrics used in this study

Score	Meaning	Description
1.0	Perfect Match	The predicted actions fully and precisely cover the ground-truth actions. No missing or incorrect actions.
0.8	Strong Match	The predicted actions mostly cover the ground truth, with only minor omissions or paraphrasing and no major errors.
0.6	Partial Match	The prediction covers some key actions but misses important parts. Still related and somewhat helpful.
0.4	Weak Match	Very limited overlap. Only a few actions (or fragments) are relevant to the ground truth.
0.2	Minimal Match	Prediction is largely irrelevant or incorrect, with almost no valid overlap with the ground truth.
0.0	No Match	Completely unrelated or invalid prediction, with no correspondence to the ground truth.

Table 2: Human evaluation scoring guidelines used in all annotation tasks.

Metric	Chatterjee $\xi$	Spearman $\rho$	Kendall $\tau$
BERT cosine similarity	0.692	0.733	0.607
LLM-as-a-Judge score	0.606	0.724	0.678
BLEURT (normalized)	0.576	0.703	0.566
TF-IDF cosine similarity	0.577	0.683	0.564
BERTScore (F1)	0.581	0.676	0.537
Edit distance similarity	0.530	0.655	0.526

Table 3: Correlation between different automated evaluation metrics and human judgments.

Model	Base model	Scale	Prompt	Fine-tuned	Training data
M1	DeepSeek-R1	8B	$\times$	$\times$	General-purpose data
M2	DeepSeek-R1	8B	$\checkmark$	$\times$	General-purpose data
M3	Qwen-Instruct	7B	$\checkmark$	$\times$	General-purpose data
M4	DeepSeek-R1	8B	$\times$	$\checkmark$	General-purpose + Domain-specific data

Table 4: Model configurations.

	Usability		Accuracy
Model	Format (%)	Validity (%)	Acc (Valid) (%)	Acc (HQ) (%)
M1: DeepSeek-R1	0.0	8.0	56.0	64.3
M2: DeepSeek-R1+ Prompt	6.5	29.5	64.4	71.2
M3: Qwen-Instruct+ Prompt	86.5	97.5	62.1	68.4
M4: DeepSeek-R1+ Fine-tune	100.0	100.0	81.5	92.0

Table 5: Main results on usability and accuracy.

Panel of Figure 4 — Figure 4: Semantic similarity distributions measured by BERT cosine similarity.

Appendix figures & tables8 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Description
$n$	Input token length
$\bm{x}=(x_{1},x_{2},\ldots,x_{n})$	Input token sequence (after tokenization)
$T$	Output token length
$\bm{y}=(y_{1},y_{2},\ldots,y_{T})$	Output token sequence
$\bm{y}_{<t}=(y_{1},\ldots,y_{t-1})$	Previously generated output tokens until $t$
$\hat{\bm{y}}_{t}$	Predicted token distribution at step $t$

Appendix

Table 10

Model	Count	Mean	Std.	Median	Max	$p(\mathrm{Sim}_{\mathrm{BERT}}\geq\kappa)$
DeepSeek-R1 + Fine-tune	1500	0.869	0.159	0.929	1.000	0.791
Gemini-2.5-Flash	1500	0.799	0.160	0.857	0.977	0.685
Claude-Haiku-4.5	1500	0.757	0.170	0.803	0.974	0.575
GPT-4o-mini	1500	0.787	0.163	0.844	0.974	0.656
GPT-4.1	1500	0.749	0.163	0.793	0.977	0.561
GPT-5.2	1500	0.719	0.167	0.762	1.000	0.472

Appendix

Table 7: Summary statistics of BERT cosine similarity of DeepSeek-R1 + Fine-tune and SoTA commercial LLM on the full evaluation set (

N=1500

Model	Count	Mean	Std.	Median	Max	$p(\mathrm{Sim}_{\mathrm{BERT}}\geq\kappa)$
DeepSeek-R1 + Fine-tune	1334	0.899	0.132	0.946	1.000	0.863
Gemini-2.5-Flash	1334	0.817	0.146	0.867	0.977	0.733
Claude-Haiku-4.5	1334	0.776	0.159	0.819	0.970	0.624
GPT-4o-mini	1334	0.806	0.149	0.856	0.974	0.705
GPT-4.1	1334	0.767	0.152	0.809	0.977	0.605
GPT-5.2	1334	0.736	0.155	0.773	1.000	0.507

Appendix

Table 8: Summary statistics of BERT cosine similarity DeepSeek-R1 + Fine-tune and SoTA commercial LLM on the HQ subset (

N=1334

Figure 5: BERT cosine similarity: DeepSeek-R1 + Fine-tune versus Gemini-2.5-Flash on the full evaluation set and HQ subset.

Figure 6: BERT cosine similarity: DeepSeek-R1 + Fine-tune versus Claude Haiku 4.5 on the full evaluation set and HQ subset.

Figure 7: BERT cosine similarity: DeepSeek-R1 + Fine-tune model versus GPT-4.1 on the full evaluation set and HQ subset.

Figure 8: BERT cosine similarity: DeepSeek-R1 + Fine-tune model versus GPT-4o-mini on the full evaluation set and HQ subset.

Figure 9: BERT cosine similarity: DeepSeek-R1 + Fine-tune versus GPT-5.2 on the full valid set and HQ subset.

LLM Fine-Tuning

LLM fine-tuning adapts pretrained large language models to tasks, domains, languages, and behaviors through additional training.

See all 754 papers

Oct 7, 2026

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain

Recent advances in reasoning-oriented large language models (LLMs) have motivated increasing evaluation of their ability to perform professional decision tasks. Insurance claim adjudication is one such task, requiring models to connect case evidence, insurance rules, intermediate judgments, and payout calculations across a structured decision process. We introduce InsClaimBench, an end-to-end benchmark for evaluating insurance claim adjudication across the decision chain. Grounded in real claim materials and structured insurance rules, InsClaimBench contains 3,780 cases in 375 case families across auto, property, and health insurance, comprising 86,656 atomic rule judgments. It evaluates each claim from atomic rules through adjudication modules to payout decisions and amounts, with controlled factual variants testing whether required changes are correctly propagated across levels. Evaluation of six LLMs reveals a progressive loss of reliability along the decision chain. Payout-decision accuracy ranges from 74.23--80.19%, while joint decision--amount accuracy drops to 47.54--73.15%. Strong local performance also fails to ensure case-level correctness: atomic-rule accuracy reaches 95.48%, whereas rule-vector exact match peaks at only 36.90%, and the most frequent module errors are not necessarily those most associated with final-decision failure. Under factual changes, these inconsistencies further become propagation failures: module updates are less reliable than rule updates, correct local judgments can still yield incorrect payouts, and correct payouts can conceal intermediate errors. These results show that reliable claim adjudication requires consistent composition and propagation across the decision chain.

Linqi Zhang, Chong Qi, Yan Cheng +4

School of Computer Science and Technology, Fudan University · School of Integrated Circuits, Nanjing University · School of Economics, Fudan University

AI-Assisted Decision Making LLM Evaluation LLM Reliability LLM Decision-Making

Ask PDF

Mar 15, 2026

An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs

Adapting Large Language Models (LLMs) to high-stakes vertical domains like insurance presents a significant challenge: scenarios demand strict adherence to complex regulations and business logic with zero tolerance for hallucinations. Existing approaches often suffer from a Competency Trade-off - sacrificing general intelligence for domain expertise - or rely heavily on RAG without intrinsic reasoning. To bridge this gap, we present INS-S1, an insurance-specific LLM family trained via a novel end-to-end alignment paradigm. Our approach features two methodological innovations: (1) A Verifiable Data Synthesis System that constructs hierarchical datasets for actuarial reasoning and compliance; and (2) A Progressive SFT-RL Curriculum Framework that integrates dynamic data annealing with a synergistic mix of Verified Reasoning (RLVR) and AI Feedback (RLAIF). By optimizing data ratios and reward signals, this framework enforces domain constraints while preventing catastrophic forgetting. Additionally, we release INSEva, the most comprehensive insurance benchmark to date (39k+ samples). Extensive experiments show that INS-S1 achieves SOTA performance on domain tasks, significantly outperforming DeepSeek-R1 and Gemini-2.5-Pro. Crucially, it maintains top-tier general capabilities and achieves a record-low 0.6% hallucination rate (HHEM). Our results demonstrate that rigorous domain specialization can be achieved without compromising general intelligence.

Qian Zhu, Xinnan Guo, Jingjing Huo +5

Ant Group

Domain Adaptation for LLMs LLM Alignment LLM Hallucination Mitigation RL for Language Model Reasoning LLM Evaluation

Ask PDF

Mar 8, 2026

Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation

The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.

David Beauchemin, Richard Khoury

Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Quebec City, Quebec, Canada

LLM Evaluation LLM Reliability Question Answering Retrieval-Augmented Generation CoT Reasoning

Ask PDF

Abstract

LLM Fine-Tuning

Explore similar work

InsClaimBench: Benchmarking Insurance Claim Adjudication Across the Decision Chain

An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs

Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation