The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.
Figures & tables
Figure 1: Example of a translated APEC-QA question, choices and response along with the prompt used for the evaluation. Blue box contains the task instructions. Yellow box contains the prefix for the model to continue. Texts in “ ≪≫ ” are role-tags to be fed to the model.
Metric
Value
Total Questions
807
Total Tokens (Questions)
96,080
Total Tokens (Justifications)
75,147
Average Sentence Length (Questions)
18.95 words
Average Sentence Length (Justifications)
17.47 words
Vocabulary Size (Questions)
6,753
Table 1: Descriptive Statistics of the AEPC-QA Dataset. Token counts and sentence lengths are reported as averages or totals across the corpus.
Figure 2: Accuracy scores with (y-axis) and without (x-axis) RAG system. Black dashed lines are our Random baseline scores. The dotted lines represent the statistical significance boundaries ( Z=±3.29 , α=0.001 ). Points falling within the central cone indicate no statistically significant difference between closed-book and RAG performance. Points above the upper dotted line show significant improvement with RAG, and points below mean the opposite. Red dots are models that performed poorer than the baseline on one of the corpora, green dots are models that performed better than 60% on both corpora, while blue dots are those that do not fit in the two other performance classes.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
LLM
Source
Size
LLM
Source
Size
DeepSeek-R1-distill-Qwen-14b ( Γ )
DeepSeek-AI (2025)
14.8B
Phi-4
Abdin et al. (2024)
14.7B
DeepSeek-R1-distill-Qwen-32b ( Γ )
DeepSeek-AI (2025)
32.8B
Qwen2.5-14b
Hui et al. (2024)
14.7B
Deepthink-reasoning-14b ( Γ )
Sakthi (2025a)
14.8B
Qwen2.5-14b-it
Hui et al. (2024)
14.7B
Deepthink-reasoning-7b ( Γ )
Sakthi (2025b)
7.62B
Qwen2.5-7b-it
Hui et al. (2024)
7.6B
French-Alpaca-Llama3-8b-it ( Υ , Γ )
Pacifico (2024)
8.03B
Qwen3-30b-a3b
Qwen Team (2025)
30B
Granite-3.2-8b-it
Granite Team (2024)
8.17B
QwQ-32b ( Γ )
Qwen Team (2025)
32.8B
Appendix
Table 3: The selected open-source LLMs used in our work (16 models), along with their source and size. “ Υ ” are models that have a specialization in French, while “ Γ ” are models marketed as reasoning LLMs.
LLM
Source
LLM
Source
Claude-3.5-Haiku-20241022
Anthropic
GPT-4o-mini-2024-07-18
OpenAI
Claude-3.7-Sonnet-20250219 ( Γ )
Anthropic
Grok-2-latest
xAI
Claude-Opus-4-20250514 ( Γ )
Anthropic
Grok-3-fast-latest ( Γ )
xAI
Claude-Sonnet-4-20250514 ( Γ )
Anthropic
Grok-3-latest ( Γ )
xAI
DeepSeek-chat
DeepSeek
Grok-3-mini-fast-latest ( Γ )
xAI
DeepSeek-reasoner ( Γ )
DeepSeek
Grok-3-mini-latest ( Γ )
xAI
Appendix
Table 4: The selected private LLMs used in our work, along with their source. “ Γ ” indicates models marketed as reasoning LLMs.
Large Language Models (LLMs) have shown strong potential in financial reasoning, but existing benchmarks often evaluate domain knowledge, numerical reasoning, long-context understanding, and tool use in separate settings. This limits their ability to assess realistic professional workflows that require auditable, context-grounded, and tool-executable decisions. We introduce INS-ActBench, a comprehensive benchmark for evaluating professional actuarial capability in LLMs. INS-ActBench contains 12,050 Q&A pairs from public exams and sample questions released by 16 actuarial associations. It covers three subsets: INS-Act-Know for standardized actuarial knowledge, INS-Act-Case for long-context insurance case reasoning, and INS-Act-Practice for spreadsheet and R-code tasks with verifiable numerical outputs. Experiments on nine representative LLMs and human actuarial experts reveal a clear capability boundary: frontier LLMs perform strongly on standardized knowledge, but remain much weaker in case reasoning, tool-based workflows, and jurisdiction-sensitive practice. INS-ActBench provides a reproducible foundation for developing actuarial LLMs toward reliable professional assistance. The code is available at https://github.com/FDU-INS/INS-ActBench.
While Large Language Models (LLMs) have achieved strong performance on general-purpose language tasks, their deployment in regulated and data-sensitive domains, including insurance, remains limited. Leveraging millions of historical warranty claims, we propose a locally deployed governance-aware language modeling component that generates structured corrective-action recommendations from unstructured claim narratives. We fine-tune pretrained LLMs using Low-Rank Adaptation (LoRA), scoping the model to an initial decision module within the claim processing pipeline to speed up claim adjusters' decisions. We assess this module using a multi-dimensional evaluation framework that combines automated semantic similarity metrics with human evaluation, enabling a rigorous examination of both practical utility and predictive accuracy. Our results show that domain-specific fine-tuning substantially outperforms commercial general-purpose and prompt-based LLMs, with approximately 80% of the evaluated cases achieving near-identical matches to ground-truth corrective actions. Overall, this study provides both theoretical and empirical evidence to prove that domain-adaptive fine-tuning can align model output distributions more closely with real-world operational data, demonstrating its promise as a reliable and governable building block for insurance applications.
Zhengda Mo, Zhiyu Quan, Eli O'Donohue +1
Actuarial and Risk Management Sciences, University of Illinois at Urbana-Champaign · PCMI Corporation
Adapting Large Language Models (LLMs) to high-stakes vertical domains like insurance presents a significant challenge: scenarios demand strict adherence to complex regulations and business logic with zero tolerance for hallucinations. Existing approaches often suffer from a Competency Trade-off - sacrificing general intelligence for domain expertise - or rely heavily on RAG without intrinsic reasoning. To bridge this gap, we present INS-S1, an insurance-specific LLM family trained via a novel end-to-end alignment paradigm. Our approach features two methodological innovations: (1) A Verifiable Data Synthesis System that constructs hierarchical datasets for actuarial reasoning and compliance; and (2) A Progressive SFT-RL Curriculum Framework that integrates dynamic data annealing with a synergistic mix of Verified Reasoning (RLVR) and AI Feedback (RLAIF). By optimizing data ratios and reward signals, this framework enforces domain constraints while preventing catastrophic forgetting. Additionally, we release INSEva, the most comprehensive insurance benchmark to date (39k+ samples). Extensive experiments show that INS-S1 achieves SOTA performance on domain tasks, significantly outperforming DeepSeek-R1 and Gemini-2.5-Pro. Crucially, it maintains top-tier general capabilities and achieves a record-low 0.6% hallucination rate (HHEM). Our results demonstrate that rigorous domain specialization can be achieved without compromising general intelligence.