Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation

David Beauchemin; Richard Khoury

cs.CLMar 8, 2026

Benchmarking Large Language Models for Quebec Insurance: From Closed-Book to Retrieval-Augmented Generation

Authors: David Beauchemin, Richard Khoury

Organizations: Group for Research in Artificial Intelligence of Laval University (GRAIL), Université Laval, Quebec City, Quebec, Canada

Abstract

The digitization of insurance distribution in the Canadian province of Quebec, accelerated by legislative changes such as Bill 141, has created a significant "advice gap", leaving consumers to interpret complex financial contracts without professional guidance. While Large Language Models (LLMs) offer a scalable solution for automated advisory services, their deployment in high-stakes domains hinges on strict legal accuracy and trustworthiness. In this paper, we address this challenge by introducing AEPC-QA, a private gold-standard benchmark of 807 multiple-choice questions derived from official regulatory certification (paper) handbooks. We conduct a comprehensive evaluation of 51 LLMs across two paradigms: closed-book generation and retrieval-augmented generation (RAG) using a specialized corpus of Quebec insurance documents. Our results reveal three critical insights: 1) the supremacy of inference-time reasoning, where models leveraging chain-of-thought processing (e.g. o3-2025-04-16, o1-2024-12-17) significantly outperform standard instruction-tuned models; 2) RAG acts as a knowledge equalizer, boosting the accuracy of models with weak parametric knowledge by over 35 percentage points, yet paradoxically causing "context distraction" in others, leading to catastrophic performance regressions; and 3) a "specialization paradox", where massive generalist models consistently outperform smaller, domain-specific French fine-tuned ones. These findings suggest that while current architectures approach expert-level proficiency (~79%), the instability introduced by external context retrieval necessitates rigorous robustness calibration before autonomous deployment is viable.

Figures & tables

Panel of Figure 1 — Figure 1: Example of a translated APEC-QA question, choices and response along with the prompt used for the evaluation. Blue box contains the task instructions. Yellow box contains the prefix for the model to continue. Texts in “ $\ll\gg$ ” are role-tags to be fed to the model.

Metric	Value
Total Questions	807
Total Tokens (Questions)	96,080
Total Tokens (Justifications)	75,147
Average Sentence Length (Questions)	18.95 words
Average Sentence Length (Justifications)	17.47 words
Vocabulary Size (Questions)	6,753

Table 1: Descriptive Statistics of the AEPC-QA Dataset. Token counts and sentence lengths are reported as averages or totals across the corpus.

Panel of Figure 2 — Figure 2: Accuracy scores with (y-axis) and without (x-axis) RAG system. Black dashed lines are our Random baseline scores. The dotted lines represent the statistical significance boundaries ( $Z=\pm 3.29$ , $\alpha=0.001$ ). Points falling within the central cone indicate no statistically significant difference between closed-book and RAG performance. Points above the upper dotted line show significant improvement with RAG, and points below mean the opposite. Red dots are models that performed poorer than the baseline on one of the corpora, green dots are models that performed better than 60% on both corpora, while blue dots are those that do not fit in the two other performance classes.

Appendix figures & tables2 assets

Supplementary material from the paper’s appendix.

Appendix

LLM	Source	Size	LLM	Source	Size
DeepSeek-R1-distill-Qwen-14b ( $\Gamma$ )	DeepSeek-AI (2025)	14.8B	Phi-4	Abdin et al. (2024)	14.7B
DeepSeek-R1-distill-Qwen-32b ( $\Gamma$ )	DeepSeek-AI (2025)	32.8B	Qwen2.5-14b	Hui et al. (2024)	14.7B
Deepthink-reasoning-14b ( $\Gamma$ )	Sakthi (2025a)	14.8B	Qwen2.5-14b-it	Hui et al. (2024)	14.7B
Deepthink-reasoning-7b ( $\Gamma$ )	Sakthi (2025b)	7.62B	Qwen2.5-7b-it	Hui et al. (2024)	7.6B
French-Alpaca-Llama3-8b-it ( $\Upsilon$ , $\Gamma$ )	Pacifico (2024)	8.03B	Qwen3-30b-a3b	Qwen Team (2025)	30B
Granite-3.2-8b-it	Granite Team (2024)	8.17B	QwQ-32b ( $\Gamma$ )	Qwen Team (2025)	32.8B

Appendix

Table 3: The selected open-source LLMs used in our work (16 models), along with their source and size. “

\Upsilon

” are models that have a specialization in French, while “

\Gamma

” are models marketed as reasoning LLMs.

LLM	Source	LLM	Source
Claude-3.5-Haiku-20241022	Anthropic	GPT-4o-mini-2024-07-18	OpenAI
Claude-3.7-Sonnet-20250219 ( $\Gamma$ )	Anthropic	Grok-2-latest	xAI
Claude-Opus-4-20250514 ( $\Gamma$ )	Anthropic	Grok-3-fast-latest ( $\Gamma$ )	xAI
Claude-Sonnet-4-20250514 ( $\Gamma$ )	Anthropic	Grok-3-latest ( $\Gamma$ )	xAI
DeepSeek-chat	DeepSeek	Grok-3-mini-fast-latest ( $\Gamma$ )	xAI
DeepSeek-reasoner ( $\Gamma$ )	DeepSeek	Grok-3-mini-latest ( $\Gamma$ )	xAI

Appendix

Table 4: The selected private LLMs used in our work, along with their source. “

\Gamma

” indicates models marketed as reasoning LLMs.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

INS-ActBench: A Comprehensive Benchmark for Assessing Professional Actuarial Capability of Large Language Models
Jul 27, 2026Changyu Chen, Chenwei Lin, Xian XuNumerical Reasoning in Language Models Long-Context Reasoning in Language Models
Fudan University
Claim Automation using Large Language Model
Feb 18, 2026Zhengda Mo, Zhiyu Quan, Eli O'Donohue +1LLM Fine-Tuning Business Process Automation
Actuarial and Risk Management Sciences, University of Illinois at Urbana-Champaign · PCMI Corporation
An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs
Mar 15, 2026Qian Zhu, Xinnan Guo, Jingjing Huo +5Domain Adaptation for LLMs LLM Alignment
Ant Group

Jul 27, 2026J/K move · Enter open · S save

INS-ActBench: A Comprehensive Benchmark for Assessing Professional Actuarial Capability of Large Language Models

Changyu Chen, Chenwei Lin, Xian Xu

Fudan University

Numerical Reasoning in Language Models Long-Context Reasoning in Language Models LLM Evaluation LLM Tool Use Finance

Open paper AskPDF arXiv

Large Language Models (LLMs) have shown strong potential in financial reasoning, but existing benchmarks often evaluate domain knowledge, numerical reasoning, long-context understanding, and tool use in separate settings. This limits their ability to assess realistic professional workflows that require auditable, context-grounded, and tool-executable decisions. We introduce INS-ActBench, a comprehensive benchmark for evaluating professional actuarial capability in LLMs. INS-ActBench contains 12,050 Q&A pairs from public exams and sample questions released by 16 actuarial associations. It covers three subsets: INS-Act-Know for standardized actuarial knowledge, INS-Act-Case for long-context insurance case reasoning, and INS-Act-Practice for spreadsheet and R-code tasks with verifiable numerical outputs. Experiments on nine representative LLMs and human actuarial experts reveal a clear capability boundary: frontier LLMs perform strongly on standardized knowledge, but remain much weaker in case reasoning, tool-based workflows, and jurisdiction-sensitive practice. INS-ActBench provides a reproducible foundation for developing actuarial LLMs toward reliable professional assistance. The code is available at https://github.com/FDU-INS/INS-ActBench.

Abstract

LLM Evaluation

Explore similar work

INS-ActBench: A Comprehensive Benchmark for Assessing Professional Actuarial Capability of Large Language Models

Claim Automation using Large Language Model

An Industrial-Scale Insurance LLM Achieving Verifiable Domain Mastery and Hallucination Control without Competence Trade-offs