LiCQA : A Lightweight Complex Question Answering System

Sourav Saha; Dwaipayan Roy; Mandar Mitra

cs.CLFeb 25, 2026

LiCQA : A Lightweight Complex Question Answering System

Authors: Sourav Saha, Dwaipayan Roy, Mandar Mitra

Organizations: Indian Statistical Institute, Kolkata, India · Indian Institute of Science Education and Research, Kolkata, India

Abstract

Over the last twenty years, significant progress has been made in designing and implementing Question Answering (QA) systems. However, addressing complex questions, the answers to which are spread across multiple documents, remains a challenging problem. Recent QA systems that are designed to handle complex questions work either on the basis of knowledge graphs, or utilise contem- porary neural models that are expensive to train, in terms of both computational resources and the volume of training data required. In this paper, we present LiCQA, an unsupervised question answer- ing model that works primarily on the basis of corpus evidence. We empirically compare the effectiveness and efficiency of LiCQA with two recently presented QA systems, which are based on different underlying principles. The results of our experiments show that LiCQA significantly outperforms these two state-of-the-art systems on benchmark data with noteworthy reduction in latency.

Figures & tables

Figure 1. Architecture of the $\mathsf{LiCQA}$ system.

Answer Type in ( Li and Roth, 2002 )	OntoNotes 5 Entity Type
HUMAN	PERSON
LOCATION	GPE, LOC, ORG
ENTITY	NORP, FAC, PRODUCT, EVENT
	LANGUAGE, LAW, WORK_OF_ART
NUMERIC	DATE, TIME, PERCENT, MONEY
	QUANTITY, ORDINAL, CARDINAL

Table 1. Mapping question / answer types to OntoNotes tags. Answer types in uppercase correspond to top-level categories, while lowercases refer to second-level sub-categories.

Collection	$x_{1}$	$x_{2}$	$x_{3}$
Top10	100	0	0
Strata-1	60	30	10
Strata-2	50	40	10
Strata-3	50	30	20
Strata-4	40	40	20
Strata-5	40	30	30

Table 2. Values of

x_{1}

x_{2}

and

x_{3}

for the document collections used in this article.

	Stages of pipeline			Evaluation metrics
Type classification	Sentence embedding	Scoring	Ranking	tMRR	tP@1	tHit@5
Traditional (SVM)			$\mathit{comb}\mbox{-}\mathit{score}_{*}$	0.412	0.282	0.633
		max-score	$\mathit{comb}\mbox{-}\mathit{score}_{+}$	0.406	0.275	0.626
			$\mathit{comb}\mbox{-}\mathit{score}_{*}$	0.123	0.063	0.242
		avg-maxscore	$\mathit{comb}\mbox{-}\mathit{score}_{+}$	0.124	0.063	0.242
			$\mathit{comb}\mbox{-}\mathit{score}_{*}$	0.387	0.253	0.620

Table 3. Performance of different stages of pipeline for CQ-W questions in terms of tie aware version of MRR, P@1 and Hit@5.

System	Metric	Top10	Strata-1	Strata-2	Strata-3	Strata-4	Strata-5
ShortestPaths		0.240	0.261	0.249	0.237	0.259	0.270
BFS		0.249	0.256	0.266	0.212	0.219	0.254
DrQA	$\mathsf{MRR}$	0.226	0.237	0.257	0.256	0.215	0.248
QUEST		0.355	0.380	0.340	0.302	0.356	0.318
$\mathsf{LiCQA}$		0.432 ∗ § †‡	0.424 ∗ § †‡	0.431 ∗ § †‡	0.420 ∗ § †‡	0.426 ∗ § †‡	0.416 ∗ § †‡
% improvement		21.6%	11.5%	26.7%	39%	19.6%	30.8%

Table 4. Performance comparison of baseline methods with

\mathsf{LiCQA}

for CQ-W questions in terms of MRR, P@1 and Hit@5. A superscript ∗ , § ,

\dagger

and

\ddagger

respectively indicate that

\mathsf{LiCQA}

is statistically significant (t-test with

p<0.05

) with the ShortestPaths , BFS , DrQA and QUEST. The percentage improvement of

\mathsf{LiCQA}

with the best performing baseline (QUEST) is presented in the last row of each group.

System	Metric	Top10	Strata-1	Strata-2	Strata-3	Strata-4	Strata-5
ShortestPaths		0.266	0.224	0.248	0.219	0.232	0.222
BFS		0.287	0.256	0.265	0.259	0.219	0.201
DrQA	$\mathsf{MRR}$	0.355	0.330	0.356	0.369	0.365	0.380
QUEST		0.467	0.436	0.426	0.460	0.409	0.384
$\mathsf{LiCQA}$		0.443 ∗ §	0.407 ∗ §	0.472 ∗ § †‡	0.476 ∗ § †‡	0.422 ∗ §	0.468 ∗ § †‡
% improvement		-5.1%	-6.6%	10.8%	3.4%	3.1%	21.8%

Table 5. Performance comparison of baseline methods with

\mathsf{LiCQA}

for CQ-T questions in terms of MRR, P@1 and Hit@5. A superscript ∗ , § ,

\dagger

and

\ddagger

respectively indicate that

\mathsf{LiCQA}

is statistically significant (t-test with

p<0.05

) with the ShortestPaths , BFS , DrQA and QUEST. The percentage improvement of

\mathsf{LiCQA}

with the best performing baseline (QUEST) is presented in the last row of each group.

System	Metric	Top10	Strata-1	Strata-2	Strata-3	Strata-4	Strata-5	Top10	Strata-1	Strata-2	Strata-3	Strata-4	Strata-5
				CQ-W						CQ-T
DrQA		0.226	0.237	0.257	0.256	0.215	0.248	0.355	0.330	0.356	0.369	0.365	0.380
QUEST	$\mathsf{tMRR}$	0.067	0.116	0.066	0.067	0.077	0.055	0.065	0.075	0.066	0.076	0.069	0.057
$\mathsf{LiCQA}$		0.412 †‡	0.411 †‡	0.408 †‡	0.398 †‡	0.418 †‡	0.399 †‡	0.433 ‡	0.388 †‡	0.443 †‡	0.455 †‡	0.399 †‡	0.434 †‡
% improvement		82.3%	73.4%	58.7%	55.4%	94.4%	60.8%	21.9%	17.5%	24.4%	23.3%	9.31%	14.2%
DrQA		0.184	0.199	0.221	0.215	0.172	0.200	0.286	0.267	0.287	0.300	0.287	0.320

Table 7. Performance comparison of baseline methods with

\mathsf{LiCQA}

for CQ-W and CQ-T questions in terms of

\mathsf{tMRR}

\mathsf{tP@1}

and

\mathsf{tHit@5}

. A

\dagger

and

\ddagger

indicate that

\mathsf{LiCQA}

is statistically significant (t-test with

p<0.05

) than DrQA and QUEST respectively. Percentage improvement of

\mathsf{LiCQA}

with best performing baseline (DrQA) is presented in last row of each group.

Question Set	Question	Answer	$\mathsf{LiCQA}$ rank	QUEST rank	DrQA rank
CQ-W	What movie starred Bruce Willis and Haley Joel Osment?	The Sixth Sense	1	$>$ 5	3
CQ-W	Who founded Apple and Pixar?	Steve Jobs	1	2	5
CQ-T	Which 2018 studio album is performed by Playboi Carti and features Nicki Minaj?	Die Lit	1	$>$ 5	2
	Which American professional golfer is married to Taylor Dowd Simpson and played for Wake Forest University?	Simpson	1	$>$ 5	$>$ 5

Table 8. Some example questions for which

\mathsf{LiCQA}

has been able to retrieve the correct answer at rank 1 where as QUEST and DrQA have failed to retrieve the answers in the top rank.

Panel of Figure 2 — Figure 2. Performance analysis of $\mathsf{LiCQA}$ , DrQA and QUEST on CQ-W and CQ-T question set on the Top10 dataset. First and second barplots (shown in ‘khaki’ and ‘olive’ color) denote QUEST result with normal and tie-aware metrics respectively. Similarly fourth and fifth barplots (shown in ‘skyblue’ and ‘steelblue’ color) denote $\mathsf{LiCQA}$ result with normal and tie-aware metrics respectively. Middle bar with teal color presents result of DrQA (as it retrieves a single entity per rank position).

Panel of Figure 3 — Figure 3. A per-query performance analysis between $\mathsf{LiCQA}$ and QUEST on CQ-W question set for Top10 dataset. Bars above x-axis correspond to questions for which $\mathsf{LiCQA}$ has achieved better performance than QUEST.

Panel of Figure 4 — Figure 4. A per-query performance analysis between $\mathsf{LiCQA}$ and QUEST on CQ-T question set for Top10 dataset. Bars above x-axis correspond to questions for which $\mathsf{LiCQA}$ has achieved better performance than QUEST.

System	Query set	Top10	Strata-1	Strata-2	Strata-3	Strata-4	Strata-5
$\mathsf{LiCQA}$	CQ-W	13	15	17	14	16.5	14.3
QUEST		144	148	151	156	149	162
DrQA		120	131	119	124.6	128	129
speedup		9.23 $\times$	8.73 $\times$	7 $\times$	8.9 $\times$	7.75 $\times$	9.02 $\times$
$\mathsf{LiCQA}$	CQ-T	14.5	16	18	15.8	18	17.6
QUEST		155	159	162	178	189.5	195.4

Table 9. Per query latency (in second) analysis of different QA models. Least latency highlighted in bold. Speedup row indicates the reduction in execution time than the baseline with the minimum latency (DrQA).

Question Answering

Question answering studies how AI systems interpret questions and produce accurate, evidence-grounded answers across text, dialogue, and multimodal data.

See all 260 papers

Gradually Excavating External Knowledge for Implicit Complex Question Answering
Mar 9, 2026Chang Liu, Xiaoguang Li, Lifeng Shang +4Iterative RAG Question Answering
The University of Hong Kong · Huawei Noah’s Ark Lab
LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
Jun 9, 2026Haonan Wang, Jiaxiang Liu, Yurong Liu +11Multi-Hop QA AI Agent Benchmarks
Columbia University · New York University · Barnard College
LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes
Jul 14, 2026Michael Solodko, Steven Gong, Guangwei Yu +3Multimodal QA Agentic Retrieval
University of Waterloo · Layer 6 AI

Mar 9, 2026J/K move · Enter open · S save

Gradually Excavating External Knowledge for Implicit Complex Question Answering

Chang Liu, Xiaoguang Li, Lifeng Shang, Xin Jiang+3

The University of Hong Kong · Huawei Noah’s Ark Lab

Iterative RAG Question Answering Knowledge Augmentation for Language Models Multi-Hop QA Retrieval-Augmented Generation

Open paper AskPDF arXiv

Recently, large language models (LLMs) have gained much attention for the emergence of human-comparable capabilities and huge potential. However, for open-domain implicit question-answering problems, LLMs may not be the ultimate solution due to the reasons of: 1) uncovered or out-of-date domain knowledge, 2) one-shot generation and hence restricted comprehensiveness. To this end, this work proposes a gradual knowledge excavation framework for open-domain complex question answering, where LLMs iteratively and actively acquire external information, and then reason based on acquired historical knowledge. Specifically, during each step of the solving process, the model selects an action to execute, such as querying external knowledge or performing a single logical reasoning step, to gradually progress toward a final answer. Our method can effectively leverage plug-and-play external knowledge and dynamically adjust the strategy for solving complex questions. Evaluated on the StrategyQA dataset, our method achieves 78.17% accuracy with less than 6% parameters of its competitors, setting new SOTA for ~10B-scale LLMs.

Abstract

Question Answering

Explore similar work

Gradually Excavating External Knowledge for Implicit Complex Question Answering

LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake

LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes