Toward Agentic Retrieval-Augmented Generation for Enterprise Retail Workflows over Heterogeneous Data Sources

Jeonghwan Choi; Jibin Hwang; Hayoon Park; Hyeonjae Cheon; Hwanjun Song

cs.AIOct 18, 2026

Toward Agentic Retrieval-Augmented Generation for Enterprise Retail Workflows over Heterogeneous Data Sources

Authors: Jeonghwan Choi, Jibin Hwang, Hayoon Park, Hyeonjae Cheon, Hwanjun Song

Organizations: Korea Advanced Institute of Science and Technology (KAIST) · GS Retail · Cluvion

Abstract

Enterprise retail workflows require agents to chain evidence across heterogeneous sources, from structured product and order records to unstructured inquiries and reviews, yet existing agentic RAG benchmarks rarely capture this challenge. To enable controlled evaluation of such workflows, we introduce RetailBridge, a compositional retail benchmark built from persona-driven simulations. Its questions are constructed by linking entities and intermediate answers across heterogeneous sources, requiring agents to identify the relevant sources, carry evidence between them, and produce a grounded answer. To systematically evaluate LLMs on these retail workflows, we develop RetailRAG, a retail-aware framework equipped with auto-schema briefing, source-routed plan chaining, and diagnostic plan revision. RetailRAG consistently outperforms existing agentic RAG baselines across all evaluation axes, serving as a strong reference system for assessing LLM agents on retail workflows. On RetailBridge, evaluating ten LLM backbones with RetailRAG reveals multi-source chaining as the dominant bottleneck across providers and model scales, suggesting that reliable retail agents require explicit cross-source orchestration rather than scaling alone. Code and data are publicly available at https://github.com/DISL-Lab/RetailBridge.

Figures & tables

Figure 1: Overview of RetailBridge . Starting from persona-driven retail simulations, RetailBridge constructs a heterogeneous retail database, then (Step 1) generates source-grounded atomic QAs and (Step 2) recursively composes them via entity-linked and answer-linked operators into multi-source, multi-hop queries stratified along three evaluation axes.

Operator	Query	Example
$O_{\mathrm{ent}}$	$q_{1}$ (Level-1)	Which seller broadcast the Libby’s Mixed Vegetables (8.5 oz, Pack of 12)? A: Robert A.
	$q_{\ell}$ (Level- $\ell$ )	What is the average product review rating for Robert A. ? A: 3.41
	$q_{\ell+1}$ (Composite)	What is the average product review rating for the seller who broadcast the Libby’s Mixed Vegetables ? A: 3.41
	Effect	Single-entity bridge ( Script $\rightarrow$ Review ); a singular $q^{\text{ref}}$ hides one entity
$O_{\mathrm{ans}}$	$q_{\ell}$ (Level- $\ell$ )	Which of Mark A.’s products received the highest rating in product reviews? A: { Stretchable Candy Necklace, … }
	$\tilde{q}_{\ell}$ (Follow-up)	What is the total revenue from orders of these products ? A: $20,712.52

Table 1: Examples of the two composition operators in RetailBridge . Colors mark each operator’s bridged value (

O_{\mathrm{ent}}

O_{\mathrm{ans}}

) and the referring expression

q^{\text{ref}}

hiding it in

q_{\ell+1}

Figure 2: Overview of RetailRAG , a retail-aware agentic RAG built on (i) auto-schema briefing, (ii) source-routed plan chaining, and (iii) diagnostic plan revision.

Figure 3: LLM-judge accuracy on RetailBridge per evidence modality (SQL / Text / Composite) across reasoning levels ( Level 1 , Level 2 , Level 3 ): (a) RetailRAG vs. six baselines; and (b) Ten LLM backbones in RetailRAG , split into open-source and proprietary. The inner axis ( ) indicates the overall performance.

Category	Method	Level-1		Level-2				Level-3				Avg.		API Call
		Level-1		Homo.		Hetero.		Homo.		Hetero.		Avg.
		Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM
Naive	Text-to-SQL	0.421	0.421	0.225	0.229	0.121	0.129	0.129	0.175	0.062	0.094	0.163	0.182	2.00
Naive	Dense RAG	0.050	0.200	0.004	0.108	0.010	0.121	0.017	0.067	0.010	0.096	0.016	0.115	2.00
Agentic	ReAct	0.196	0.258	0.058	0.096	0.033	0.079	0.008	0.021	0.006	0.029	0.049	0.085	4.20
	MAC-SQL	0.404	0.463	0.175	0.204	0.150	0.204	0.092	0.125	0.075	0.102	0.160	0.201	3.08

Table 2: Performance on RetailBridge . Homo. / Hetero. : evidence from one/multiple sources. Acc : accuracy. LLM : LLM-as-judge accuracy. API Call : API calls per query. See Appendix E.2 for the stability of RetailRAG and Appendix E.1 for cross-dataset results.

Figure 4: Ablation of RetailRAG ’s three retail-specific designs on RetailBridge (LLM-judge). The - - - line marks the best baseline (HetaRAG).

Category	Method	Default		w/ Briefing
Category	Method	LLM	API Call	LLM	API Call
Naive	Text-to-SQL	0.182	2.00	0.296	2.00
Agentic	ReAct	0.085	4.20	0.102	4.29
	MAC-SQL	0.201	3.08	0.205	3.13
	TableRAG	0.185	6.55	0.229	6.23
	HetaRAG	0.293	25.21	0.336	23.37

Table 3: Effect of RetailRAG ’s auto-schema briefing on baselines. w/ Briefing : the briefing is injected into each baseline. LLM : LLM-as-judge accuracy. API Call : API calls per query.

Category	Model	Level-1		Level-2				Level-3				Avg.		API Call
		Level-1		Homo.		Hetero.		Homo.		Hetero.		Avg.
		Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM
Open-source	Qwen3.6-35B-A3B	0.562	0.608	0.154	0.242	0.135	0.217	0.092	0.121	0.052	0.125	0.169	0.236	12.76
	Kimi-K2.5	0.621	0.658	0.179	0.254	0.129	0.204	0.113	0.158	0.090	0.154	0.193	0.255	11.81
	Gemma-4-31B-IT	0.692	0.704	0.346	0.492	0.317	0.529	0.242	0.354	0.208	0.440	0.333	0.498	10.50
	GPT-oss-120B	0.688	0.725	0.342	0.463	0.279	0.479	0.179	0.250	0.158	0.302	0.298	0.429	7.45

Table 4: LLM benchmark on RetailBridge by reasoning level and source diversity. Homo. / Hetero. : evidence from one or multiple sources. Acc : accuracy. LLM : LLM-as-judge accuracy. API Call : mean LLM calls per query.

Model	Level-1	Level-2		Level-3		Avg.
Model	Level-1	Homo.	Hetero.	Homo.	Hetero.	Avg.
Qwen3.6-35B-A3B	0.606	0.359	0.326	0.241	0.273	0.343
Kimi-K2.5	0.616	0.352	0.273	0.250	0.299	0.337
Gemma-4-31B-IT	0.684	0.606	0.590	0.545	0.607	0.604
GPT-oss-120B	0.694	0.588	0.533	0.474	0.484	0.541
DeepSeek-V4-Flash	0.703	0.563	0.551	0.509	0.529	0.562

Table 5: Source routing F1 on RetailBridge (1,680 queries) by reasoning level and source diversity. Avg. is weighted by the number of queries per cell (L1/L2-Homo/L3-Homo: 240 each; L2-Hetero/L3-Hetero: 480 each).

Figure 5: Efficiency-accuracy trade-off of LLM backbones on RetailBridge . Arrow indicates the ideal.

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Schema of the heterogeneous retail database $\mathcal{D}$ used in RetailBridge .

Operator	Query	Example
$O_{\mathrm{ans}}$ (Entity Set)	$q_{\ell}$ (Level- $\ell$ )	Which products sold by Leslie B. have the highest refund rate? A: { Dell 2019 New …, Herstyle Certain… } (2 products)
	$\tilde{q}_{\ell}$ (Follow-up)	What is the average product review rating for these products ? A: 3.37
	$q_{\ell+1}$ (Composite)	What is the average product review rating for the products sold by Leslie B. that have the highest refund rate ? A: 3.37
	Effect	Set bridge ( Order $\rightarrow$ Review ); a plural $q^{\text{ref}}$ hides the whole answer set
$O_{\mathrm{ans}}$ (Scalar)	$q_{\ell}$ (Level- $\ell$ )	What is the total amount spent by Lisa J. across all her orders? A: $3,161.72
	$\tilde{q}_{\ell}$ (Follow-up)	Which buyers have a total order value greater than $3,161.72 ? A: {Adam N., Brent B., …} (18 buyers)

Appendix

Table 6: Examples of answer-linked composition with

O_{\mathrm{ans}}

in RetailBridge , covering entity-set and scalar answers. Colors mark the bridged value (

O_{\mathrm{ans}}

) and the referring expression

q^{\text{ref}}

hiding it in

q_{\ell+1}

Statistic		Number
Sources		5 tabular / 3 textual
Total rows: 49,718	Total docs: 43,140
Tabular (rows / cols)
Product: 200 / 6	Order: 6,378 / 14	Script: 287 / 11
Review: 24,104 / 11	Inquiry: 18,749 / 9
Textual (docs / avg. chars)

Appendix

Table 7: Statistics of RetailBridge : source-level counts and QA distribution by reasoning capability, multi-source chaining, and evidence modality. Homo./Hetero. denote single- or multi-source evidence; Composite , SQL+Text questions.

Level	Source	Modality	Example question / answer
Level-1	Homo. (Order ∗ )	SQL	Q: What is the total revenue for Michael K.? A: $11,296.83
Level-1	Homo. (Script † )	Text	Q: What input connections did Mark A. mention for the INSIGNIA 2.1-Channel 80W Soundbar …? A: RCA, optical digital, mini-jack, and Bluetooth.
Level-2	Homo. (Order ∗ )	SQL	Q: Which sellers have a refund rate greater than Mark A.’s refund rate? A: Natalie H., Joshua H., Joann I., Robert A., Olivia F., Michael K., Diana G., Daniel X., Ashley X., Steven C., Leslie B., Allen G., Alexander G., Michelle W.
	Hetero. (Order ∗ , Product ∗ )	SQL	Q: What is the original price of the product sold by Robert A. that has the highest conversion rate? A: $21.31
	Homo. (Inquiry † )	Text	Q: What did Ashley X. tell Gabriel Z. about the return shipping costs for a change-of-mind return of the product that was the subject of a pre-purchase inquiry between Beth A. and Steven C.? A: The buyer must cover the return shipping cost; no prepaid label is provided for a change-of-mind return.
	Hetero. (Inquiry † , Script † )	Text	Q: What structural features did the seller highlight to assure buyers that the product the buyer asked about (regarding weight limits and dimensions) can support heavy winter coats? A: The rack features a solid wood core and sturdy metal anchors, providing a rock-solid structure designed to securely hold heavy winter coats.

Appendix

Table 8: Representative RetailBridge examples across reasoning capability, multi-source chaining, and evidence modality. Source gives the source diversity (Homo./Hetero.) and the underlying sources an agent must route to; SQL tables are marked with ∗ and text stores with † . Long product names in questions are abbreviated; answers are given in full.

Level	Shared query pattern	RetailBridge (ours)	Human-authored workload
L1	Ranking by frequency	Which buyer has sent the most pre-purchase inquiries to Ashley X.?	Identify the top-3 customers with the highest number of delivered orders. (Olist)
L1	Threshold filtering	How many broadcasts did Michael K. conduct with a discount rate greater than 20%?	Which liquor categories each contribute at least 1% of monthly sales volume? (Iowa Liquor)
L2	Best entity within a resolved group	Among the buyers who gave Diana G. the lowest product-quality rating, which buyer has the highest conversion rate?	Find the top-selling product among customers who bought “YouTube Men’s Vintage Henley”. (GA360)
L3	Chained lookups with aggregation	Most common delivery date for orders of the most expensive product Jacob Z. ordered from the seller who received the highest total spending from him.	Most-purchased other products by customers who bought the Google Red Speckled Tee, per month. (GA4)

Appendix

Table 9: Query-pattern correspondence between RetailBridge and human-authored queries in real retail workloads. Parentheses indicate the source database of each human-authored query in Spider 2.0-lite.

Level	1 Step	2 Steps	3+ Steps
Level-1	89.2%	8.8%	2.0%
Level-2	12.7%	64.4%	22.9%
Level-3	7.9%	8.3%	83.8%

Appendix

Table 10: Number of steps taken by correct runs at each reasoning level.

Table 17

Stage	Type	Model	Usage	Model Checkpoint	Source	Reference
Simu- lation	Open- source	DeepSeek-V3.2	Simulation	deepseek-ai/DeepSeek-V3.2	HuggingFace	Liu et al. (2025)
		GPT-oss-120B	Simulation	openai/gpt-oss-120b	HuggingFace	Agarwal et al. (2025)
		Qwen3-235B	Simulation	Qwen/Qwen3-235B-A22B-Instruct	HuggingFace	Yang et al. (2025a)
	Proprietary	Gemini-3-Flash	Simulation	gemini-3-flash-preview	Google API	Google DeepMind (2025)
Benchmark Construction	Open-source	Qwen3-Embedding-8B	QA Sampling	qwen/qwen3-embedding-8b	HuggingFace	Zhang et al. (2025)
	Proprietary	Qwen3.6-Max	QA Generation	qwen3.6-max-preview	Qwen API	Qwen Team (2026b)

Appendix

Table 13: LLM configurations across the three pipeline stages: simulation, benchmark construction (QA generation and verification), and RetailRAG evaluation. Models used in multiple stages are listed under each relevant stage.

Judge Model	Cohen’s $\kappa$	Precision	Recall	F1
Qwen3.6-Plus (Paper)	0.960	0.971	0.984	0.978
Muse-Spark-1.3	0.959	0.991	0.963	0.977
MiMo-V2.6-Flash	0.939	0.944	0.990	0.966

Appendix

Table 14: Agreement between LLM judges and human labels.

Reference Judge	Agreement (Cohen’s $\kappa$ )	Ranking (Kendall’s $\tau$ )	Bias (Qwen family)	Bias (Qwen3.6-Plus)
Muse-Spark-1.3	0.949	0.954	$-$ 0.1	+0.7
MiMo-V2.6-Flash	0.932	0.928	$-$ 0.8	$-$ 1.2

Appendix

Table 15: Multi-judge agreement and self-preference bias of the paper judge (Qwen3.6-Plus).

Answer Type	$n$	Rule-based	LLM Judge
Number	4,650	0.412	0.417
Temporal	840	0.647	0.647
Set	2,110	0.409	0.411
Entity	3,290	0.539	0.547
Total (excl. free text)	10,890	0.468	0.473

Appendix

Table 16: Rule-based vs. LLM-judge accuracy by answer type.

	Supported	Not Supported	Total
Correct	7,057 (42.0%)	148 (0.9%)	7,205 (42.9%)
Incorrect	4,145 (24.7%)	5,450 (32.4%)	9,595 (57.1%)
Total	11,202 (66.7%)	5,598 (33.3%)	16,800 (100%)

Appendix

Table 17: Correctness vs. support by retrieved evidence over all 16,800 answers.

Category	Method	F1	Cards	Hero	Football	Throm.	Code	Student	Tox.	Fin.	Debit	Schools	Avg.
Naive	Text-to-SQL	0.606	0.577	0.865	0.804	0.520	0.625	0.729	0.450	0.406	0.600	0.367	0.615
Naive	Dense RAG	0.136	0.096	0.019	0.098	0.060	0.082	0.062	0.150	0.000	0.100	0.000	0.078
Agentic	ReAct	0.455	0.365	0.615	0.549	0.340	0.469	0.562	0.325	0.344	0.233	0.300	0.432
	MAC-SQL	0.591	0.558	0.846	0.804	0.540	0.653	0.812	0.350	0.375	0.533	0.433	0.612
	TableRAG	0.121	0.115	0.096	0.157	0.020	0.061	0.042	0.100	0.062	0.033	0.000	0.080
	HetaRAG	0.515	0.577	0.865	0.784	0.480	0.653	0.812	0.425	0.562	0.533	0.533	0.622

Appendix

Table 18: Cross-dataset evaluation on the BIRD benchmark (

n{=}500

) across

11

domains. RetailRAG outperforms all baselines in overall accuracy, demonstrating that its retail-specific designs generalize to other domains.

Run	Level-1		Level-2		Level-3		Avg.
Run	Acc	LLM	Acc	LLM	Acc	LLM	Acc	LLM
Run 1	0.688	0.725	0.300	0.474	0.165	0.285	0.298	0.429
Run 2	0.662	0.717	0.287	0.449	0.174	0.296	0.292	0.421
Run 3	0.679	0.729	0.293	0.436	0.165	0.303	0.293	0.421
Run 4	0.683	0.725	0.293	0.450	0.165	0.290	0.294	0.421
Run 5	0.688	0.721	0.304	0.461	0.179	0.300	0.305	0.429

Appendix

Table 19: Stability of RetailRAG across five runs on RetailBridge (gpt-oss-120b). Bottom rows report the bootstrap mean and 95% percentile CI (

B{=}10{,}000

). Single-run results in Table 4 lie within all CIs.

Framework	LLM	Input Token	Output Token	API Call
ReAct	0.085 $\rightarrow$ 0.165	2.4k $\rightarrow$ 6.7k	1.3k $\rightarrow$ 2.1k	4.2 $\rightarrow$ 6.2
MAC-SQL	0.201 $\rightarrow$ 0.270	6.1k $\rightarrow$ 20.4k	2.4k $\rightarrow$ 4.3k	3.1 $\rightarrow$ 5.0
TableRAG	0.185 $\rightarrow$ 0.230	11.8k $\rightarrow$ 17.7k	2.9k $\rightarrow$ 3.2k	6.6 $\rightarrow$ 6.8
HetaRAG	0.293 $\rightarrow$ 0.368	19.3k $\rightarrow$ 90.6k	14.4k $\rightarrow$ 18.7k	25.2 $\rightarrow$ 27.6
RetailRAG	0.429	19.8k	5.4k	7.45

Appendix

Table 20: Baselines equipped with RetailRAG ’s schema briefing and retry (default

\rightarrow

briefing + retry). LLM : LLM-as-judge accuracy. API Call : API calls per query.

Condition	Level-1		Level-2			Level-3
Condition	SQL	Text	SQL	Text	Comp.	SQL	Text	Comp.
RetailRAG	0.857	0.625	0.561	0.400	0.464	0.381	0.221	0.255
Oracle-source	0.867	0.633	0.527	0.438	0.490	0.393	0.267	0.272
Oracle-plan	0.876	0.750	0.690	0.329	0.611	0.594	0.250	0.444
Oracle-intermediate	–	–	0.469	0.442	0.619	0.502	0.433	0.469
Oracle-evidence	0.971	0.990	0.975	0.992	0.992	0.983	0.971	0.987

Appendix

Table 21: LLM-judge accuracy of RetailRAG under each oracle condition.

Panel of Table 22 — Table 22: Prompt for Atomic QA Generation.

Panel of Table 23 — Table 23: Prompt for Entity-Answer Text Atomic Question Generation

Panel of Table 24 — Table 24: Prompt for SQL Query Regeneration after Execution Failure.

Panel of Table 25 — Table 25: Prompt for Text Atomic Answer Generation, which generates the grounded answer for a text atomic from its scoped evidence records.

Applied case	Hint
Level-1 atomic question entity exact-match block	# tier == "seller": [!] CRITICAL - use this EXACT string in SQL, character-for-character: WHERE seller_name = ’{anchor_id}’ Do NOT add, remove, or change any character - no trailing quotes, no possessive. # tier == "buyer": [!] CRITICAL - use this EXACT string in SQL, character-for-character: WHERE buyer_name = ’{anchor_id}’ Do NOT add, remove, or change any character - no trailing quotes, no possessive. # tier == "product": [!] CRITICAL - exact DB values (copy this FULL title literally into SQL): product_name: ’{ptitle}’ CORRECT form: WHERE product_name = ’{ptitle}’ DO NOT truncate the title, DO NOT add a trailing apostrophe.
Level-1 SQL atomic question	# answer-shape hint constants (ans_hint): _HINT_NAME = "The answer should be an entity NAME (buyer name, seller name, or product name)." _HINT_NUMERIC = "The answer should be a numeric value (count, average, total, ratio, etc.)." _HINT_DATE = "The answer should be a DATE or DATETIME (e.g. broadcast date, order date, delivery date)." # (sellers source) ans_hint: "The answer should be a PRODUCT NAME. Ask which product of this seller has the highest or lowest {col}." # _make_focus_hint(focus) -> (note leading space): " Focus the question on `{focus}`." # appended when src == "reviews": " Always filter by one specific review_category value: ’broadcast’, ’product’, ’pre-cs’, or ’post-cs’." # appended when src == "inquiries": " Always filter by one specific inquiry_type value: ’pre’ or ’post’."
Level-1 SQL atomic question with triplet constraint	TRIPLET-COMBINATION MODE - still generate a SINGLE-TABLE atomic question. Use ONLY the `{src}` primary source table; do NOT use JOINs or any other table. The question must be specifically constrained to this triplet entity combination: {col}=’{val}’, {col}=’{val}’. The SQL MUST include these exact filters: {col} = ’{val}’ AND {col} = ’{val}’.

Appendix

Table 26: Generation hints inserted into the {hint} field of the Atomic QA Generation prompt when building Level-1 SQL atomic questions.

Applied case	Hint
Level-1 text atomic question sales scripts	# src == "sales_scripts": TRIPLET TEXT CONTEXT - this text source is scoped by seller + product only: - seller: {seller} - product: {product} - buyer is NOT part of sales_scripts scope; do not mention the buyer in the question or search_query. The question must explicitly name the seller ({seller}) and the product ({product}). Ask about what the seller said, emphasized, described, or promised in the sales script. # + diversity_note, appended only when diversity_idx > 0: This is question variation #{idx} - generate a question from a DIFFERENT angle than you would generate by default. Be specific to the actual content of the script.
Level-1 text atomic question reviews text	# src == "reviews_text": TRIPLET TEXT CONTEXT - this text source is scoped by seller + buyer + product: - seller: {seller} - buyer: {buyer} - product: {product} The question must explicitly name the buyer ({buyer}), seller ({seller}), and product ({product}). # category_line: if category_filter -> first line; else -> second line: Focus ONLY on the ’{category_filter}’ review_category record. Use review_category metadata to make the question specific. # + diversity_note (diversity_idx > 0), with one review angle: This is question variation #{idx} - generate a question from a DIFFERENT angle than the default one. [angle 1] Ask about the buyer’s overall opinion, satisfaction, or impression. [angle 2] Ask only about the explicit reason, praised point, criticized point, or concrete detail stated in the review. Do not ask for an overall summary. Avoid near-duplicate wording across variations.
Level-1 text atomic question inquiries text	# src == "inquiries_text": TRIPLET TEXT CONTEXT - this text source is scoped by seller + buyer + product: - seller: {seller} - buyer: {buyer} - product: {product} The question must explicitly name the buyer ({buyer}), seller ({seller}), and product ({product}). # type_line: if category_filter -> first line; else -> second line: Focus ONLY on the ’{category_filter}’ inquiry_type record. Use inquiry_type metadata (pre or post) to make the question specific. # + diversity_note (diversity_idx > 0), with one inquiry angle: This is question variation #{idx} - generate a question from a DIFFERENT angle than the default one. [angle 1] Ask about the buyer’s main question, request, or concern. [angle 2] Ask about concrete details discussed, such as price, shipping, timing, or policy. [angle 3] Ask about the seller’s response, clarification, or resolution to the buyer’s concern. Avoid near-duplicate wording across variations.

Appendix

Table 27: Generation hints inserted into the {hint} field of the Atomic QA Generation prompt when building Level-1 text atomic questions.

Applied case	Hint
Set-answer follow-up atomic question AGG variant	[!] CRITICAL - {col} filter MUST use this exact IN clause (do NOT hardcode names): WHERE {col} IN ({in_clause}) {review_hint} These {n} {entity_label}s form a SET. Generate an AGGREGATION question: - Returns a SINGLE scalar (COUNT / SUM / AVG / MIN / MAX) - About ALL {n} {entity_label}s as a group - not about any individual - Use ’{placeholder}’ to refer to the group in the question text SQL examples (adapt to the actual columns in the data above): SELECT COUNT(*) FROM ... WHERE {col} IN ({in_clause}) SELECT AVG(review_rating) FROM ... WHERE {col} IN ({in_clause}) [+ AND review_category = ’...’ when source = reviews]
Set-answer follow-up atomic question FILTER variant	[!] CRITICAL - {col} filter MUST use this exact IN clause (do NOT hardcode names): WHERE {col} IN ({in_clause}) {review_hint} These {n} {entity_label}s form a SET. Generate a FILTERING/NARROWING question: - Returns one or more {entity_label} names from this set (NOT a scalar) - Use a single condition: argmax/argmin OR a numeric threshold - Use ’{placeholder}’ to refer to the group in the question text - Do NOT use LIMIT - use a WHERE condition instead: argmax -> AND metric = (SELECT MAX(metric) FROM ... WHERE {col} IN (...)) threshold -> AND metric > value SQL examples: SELECT DISTINCT {col} FROM ... WHERE {col} IN ({in_clause}) AND total_price = (SELECT MAX(total_price) FROM ... WHERE {col} IN ({in_clause})) SELECT DISTINCT {col} FROM ... WHERE {col} IN ({in_clause}) [+ AND review_category=’...’ AND review_rating >= 4 when source = reviews; else AND refund_rate > 20]
Scalar-answer follow-up atomic question	[!] SCALAR THRESHOLD HINT - use this numeric threshold from a previous step: Threshold value (MUST appear verbatim in the question): {scalar_val} Threshold measures: ’{measure_label}’ {review_hint} Generate a question that finds which entities have {measure_label} {op} {scalar_val}. - Include the exact numeric value {scalar_val} in the question text - Entity columns to group/filter by: {entity_cols} # if metric table (buyer_metrics/seller_metrics/products/sellers): SQL pattern (metric table - direct WHERE comparison): SELECT DISTINCT {entity_example} FROM {table} WHERE metric_col {op} {scalar_val} # else (raw table): SQL pattern (raw table - GROUP BY + HAVING): SELECT DISTINCT {entity_example} FROM {table} WHERE review_category = ’...’ [only when source = reviews] GROUP BY {entity_example} HAVING {agg_expr} {op} {scalar_val}

Appendix

Table 28: Generation hints inserted into the {hint} field of the Atomic QA Generation prompt when generating follow-up atomic questions. The hint is selected according to the answer type of the corresponding Level-1 atomic question: Set, or Scalar.

Panel of Table 29 — Table 29: Prompt for SQL Atomic QA Validation.

Panel of Table 30 — Table 30: Prompt for Text Atomic QA Validation.

Panel of Table 31 — Table 31: Prompt for Composite Question Composition via Referring-Expression Substitution, shared by the entity-linked and answer-linked (Set, S) pipelines. The {entity} field of the user prompt is filled with an entity name, a set placeholder, or a scalar value depending on the variant.

Panel of Table 32 — Table 32: Prompt for Composite QA Validation.

Panel of Table 33 — Table 33: Prompt for Auto-Schema Briefing.

Panel of Table 34 — Table 34: Prompt for Adaptive Complexity Routing.

Panel of Table 35 — Table 35: Prompt for SQL Source Picker

Panel of Table 36 — Table 36: Prompt for Planning.

Panel of Table 37 — Table 37: Prompt for SQL Query Generation in the SQL Tool.

Panel of Table 38 — Table 38: Prompt for Step Diagnosis.

Panel of Table 39 — Table 39: Prompt for Plan Revision.

Panel of Table 40 — Table 40: Prompt for Final Answer Generation.

Figure 7: Example of annotation template for verifying query quality over 4 criteria (Pass/Fail).

Agentic RAG

Agentic retrieval-augmented generation lets language models plan searches, use tools, and refine evidence before producing grounded answers.

See all 240 papers

May 7, 2026

AgenticRAG: Agentic Retrieval for Enterprise Knowledge Bases

We present AgenticRAG, a practical agentic harness for retrieval and analysis over enterprise knowledge bases. Standard RAG pipelines place significant burden of grounding on the search stack, constraining the language model to a fixed candidate set chosen deep in the retrieval process. Our approach reduces this overdependence by layering a lightweight harness on top of existing enterprise search infrastructure, equipping a reasoning LLM with search, find, open, and summarize tools enabling the model to iteratively retrieve information, navigate within documents, and analyze evidence autonomously. On three open benchmarks we observe substantial gains:

49.6\%

recall@1 on BRIGHT (+21.8 pp over the best embedding baseline), 0.96 factuality on WixQA (

+13\%

relative improvement), and

92\%

answer correctness on FinanceBench--within 2 pp of oracle access to true evidence. Ablation studies show that the most significant factor is the shift from single-shot retrieval to agentic tool use (

5.9\times

improvement), while multi-query search and in-document navigation contribute to both quality and efficiency. We present various design choices in our agentic harness that were informed by pre-production deployments. Our results demonstrate its suitability for real-world enterprise production environments.

Susheel Suresh, Hazel Mak, Shangpo Chou +2

Microsoft Corporation

Knowledge-Intensive QA Agentic Search Tool-Augmented Language Model Agents Agentic Retrieval Agentic RAG Agentic Reasoning+3

Ask PDF

Mar 6, 2026

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

Retrieval-Augmented Generation (RAG) is a framework in which a Generator, such as a Large Language Model (LLM), produces answers by retrieving documents from an external collection using a Retriever. In practice, Generators must integrate evidence from long contexts, perform multi-step reasoning, interpret tables, and abstain when evidence is missing. However, existing benchmarks for Generators provide limited coverage, with none enabling simultaneous evaluation of multiple capabilities under unified conditions. To bridge the gap between existing evaluations and practical use, we introduce LIT-RAGBench (the Logic, Integration, Table, Reasoning, and Abstention RAG Generator Benchmark), which defines five categories: Integration, Reasoning, Logic, Table, and Abstention, each further divided into practical evaluation aspects. LIT-RAGBench systematically covers patterns combining multiple aspects across categories. By using fictional entities and scenarios, LIT-RAGBench evaluates answers grounded in the provided external documents. The dataset consists of 114 human-constructed Japanese questions and an English version generated by machine translation with human curation. We use LLM-as-a-Judge for scoring and report category-wise and overall accuracy. Across API-based and open-weight models, no model exceeds 90% overall accuracy. By making strengths and weaknesses measurable within each category, LIT-RAGBench serves as a valuable metric for model selection in practical RAG deployments and for building RAG-specialized models. We release LIT-RAGBench, including the dataset and evaluation code, at https://github.com/Koki-Itai/LIT-RAGBench.

Koki Itai, Shunichi Hasegawa, Yuta Yamamoto +2

neoAI Inc., Tokyo, Japan · Tokyo Metropolitan University, Tokyo, Japan · The University of Tokyo, Tokyo, Japan

RAG Evaluation LLM Evaluation LLM Abstention

Ask PDF

Mar 2, 2026

RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm

Despite the success of Retrieval-Augmented Generation (RAG) in grounding LLMs with external knowledge, its application over heterogeneous sources (e.g., private databases, global corpora, and APIs) remains a significant challenge. Existing approaches typically employ an LLM-as-a-Router to dispatch decomposed sub-queries to specific sources in a predictive manner. However, this "LLM-as-a-Router" strategy relies heavily on the semantic meaning of different data sources, often leading to routing errors when source boundaries are ambiguous. In this work, we introduce RealRoute System, a framework that shifts the paradigm from predictive routing to a robust Retrieve-then-Verify mechanism. RealRoute ensures evidence completeness through parallel, source-agnostic retrieval, followed by a dynamic verifier that cross-checks the results and synthesizes a factually grounded answer. Our demonstration allows users to visualize the real-time "re-routing" process and inspect the verification chain across multiple knowledge silos. Experiments show that RealRoute significantly outperforms predictive baselines in the multi-hop Rag reasoning task. The RealRoute system is released as an open-source toolkit with a user-friendly web interface. The code is available at the URL: https://github.com/Joseph1951210/RealRoute.

Jiahe Liu, Qinkai Yu, Jingcheng Niu +5

Technical University of Denmark · University of Exeter · University of Toronto +4

Retrieval-Augmented Generation LLM Routing Multi-Hop QA LLM Answer Verification

Ask PDF

Abstract

Agentic RAG

Explore similar work

AgenticRAG: Agentic Retrieval for Enterprise Knowledge Bases

LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation

RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm