Organizations: Virginia Tech Alexandria, Virginia, U.S. · Florida Polytechnic University Lakeland, Florida, U.S. · Kuwait University Kuwait City, Kuwait
Current work on improving reliability in large language model (LLM)- generated answers has primarily leveraged Retrieval-Augmented Generation (RAG), knowledge-graph augmentation, and reinforcement learning. While these methods are adept at enhancing and measuring reliability through semantic similarity and faithfulness, they often struggle to distinguish semantic similarity from geographic validity. This is especially critical in natural hazard management domains where geographic granularity (i.e., town vs. city vs. state) is significant for decision-making, as responses valid in one municipality may not transfer to another. In such domains, a confidently wrong answer carries greater risk than abstaining. We present GeoRisk-RAG, a novel hierarchy-aware framework that addresses this geographic-validity gap through selective answering. This framework explicitly estimates geographic applicability using a Directed Acyclic Graph (DAG)-based distance for context retrieval before response generation. Experiments on a novel held-out wildfire-related question-answering (QA) dataset show that GeoRisk-RAG significantly reduces false confidence rates for location-dependent questions, lowering the rate to 0.009 compared with ~0.090 for standard semantic similarity and reranking baselines, while consistently achieving higher human preference alignment. This work provides a more comprehensive assessment of end-to-end RAG pipelines by integrating geographic validity and selective-answering behavior for safer decision-making in geospatial domains.
Figures & tables
Figure 1. Example responses from Standard RAG (top) and GeoRisk-RAG (bottom) with the latter showing a warning+response despite a lack of geographical context for Osage County in the document-base, while the former abstains.
Figure 2. High-level framework of the proposed GeoRisk-RAG for hierarchy-aware QA.
Figure 3. Distribution of the custom Wildfire QA benchmark dataset across states, question topics, and location dependency. Each row and column represents a question topic and a state respectively.
Statistic
Count
Percent
Total question-answer pairs
449
–
States covered
19
–
Unique cities covered
28
–
Avg tokens in Question
18
–
Avg tokens in Answer
156
–
Geographic granularity
Table 1. Statistics on Custom Wildfire QA Dataset
Metric
Flair
SpaCy
BERT
DeBERTa
Precision
0.943
0.829
0.842
0.865
Recall (Completeness)
0.928
0.775
0.884
0.891
F1-Score
0.929
0.789
0.853
0.878
Avg Latency (s) per Text
0.592
0.062
0.128
0.744
Total Execution Time (s)
135.670
13.900
29.399
170.404
Table 2. Location NER Performance on Out-of-Sample GeoVirus Dataset (n=229 texts)
Example query: “How do wildfire risks vary in Oregon?”
Query hierarchy: [Oregon, Pacific Northwest, United States]
Passage type
Passage hierarchy
Dist.
Shared ancestor
Relation
Decision
Exact Oregon passage
[Oregon, Pacific Northwest, United States]
0
Oregon
exact match
answer
Broader U.S. passage
[United States]
2
United States
passage broader than query
warn
Same-region passage
[California, West, United States]
2
West
different place, same ancestor
abstain
County-level passage
[Multnomah County, Oregon, Pacific Northwest, United States]
1
Oregon
passage more granular than query
warn
Table 3. Example of GeoRisk-RAG behavior for an Oregon-state level wildfire query.
Figure 4. Example of geographical hierarchical DAG, illustrating the hierarchy structure and handling for ambiguous entities. Each color represents a granularity level.
Method
Data Structure
Retrieval Logic
Purpose
Standard RAG
Text Embeddings
Vector cosine similarity only.
Standard, commonly used baseline.
Standard RAG + Cross Encoder
Reranked List
Semantic reranking of top 10 results.
Demonstrates that reranking alone is insufficient for geographic retrieval.
Granularity Match
Ordinal Scale
Matches based on geographic scope (city, county, state, region, country).
Highlights the need for relationships other than ancestor relationships.
Exact Keyword Match
String Literal
Text must match exactly (e.g., “Texas” == “Texas”).
Shows insufficiency of literal string matching for varied geographic aliases.
GeoRisk-RAG
Directed Acyclic Graph
Knowledge graph logic using Wikidata for multi-parent resolution.
Resolves tribal lands, ambiguous places, and census-designated areas.
Table 4. Summary of baseline methods used for comparison. Each method differs in its data structure, retrieval logic, and role in the study.
Method
Good Decision Quality Rate
Faithfulness
Appropriate Warning Rate
Over-Abstention Rate
False Confidence
Factual Correctness
Location-Dependent Queries
Standard RAG+Cross Encoder
0.751 (0.692–0.810)
0.773 (0.693–0.852)
0.001 (0.000–0.001)
0.086 (0.054–0.127)
0.090 (0.054–0.131)
0.557 (0.493-0.620)
Standard RAG
0.819 (0.769–0.869)
0.800 (0.714–0.876)
0.009 (0.000–0.023)
0.036 (0.014–0.063)
0.095 (0.059-0.136)
0.679 (0.615-0.742)
Keyword Match
0.908 (0.896–0.939)
0.909 (0.878–0.958)
0.005 (0.000–0.014)
0.045 (0.018–0.073)
0.023 (0.005–0.045)
0.615 (0.552–0.683)
Granularity Match
0.824 (0.769–0.873)
0.750 (0.666–0.845)
0.001 (0.000–0.001)
0.041 (0.018–0.068)
0.095 (0.059–0.136)
0.561 (0.498–0.624)
GeoRisk-RAG
0.946 (0.910–0.977)
0.965 (0.912–1.000)
0.195 (0.140–0.244)
0.030 (0.014–0.053)
0.009 (0.000–0.023)
0.761 (0.708–0.824)
Table 5. Reliability and selective-answering performance across geographically dependent and non-dependent wildfire QA tasks. GeoRisk-RAG achieves the strongest grounded decision behavior while substantially reducing false confidence.
bge-base-en-v1.5
bge-small-en-v1.5
Retrieved Passages (Semantic Retrieval Stability)
Standard+Cross Encoder
0.815
0.816
Granularity Match
0.820
0.829
Keyword Match
0.863
0.872
Standard RAG
0.814
0.821
GeoRisk-RAG
0.872
0.873
Table 6. Overall stability analysis across RAG model configurations.
Geographic- Error Type
Standard+ Cross -Encoder
Keyword Match
Granularity Match
Standard RAG
GeoRisk- RAG
Exact Geo Match
0.718
0.745
0.747
0.683
0.811
Geo Mismatch
0.732
0.700
1.000
0.686
0.685
No Geo context
0.350
0.420
0.419
0.472
0.357
Coarse Geo
0.812
0.889
0.867
0.889
1.000
Fine Geo
0.500
1.000
1.000
0.723
1.000
Table 7. Response correctness by geographic error type across retrieval methods.
Figure 5. GDR across same questions with different locations to evaluate answering behavior. Higher values indicate more reliable selective-answering behavior.
Figure 6. A location-specific query about Albuquerque, New Mexico highlights GeoRisk-RAG’s reasoning: rather than abstaining as all baselines do, the system retrieves state-level context and responds (which correctly aligns with presented ground truth answer) with a warning.
Appendix figures & tables5 assets
Supplementary material from the paper’s appendix.
Appendix
Topic
Description
Count
General Wildfire Behavior
Causes, spread, fire science, smoke behavior, seasons, risk factors.
42
Home and Property
Protecting homes, defensible space, structures, insurance for homes, property damage.
52
Codes and Regulations
Laws, building codes, permit rules, compliance requirements, official regulations.
29
Specific Fire Event
Questions about a named wildfire, a particular incident, location-specific active or past fire.
3
Community
Questions about how the community can help, community-level impacts.
10
Land
Forests, vegetation, soil, erosion, watersheds, land management, acreage, habitat.