FieldGuideRAG: Image-Grounded GraphRAG for Fine-Grained Recognition with Vision-Language Models
Organizations: University of Memphis · National University of Singapore
Abstract
We present FieldGuideRAG, a training-free GraphRAG method in which a vision-language model, or VLM, recognizes fine-grained species by comparing the query with real photographs stored on the nodes of a knowledge graph. Multimodal GraphRAG systems describe each entity in text, and text conveys little of the visual detail that separates look-alike species. Our graph stores one photograph on every species node and links each species to its relatives, its geographic range and its ecological facts. A frozen VLM verifies the query against the photograph of each candidate node. FieldGuideRAG fuses the verification scores with the range edge and answers questions about the organism from the fact edges of the verified species. On CUB-200, a Wikidata graph and four iNaturalist domains, the node photograph improves verification over the species name on every graph, while text descriptions add little. Verification of the candidate pool exceeds a SigLIP retriever by up to 16 points, the range edge adds 7 to 19 points on iNaturalist, and answering from fact edges is correct on up to 89.5% of 28,820 ecological questions. The verification scores also detect queries whose species is missing from the graph. Graphs, prompts and code will be released.
Figures & tables
| pool | shortlist | ||||||
|---|---|---|---|---|---|---|---|
| Graph | species | test | dev | node text | size | recall | recall |
| CUB-93 | 93 | 372 | 279 | attributes | 16.8 | 99.2 | 75.8 |
| CUB-200 [ 35 ] | 200 | 5,794 | 600 | attributes | 12.8 | 95.1 | 78.6 |
| Wikidata graph | 93 | 372 | 279 | Wikipedia | 16.4 | 99.5 | 90.1 |
| iNat birds [ 33 ] | 406 | 812 | 812 | – | 14.3 | 83.4 | 54.6 |
| iNat fungi | 341 | 682 | 682 | Wikipedia | 10.8 | 71.6 | 52.2 |
| Method | CUB-200 | Wikidata graph | iNat birds | iNat fungi | iNat Lepidoptera | iNat plants | |
| SigLIP retriever | 53.2 | 63.7 | 25.0 | 30.2 | 16.3 | 32.5 | |
| Zero-shot VLM | 51.9 | 45.4 | 25.5 | 4.8 | 5.5 | 14.5 | |
| Text RAG, | 62.1 | 58.9 | – | 29.6 | 14.9 | – | |
| Multi-image RAG, | 66.7 | 72.8 | 33.1 | 28.3 | 18.5 | 33.0 | |
| Verification, | 63.0 | 72.8 | 30.8 | 33.1 | 20.5 | 36.0 | |
| FieldGuideRAG, pool | 69.0 | 70.7 | 39.4 | 34.5 | 28.4 | 38.8 |
| Qwen2.5 | Qwen3 | |||||
|---|---|---|---|---|---|---|
| Graph | name | text | photo | name | text | photo |
| CUB-93 | 42.5 | 45.2 | 51.9 | 43.0 | 44.6 | 55.4 |
| CUB-200 | 54.6 | 54.0 | 63.0 | 49.9 | – | 62.9 |
| Wikidata | 49.2 | 48.7 | 72.8 | 51.1 | 52.7 | 73.4 |
| iNat birds | 27.1 | – | 30.8 | 25.6 | – | 32.0 |
| iNat fungi | 22.3 | 23.6 | 33.1 | 22.6 | 26.4 | 34.2 |
| Photographs | Verification | ||
|---|---|---|---|
| per node | Retriever | Qwen2.5 | Qwen3 |
| 1 | 43.8 | 51.9 | 55.4 |
| 3 | 65.3 | 66.7 | 68.3 |
| 30 | 85.2 | 70.2 | 73.7 |
| Qwen2.5 | Qwen3 | |||||
|---|---|---|---|---|---|---|
| Graph | ||||||
| CUB-200 | ||||||
| shortlisted only | ||||||
| iNat fungi | – | – | ||||
| iNat Lepidoptera | – | – | ||||
| iNat plants | – | – | ||||
| Qwen2.5 | Qwen3 | |
| Multi-image RAG on CUB-93 by slot of the correct photograph | ||
| slot 1 | 86.5 | 74.1 |
| slot 2 | 63.8 | 72.0 |
| slot 3 | 54.6 | 69.9 |
| slot 4 | 57.4 | 67.4 |
| answers naming slot 1 | 43.2 | 31.2 |
| birds | fungi | Lepid. | plants | ||
|---|---|---|---|---|---|
| Method | Qwen3 | Qwen2.5 | Qwen3 | Qwen3 | Qwen3 |
| Retriever, range edge | 45.6 ∗ | 45.6 ∗ | 36.4 ∗ | 29.1 ∗ | 49.5 ∗ |
| FieldGuideRAG, range edge | 57.6 | 58.5 | 46.2 | 44.9 | 58.2 |
| Verification, location in prompt | 56.9 | 52.1 ∗ | 43.5 | 43.8 | 57.0 |
| Multi-image, GPS | 40.3 ∗ | 36.2 ∗ | 33.0 ∗ | 25.2 ∗ | 42.3 † |
| Multi-image, range-ranked, GPS, counts | 58.9 | 53.7 ∗ | 40.9 ∗ | 40.9 † | 57.0 † |
| Answer source | Qwen2.5 | Qwen3 |
| Zero-shot VLM | 47.9 ∗ | 44.1 ∗ |
| Prompt with facts of the most similar species | 82.2 ∗ | 80.5 ∗ |
| Prompt with facts of four candidates | 83.3 ∗ | 83.2 ∗ |
| Prompt with photographs and facts of four candidates | 83.8 ∗ | 83.5 ∗ |
| Fact edge of the multi-image RAG answer | 88.8 † | 87.7 † |
| FieldGuideRAG, fact edge of the verified species | 89.5 | 89.2 |
| Model | Source | diet | migration | habitat | lifestyle | mass |
|---|---|---|---|---|---|---|
| Qwen2.5 | prompt | 87.1 | 85.9 | 75.0 | 88.1 | 82.9 |
| FieldGuideRAG | 91.8 | 90.7 | 83.0 | 93.5 | 88.6 | |
| Qwen3 | prompt | 87.6 | 87.2 | 74.9 | 83.1 | 84.6 |
| FieldGuideRAG | 91.6 | 89.8 | 81.9 | 93.4 | 89.2 |
| Candidates | size | recall | Qwen2.5 | Qwen3 |
| Graph shortlist | 4 | 78.6 | 63.0 | 62.9 |
| + generic sentence | 4 | 78.6 | – | 63.6 ∗ |
| + note of another species | 4 | 78.6 | 61.8 ∗ | 59.5 ∗ |
| + contrastive note | 4 | 78.6 | 63.1 | 65.1 ∗ |
| Name group of top species | 8.6 | 83.0 | 62.7 | 62.4 |
| Mined confusion edges | 4 | 83.0 | 67.1 | 66.3 |
| development | test | ||||
|---|---|---|---|---|---|
| Domain | Model | additive | log-linear | additive | log-linear |
| birds | Qwen2.5 | 57.3 | 60.2 | 58.0 | 58.5 |
| birds | Qwen3 | 54.3 | 57.0 | 53.9 | 57.6 |
| fungi | Qwen2.5 | 43.3 | 44.9 | 44.4 | 41.8 |
| fungi | Qwen3 | 43.5 | 44.9 | 40.9 | 46.2 |
| Lepidoptera | Qwen2.5 | 38.9 | 41.9 | 40.8 | 44.4 |
| Graph | Model | name | name and text | name and photograph | name, photograph and text |
|---|---|---|---|---|---|
| CUB-93 | Qwen2.5 | 42.5 / 0.786 | 45.2 / 0.787 | 51.9 / 0.857 | 54.6 / 0.855 |
| CUB-93 | Qwen3 | 43.0 / 0.751 | 44.6 / 0.769 | 55.4 / 0.850 | 57.5 / 0.866 |
| CUB-93 | Qwen3-32B | 50.0 / 0.844 | 50.0 / 0.841 | 57.8 / 0.901 | – |
| CUB-200 | Qwen2.5 | 54.6 / 0.851 | 54.0 / 0.841 | 63.0 / 0.913 | – |
| CUB-200 | Qwen3 | 49.9 / 0.822 | – | 62.9 / 0.910 | – |
| Wikidata | Qwen2.5 | 49.2 / 0.776 | 48.7 / 0.750 | 72.8 / 0.884 | 71.5 / 0.869 |
| Graph | Model | no sentence | attribute note | generic | note of another | LLM note | LLM note of another |
|---|---|---|---|---|---|---|---|
| CUB-93 | Qwen2.5 | 51.9 | 54.8 ( ) | 53.2 ( ) | 54.0 ( ) | 52.4 ( ) | – |
| CUB-93 | Qwen3 | 55.4 | 59.1 ( ) | – | 50.8 ( ) | 56.2 ( ) | – |
| CUB-93 | Qwen3-32B | 57.8 | 58.6 ( ) | – | – | – | – |
| iNat fungi | Qwen2.5 | 33.1 | – | – | – | 30.2 ( ) | 31.4 ( ) |
| iNat fungi | Qwen3 | 34.2 | – | – | – | 34.6 ( ) | 33.1 ( ) |
| iNat Lepidoptera | Qwen2.5 | 20.5 | – | – | – | 19.2 ( ) | 18.7 ( ) |
| Qwen2.5 | Qwen3 | |
|---|---|---|
| Multi-image RAG | 73.0 | 72.7 |
| Verification | 68.8 | 73.0 |
| Graph-shortlist recall | 75.8 | 75.8 |
| Qwen2.5 | Qwen3 | |||
|---|---|---|---|---|
| Graph | pool | chained | pool | chained |
| CUB-93 | 59.7 | 63.4 ∗ | 61.8 | 58.3 |
| CUB-200 | 69.0 | 70.4 ∗ | 68.8 | 68.3 |
| Wikidata graph | 70.7 | 71.0 | 69.4 | 68.3 |
| iNat fungi | 34.5 | 30.9 ∗ | 38.1 | 35.2 |
| iNat Lepidoptera | 28.4 | 27.5 | 32.6 | 31.2 |
| Model | Source | diet | migration | habitat | lifestyle | mass |
|---|---|---|---|---|---|---|
| Qwen2.5 | weighted fact edges | 91.5 | 90.9 | 82.2 | 93.6 | 88.4 |
| Qwen3 | weighted fact edges | 91.0 | 89.7 | 80.9 | 92.2 | 88.2 |
| Both | candidates agree | 29.0 | 14.3 | 2.0 | 15.6 | 1.0 |
| Method | accuracy |
|---|---|
| Retriever | 43.8 |
| Verification of the graph shortlist | 57.8 |
| Verification of the pool | 63.4 |
| Multi-image RAG on the graph shortlist | 62.9 |
| on queries whose shortlist contains the true species | 78.0 |
| Verification on the same queries | 76.6 |
Graph Retrieval-Augmented Generation
Graph Retrieval-Augmented Generation connects language models to knowledge graphs to retrieve relational evidence for grounded, multi-hop answers.
See all 247 papers