Conversational AI-Enhanced Exploration System to Query Large-Scale Digitised Collections of Natural History Museums
Authors: Yiyuan Wang, Andrew Johnston, Zoë Sadokierski, Rhiannon Stephens, Shane T. Ahyong
Organizations: School of Computer Science, University of Technology Sydney, Australia · School of Design, University of Technology Sydney, Australia · Australian Museum Research Institute, Australian Museum, Australia
Recent digitisation efforts in natural history museums have produced large volumes of collection data, yet their scale and scientific complexity often hinder public access and understanding. Conventional data management tools, such as databases, restrict exploration through keyword-based search or require specialised schema knowledge. This paper presents a system design that uses conversational AI to query nearly 1.7 million digitised specimen records from the life-science collections of the Australian Museum. Designed and developed through a human-centred design process, the system contains an interactive map for visual-spatial exploration and a natural-language conversational agent that retrieves detailed specimen data and answers collection-specific questions. The system leverages function-calling capabilities of contemporary large language models to dynamically retrieve structured data from external APIs, enabling fast, real-time interaction with extensive yet frequently updated datasets. Our work provides a new approach of connecting large museum collections with natural language-based queries and informs future designs of scientific AI agents for natural history museums.
Figures & tables
Figure 1. Development process of the Australian Museum Collection Explorer.
Figure 2. Photos taken at the workshops (top) and interview questions in focus-group activities (bottom).
Expressed Need
Current Challenge
Design Goal
Communicate the vast scale and diversity of back-of-house collections
Collections are invisible to the public; the volume and richness of data is difficult to convey
Comprehensibility of scale : Make the scale of collections immediately perceptible and understood
Reduce barriers to accessing scientific and research information
Non-specialised audiences lack both technical database skills and natural science domain knowledge
Query accessibility : Enable information retrieval without technical expertise
Automate audience–expert message flow (i.e. “Ask an Expert” service)
Manual workflows to answer web-submitted questions are not scalable and require multi-step staff routing
Response reliability : Deliver accurate, collection-specific information efficiently
Reach diverse audiences including remote and underserved communities
Physical museum visits and staff consultations have geographic and temporal constraints
Ubiquitous access : Enable exploration of collections from anywhere, at any time
Table 1. Mapping stakeholder requirements to design goals. A three-column table mapping expressed needs from focus group attendees to current challenges and resulting design goals. Four rows cover: communicating collection scale leading to comprehensibility of scale; reducing access barriers leading to query accessibility; automating expert services leading to response reliability; and reaching diverse audiences leading to ubiquitous access.
Figure 3. The first interactive prototype we developed (left); ALA Lens (middle); Museum Animal Factsheets (right).
Figure 4. The second interactive prototype we developed: a bird collection explorer consisting of an initial version of the interactive map (left) and the conversational agent (right).
Item
Summary
Age ( M±SD )
51.6±24.9 years
Gender
7 female, 5 male
Prior AI experience
4 regular, 5 limited, 3 none
In-person museum engagement
All weekly (volunteer work)
Front-of-house exhibition visits
2 monthly, 3 bimonthly, 7 few times per year
Remote museum engagement
3 weekly, 2 monthly, 5 few times per year, 2 never
Table 2. Summary of participant profile in Design Iteration II ( N=12 ).
Figure 5. An overview of the system architecture for the Australian Museum Collection Explorer.
Figure 6. The Australian Museum Collection Explorer interface, showing the interactive map with a popup displaying details of a Musk Lorikeet specimen (left), and the conversational agent with an introduction message (right).
Figure 7. Examples of collection data queries: searching for sugar gliders collected from 2000–2010 (left); requesting a count of Christmas beetles from NSW (right). Both responses include direct links to ALA (information source) for verification and further exploration.
Figure 8. Image output example 1: spotting a Macleay’s Swallowtail record on the map and requesting additional images of this species.
Figure 9. Image output example 2 (left): requesting fish specimens from Queensland. Image input example (right): identifying birds from a photograph taken in the wild as crested pigeons, with descriptive information and a link to related museum records.
Digital cultural heritage collections encode rich relations among objects, places, periods, providers, images, and source institutions, yet users typically access them through keyword search and faceted browsing. We present CULTUREBOT, an interactive framework for natural-language exploration of cultural heritage collections that combines knowledge-graph-guided retrieval, lexical-semantic retrieval, and source-linked generation. CULTUREBOT maps user queries to structured catalogue relations, enriches retrieved records with textual evidence, and generates answers that link back to the underlying repository records. We instantiate the framework on 81,852 SearchCulture records and release an online demonstration. In an expert evaluation involving six cultural-heritage specialists, 55 persona-grounded queries, and 250 evaluations, the graph-guided system was rated significantly higher than two general-purpose assistants and a non-graph ablation, with the largest gains in evidence grounding, diversity, reasoning, and completeness. These results support graph-guided retrieval as a practical way to expose the relational structure of cultural collections through conversational access while preserving links to catalogue evidence.
We present the Living Library, an end-to-end framework for transforming fragmented digital archives into governed, conversational, in-person exhibit experiences. Developed and deployed at the Theodore Roosevelt Presidential Library, the framework comprises four layers: digitization and corpus creation, AI-powered processing, retrieval and reasoning, and an optional embodied conversational interface. The first three layers aggregate a 300,000-record collection, apply OCR and structured metadata enrichment for expert curatorial review, and publish records to a hybrid dense/semantic index. Expert review is conducted through the Archivist App, a curator-facing interface that supports correction of AI-generated transcriptions and metadata. The governed corpus powers both a researcher-facing interface and Talk to TR, a continuously operating exhibit that embodies Theodore Roosevelt as a full-scale digital human within a museum environment. To support live, face-to-face interactions, Cross-Era Analogical Grounding reframes contemporary questions through documented historical parallels, allowing Roosevelt to address present-day topics without inventing facts. Dual-path retrieval and end-to-end streaming keep responses grounded and responsive. Layered watchdogs, visitor-session isolation, automated conversation management, and independently restartable services enable reliable unattended operation for hundreds of visitors. Avatar realism, spatial audio, lighting, staging, and conversational design are developed and evaluated as an integrated experience. Rather than report a controlled benchmark, we describe lessons from operating Talk to TR as a public exhibit and offer a transferable model for transforming archival collections into believable, in-person conversational experiences.
Modern scientific facilities and instruments generate datasets at scales that are difficult for individual researchers to discover, access, and explore. Although many datasets are publicly available, using them often requires familiarity with repository organization, data formats, multiresolution structures, and visualization parameters. We present WebVisus, a constrained and resource-aware multi-agent system for discovering and autonomously exploring remote, multiresolution scientific datasets. Given a natural-language research question, WebVisus identifies the user's intent and launches an autonomous exploration agent that examines slices, volumes, and timesteps while adapting data resolution and retrieval quality to available client memory and computational resources. This design supports progressive exploration without complete dataset downloads or manual configuration of low-level visualization parameters using natural languages. We report the system architecture, constrained agent protocol, resource-aware access mechanism, and case studies evaluating autonomous visual exploration and resource-aware agentic access across scientific datasets.
Aashish Panta, Hugo Lee, Giorgio Scorzelli +2
University of Utah Salt Lake City, Utah, USA · Jet Propulsion Laboratory, Caltech Pasadena, CA, USA