Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset

Dany Haddad; Dan Bareket; Joseph Chee Chang; Jay DeYoung; Jena D. Hwang; Uri Katz; Mark Polak; Sangho Suh

cs.HCFeb 26, 2026

Understanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset

Authors: Dany Haddad, Dan Bareket, Joseph Chee Chang, Jay DeYoung, Jena D. Hwang, Uri Katz, Mark Polak, Sangho Suh,

Organizations: Allen Institute for AI

Abstract

AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools (a literature discovery interface and a scientific question-answering interface) within an LLM-powered retrieval-augmented generation platform. Using this dataset, we characterize query patterns, engagement behaviors, and how usage evolves with experience. We find that users submit longer and more complex queries than in traditional search, and treat the system as a collaborative research partner, delegating tasks such as drafting content and identifying research gaps. Users treat generated responses as persistent artifacts, revisiting and navigating among outputs and cited evidence in non-linear ways. With experience, users issue more targeted queries and engage more deeply with supporting citations, although keyword-style queries persist even among experienced users. We release the anonymized dataset and analysis with a new query intent taxonomy to inform future designs of real-world AI research assistants and to support realistic evaluation.

Figures & tables

Panel of Figure 1 — Figure 1: Screenshots of the two Asta interfaces used in our study ( Singh et al., 2025 ; Allen Institute for Artificial Intelligence, 2025 ) .

Intent	Example Query
Broad Topic Exploration	GLP-1 and diabetes
Specific Factual Retrieval	What are the four core concepts of Rotter’s theory?
Concept Def. & Exploration	Summarize the concept of “Technoimagination” […] by Vilém Flusser
Comparative Analysis	Describe the trade-offs between HBr and Cl2 plasma gases for reactive ion etching of polysilicon
Causal & Relational Inquiry	Relation between nighttime digital device use and sleep quality […]
Method. & Proced. Guidelines	How often should I collect mosquitoes for dengue surveillance?

Table 1: Representative examples of query intents illustrating of user information needs.

Phrasing Style	Example Query
Keyword-style	piracetam efficacy
Natural Language Q.	How are emerging digital technologies reshaping sustainable development outcomes?
Explicit Instruction	Write an essay as a doctor of literary discourse analysis[…]
Complex Context Narrative	When training large language models, it’s essential to develop benchmarks that assess true problem-solving ability, not just factual recall. [detailed discussion about knowing Berlin’s train stations vs. planning a novel journey] [C]onduct a literature review on efforts to curate evaluations that explicitly test synthesis rather than memorization.
Multi-part Query	9. Identify and summarise key international instruments (e.g., UN Charter, Budapest Convention, UNSC Resolutions). Discuss their relevance and application […]
Boolean/Logical Ops	(“Family Centered” OR “Family Centred”) AND (challeng* OR implantat*)

Table 2: Representative examples of query phrasing styles observed across PF and SQA.

Search Constraint	Example Query
Methodology-Specific	federated learning where the authors had coined the new terms for defining client types like homogenous clients, selfish clients…
Publication Type/Quality	Please find scientific references only from highly reputable and scopus-indexed international journals…
Temporal Constraints	Pharmacological Activities of Ganoderma applanatum. Use journals from 2020-2025
Metadata-Based	Papers from SIG CHI that have use cases with an “adaptive task challenge”
Cit./Impact-Based	What is the most cited paper of Orit Hazzan ?
Data/Resource Availability	search for documents that provide data of the amount of comic books are purchased among different classes in the United States.

Table 3: Representative examples of search constraints (criteria) specified in user queries across PF and SQA.

Panel of Figure 3 — Figure 3: Query phrasing styles and intents (% of queries). Keyword queries dominate the distribution. Asta users initiate more natural language queries than S2 users. Most queries are broad exploration or concept explanation; S2 is skewed towards broad and specific paper retrieval queries. See Table 6 for complete intent distribution, and Table 7 for criteria distribution.

Query Label	single-query	inexperienced	experienced
Broad Topic Exploration	$61.23_{\pm 1.38}$	$55.63_{\pm 0.98}$	$53.48_{\pm 0.99}$
Causal and Relational Inquiry	$15.78_{\pm 1.03}$	$15.97_{\pm 0.72}$	$17.06_{\pm 0.75}$
Citation & Evidence Finding	$6.25_{\pm 0.69}$	$8.65_{\pm 0.55}$	$9.65_{\pm 0.59}$
Methodology-Specific Criteria	$41.84_{\pm 1.4}$	$45.50_{\pm 0.98}$	$47.17_{\pm 0.99}$

Table 5: Query label by experience stage, tracking the same user population over time (percentages with 95 % confidence intervals).

Appendix figures & tables36 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 4: Section expansion distribution (on a log scale) showing which sections users expand first in SQA responses. Section index 1 has the largest number of expansions. Users tend to start on section 0 or 1 and end on a section between 2–4.

Figure 5: Section transition heatmap showing in-order vs out-of-order reading patterns in SQA. Sequential expansion is the dominant behavior, but there is notable backward traversal behavior as well as return to 0 (the introduction) behavior. Users also close and then reopen a section (presumably to read the TL;DR which is only visible when a section is collapsed).

Intent	%	Example
SQA
Broad Topic Expl.	51.6	What are the types of approaches to employer branding?
Concept Def. & Expl.	28.2	Digital Banking BI Maturity Models
Causal & Rel. Inquiry	19.1	why evaluation as a construct of metacognition is very important for writing competence to emerge?
Specific Factual Retr.	12.6	what do say paper of sharma2019 and kamga 2016 about resistace in chemotherapy in AML IN NOTCH pathway
Method. & Proc. Guid.	9.1	design non-stationnary wavelets trough […] bezout’s equation

Appendix

Table 6: Query intent distribution (% of queries) with random examples.

	Asta	S2
Methodology	42	29
Pub. Quality	11	3
Metadata	9	12
Temporal	5	2
Data Avail.	1	0.4
Cit. Impact	0.8	$<$ 0.1

Appendix

Table 7: Query criteria distribution (% of queries). Methodology-related criteria are most common on Asta.

Figure 6: Distribution of coarse-grained fields of study across PF, SQA, and S2.

Figure 7: Reading order shown through section flow Sankey diagram, illustrating user navigation patterns through SQA response sections.

Characteristic	Description
Keyword-style Query (phrasing style)	Short, often fragmentary, queries resembling search engine keywords or subject headings. No verbs or complete sentences; typically a list of nouns or concepts separated by spaces or simple punctuation.
Methodology-Specific Criteria (query criteria)	Queries where the user requires specific methods, approaches, analytical techniques, or experimental designs to be present in the search results. This can include demands for computational models, experimental paradigms, or meta-analyses.

Appendix

Table 8: Common query characteristics observed in Asta.

Figure 8: The fraction of queries from each field of study that have a given intent on Asta. The distribution reflects the research tasks common in the given fields.

Figure 9: Action engagement trends by query index showing how users perform different actions as they gain experience with the system. Shaded regions represent 95 % confidence intervals. PF reports tend to receive less click engagement over time compared to SQA reports which grow in engagement, the reason likely being that PF results can be consumed passively without interacting with the web content at all whereas most of the content generated by SQA can only be accessed after clicking on the web page.

Figure 10: Distribution of text feedback categories reflecting user expectations and experiences. The comprehensiveness of the response is the most common complaint ( Lack of Depth and Citation & Referencing Issues ).

Category	Example Feedback
Response Quality	develop more the introduction and add more citations (references)
Response Quality	this is leaving out some key details from Magalhaes et al 2013
Response Quality	Out of date
Query Refinement	Its very good but I need to add a historical component
Feature Requests	would be a great place to a diagram
Feature Requests	amazing! Coudl you create a PDF

Appendix

Table 9: Example user feedback by category. Some users seem to expect the feedback submission to provide a multiturn chat experience.

Pattern	Tool	Example Query
Template Filling	PF	fill this tabel with 10 jurnal bellow:… [table template and citations]
Template Filling	SQA	for sacubitril find all: "IUPAC Name: CAS Number: Molecular Formula:…" [15+ fields]
Prompting	SQA	You are an expert research assistant specializing in computational geosciences and machine learning.
Prompting	PF	Find papers…The model must be capable of…
Persona Adoption	SQA	Think of yourself as experienced professor…Please write me a phd proposal…devour Turnitin detection bots
Collaborative Writing	SQA	I’m working on my paper… [LaTeX section] add papers from TSE, TOSEM, ICSE

Appendix

Table 10: Examples of interesting usage patterns showing how users probe the capabilities of LLM-powered research tools. Users go beyond basic IR tasks and expect Asta to work as a collaborative research assistant.

Metric	SQA	PF
User Prevalence
Users with duplicate queries	18.8 %	14.8 %
Users who revisited reports	50.5 %	42.1 %
Temporal Patterns
Median time between duplicates	15.9 min	5.8 min
Median time between revisits	3.8 hours	5.9 hours

Appendix

Table 11: Comparison of duplicate query submission versus report revisiting behavior.

Tool	Gap	Query Pair
SQA	4.0 min	First: History of experiential learning of science
		Second: History of experiential learning of science
SQA	1.4 min	First: postbiotics food
		Second: postbiotics in food industry
SQA	6.4 min	First: [Long query requesting literature review]
		Second: [Same query + “write this in future tense”]

Appendix

Table 12: Representative examples of duplicate queries showing exact duplicates and incremental refinements. Time gaps indicate minutes between submissions.

Figure 11: Fraction of reports on which the given action has been performed across PF and SQA, showing how users engage with different features. Section expansion actions are the most common, likely because users must click to expand the section text in the SQA response. Note that link clicks are much more common than thumbs up/down feedback; below we also show that link clicks are also strongly associated with use satisfaction.

Figure 12: Survival analysis showing the Kaplan-Meier curve associated with various actions that users performed during their first visit to Asta. Users who did not perform any action take longer to return to Asta than those who do.

Panel of Figure 13 — Figure 13: Action return rates showing the probability of users returning after different action types. Actions of first time users are likely more affected by novelty effect.

Query Type	Example Query
Citation & Evidence Finding	Can you find for me explicitly mentioned in a paper when an anti-reflection coating was used to improve the temporal contrast of a high power laser due to it having to be at s polarization.
Broad Topic Exploration	find me articles on AI implementation in SMEs.
Temporal Constraints	What are the drawbacks of Intelligent tutoring systems from before they used LLMs
Data/Resource Availability	dataset with image and sun azimuth and sun elevation
Content Generation and Expansion	Create an article with the title “Material Strength Design on Excavator Arm Model” with references of 40 articles or journals

Appendix

Table 13: Example PF queries by type. PF performs well on Citation & Evidence Finding and Broad Topic Exploration queries but struggles with queries having Temporal Constraints, Data/Resource requirements, or Content Generation requests.

Query Type	Example Query
Concept Definition and Explanation	explain what multimodal and multisensor are and the differents betwen them
Complex Contextual Narrative	Recent evidence highlights the critical role of growth hormone secretagogue receptor signaling in hippocampal synaptic physiology, mediated through dopamine receptor activity… [describes detailed molecular mechanisms]
Citation Format Specification	Use APA style 7 for in-text citations, citing authors’ names, publication dates, and page numbers. Write a theoretical framework of 17,000 words about schooling of rural girls…

Appendix

Table 14: Example SQA queries by type. SQA excels at Concept Definition and Complex Contextual Narrative queries but struggles with Citation Format Specification requests.

Table 27

Panel of Figure 14 — Figure 14: Estimated coefficients for the linear models predicting clicks and return on PF and SQA. Coefficients are shown with 95% confidence intervals. Only coefficients significant after Benjamini-Hochberg correction are displayed.

Field of Study	OR
Law & Legal Studies	$1.25_{\pm 0.22}$
Sociology	$1.24_{\pm 0.12}$
Agricultural Sciences	$1.23_{\pm 0.16}$
Electrical Engineering	$1.20_{\pm 0.13}$
Clinical Medicine	$1.20_{\pm 0.12}$
Business & Management	$1.20_{\pm 0.11}$

Appendix

Table 17: Field of study odds ratios for S2 link click on PF (95% CI).

LLM Assessment	$n$	Return Rate	95% CI
Negative	1,275	54.7%	[52.0%, 57.5%]
Positive	8,725	62.3%	[61.3%, 63.3%]

Appendix

Table 18: User return rate by LLM quality assessment of SQA reports. Users who received reports assessed as high-quality by an LLM are significantly more likely to return (

p<0.001

r=0.052

LLM Assessment	$n$	Click Rate	95% CI
Negative	1,373	3.9%	[3.0%, 5.0%]
Positive	8,631	5.5%	[5.1%, 6.0%]

Appendix

Table 19: S2 link click rate by LLM quality assessment of SQA reports. Users who received reports assessed as high-quality by an LLM are more likely to click on citation links (

p=0.01

r=0.026

	LLM Assessment
User Feedback	Positive	Negative
Thumbs Up	86	14
Thumbs Down	22	16

Appendix

Table 20: Confusion matrix comparing user feedback (thumbs up/down) with LLM quality assessment. The LLM achieves 73.9% accuracy, with higher agreement on positive assessments (F1 = 0.83) than negative ones (F1 = 0.47), reflecting the challenge of identifying subtle quality issues.

Intent	Description	Examples
Broad Topic Exploration	The user wants a general overview, literature review, or summary of a broad academic field or topic without highly specific constraints.	“Conduct a literature review on the use of deep learning in focussed ultrasound stimulation” “E-commerce” “Theories of internalization”
Specific Factual Retrieval	The user is asking for a precise, factual piece of information, a specific data point, a statistic, or a historical detail. This intent is characterized by its specificity and aims for a direct, verifiable answer.	“what is the best wavelength to check the hemolysis in plasma samples?” “What was the first study examine sexual double standards? When was the term coined?” “find user statistics for UniProt”
Concept Definition and Explanation	The user seeks to understand the meaning, core principles, and fundamental aspects of a specific academic theory, concept, model, or term.	“What are reasoning LLMs?” “overview of the Scholarly Primitive Theory by John unsworth” “What is ‘Administration’ in Public Financial Administration”
Comparative Analysis	The user wants to understand the similarities, differences, advantages, and disadvantages between two or more concepts, theories, models, or methods.	“employer brand vs employer branding” “What are the differences and similarity between information foraging theory and exploratory search?” “Compare the concept of world order between the theory of realism, liberalism, constructivism…”
Methodological and Procedural Guidance	The user is looking for instructions, protocols, frameworks, or best practices for carrying out a specific research task, experiment, analysis, or procedure.	“what can I do to parse effectively PDFs for RAG?” “how to extract dna from blood for sepsis molecular diagnosis” “What are the best research questions related to the following research Gap?…”

Appendix

Table 21: Complete ScholarQA query intent taxonomy (part 1 of 3).

Intent	Description	Examples
Tool and Resource Discovery	The user is searching for specific academic resources such as datasets, software tools, questionnaires, evaluation benchmarks, or code repositories relevant to their research.	“Question answering datasets” “find the deepfake dataset which contains some real-world perturbations.” “What is the best questionnaire to measure mathematics anxiety?”
Research Gap and Limitation Analysis	The user aims to identify the limitations, unsolved problems, or unexplored areas (gaps) within a specific field of research, often to justify a new study or find a novel research direction.	“Systematic reviews on AWE tools in EFL argumentative writing. What is the gap?” “Limitations: What limitations are present in the theoretical literature on spatial poverty?” “Review the SOTA and research gaps of organic neuromorphic computing in AI…”
Causal and Relational Inquiry	The user wants to understand the relationship, impact, effect, or influence of one variable or concept on another. These queries often explore cause-and-effect connections or correlations.	“Does sleep consistency benefit overall health” “how do CEO power influence sustainability performance and financial performance of companies?” “drivers of public support for cost-intensive policies”
Focused Academic Synthesis	The user requests a structured and constrained synthesis of academic literature, such as a systematic review, a meta-analysis, or a review limited to specific journals, timeframes, or methodologies.	“Future of the Labor Market and Higher Education ‘Only use articles from Q1 and Q2 journals.”’ “Write a systematic review on soybean production challenges and opportunities in Ethiopia” “Generate a comprehensive, citation-rich literature review (2014–2025)…”
Academic Document Drafting	The user explicitly asks for the generation of a specific academic document or a section of one, such as a proposal, a chapter, an abstract, or a detailed review, often providing a structured template or detailed instructions.	“using APA 7 style with in text citation, write a literature review of about 2000 words…” “make me a study justification for Impacted canines and its association with dental anomalies…” “write a comprehensive theoretical framework for attention and sustained attention”

Appendix

Table 22: Complete ScholarQA query intent taxonomy (part 2 of 3).

Intent	Description	Examples
Ideation	The user presents a complex problem, a scenario, or a research objective and asks for potential solutions, innovative ideas, hypotheses, or new research directions.	“I want to do ‘quantization in MoE model inference’, find some related papers and recommend some ideas on this” “how can the anthropomorphization of artificial intelligence influence teaching practices…” “Has any research been done to more fully articulate the collaboration/cooperation required…”
Application Inquiry	User seeks research focused on solving a specific, practical problem or demonstrating the application of a theory or technology in a real-world context.	“The Effect of a Program Based on Using Zoom Application on Developing Level Two Students’ Business Vocabulary…” “Golf shot/swing analysis and feedback for self training” “research paper on benefit of construction machineries application”
Data Interpretation and Analysis Support	The user needs help with interpreting or analyzing specific data, results, or observations. This can include understanding statistical outputs, analyzing images/spectra, or making sense of experimental findings.	“analyze UV-Visible peak of HDPE at 281 nm” “What is the clinical and biological implication and meaning of the blood and nail Selenium levels…” “analyzed uv visible spectrum of Fe3O4 nanopartticles at 262 nm”
Content Generation and Expansion	User provides a piece of existing text (such as an outline, abstract, or draft) and explicitly requests for it to be expanded, rewritten, or to have components like references generated for it.	“My research topic is ‘The Effects of Virtual Reality Environments on Adult English Language Acquisition.’ Personalize the following questions…” “melhore essa analise A Figura exibe dois picos expressivos…” “The method of chemical coprecipitation allows to obtain barium hexaferrite… select literary sources…”
Complex Cross-Paper Synthesis	The user requires complex reasoning that involves synthesizing information across multiple research papers to understand relationships, trace evolution of concepts, identify emergent patterns, or connect insights from different disciplines.	“How has the conditional Pesin entropy formula evolved in the literature on coupled dynamical systems…” “Has any research been done to more fully articulate the collaboration/cooperation required…” “topological structure of knowledge networks or knowledge maps…”
Citation & Evidence Finding	User provides a specific claim, statement, or fact and requests academic sources that can be used as a citation to support it.	“Please search for literature to prove that BAK is mainly distributed on the outer membrane of mitochondria” “it was shown that hydroxyl groups… can form intramolecular hydrogen bonds… find me a citation for that” “Media video dan televisi adalah media pembelajaran… carikan sumbernya”

Appendix

Table 23: Complete ScholarQA query intent taxonomy (part 3 of 3). Taxonomy was built through iterative human review and LLM-assisted analysis using Gemini-2.5-pro on a sample of 1000 queries.

Intent	Description	Examples
Broad Topic Exploration	The user wants a general overview, literature review, or summary of a broad academic field or topic without highly specific constraints.	“Do a literature review on artificial intelligence in management accounting” “papers about the technological advances, debates and challenges in humanitarian aid” “Show some exploratory qualitative studies on ELT”
Specific Paper Retrieval	User attempts to locate a single, known academic paper, often using a DOI, a partial citation, or specific author and title information.	“10.1097/PRS.0000000000008882” “To improve the adaptability of control systems… Amiri et al…. find this paper” “Smith J, et al. The regulation of sperm motility… [citation]”
Complex Cross-Paper Synthesis	The user requires complex reasoning that involves synthesizing information across multiple research papers to understand relationships, trace evolution of concepts, identify emergent patterns, or connect insights from different disciplines.	“How has the conditional Pesin entropy formula evolved in the literature on coupled dynamical systems…” “Has any research been done to more fully articulate the collaboration/cooperation required…” “topological structure of knowledge networks…”
Citation & Evidence Finding	User provides a specific claim, statement, or fact and requests academic sources that can be used as a citation to support it.	“Please search for literature to prove that BAK is mainly distributed on the outer membrane of mitochondria” “it was shown that hydroxyl groups… find me a citation for that” “Media video dan televisi… carikan sumbernya”
Methodological and Procedural Guidance	The user is looking for instructions, protocols, frameworks, or best practices for carrying out a specific research task, experiment, analysis, or procedure.	“How to screen the downstream pathways of ERK” “How is alignment consistency ensured from part identification to measurement feedback…” “what is the importance of covering many different spatial configurations…”

Appendix

Table 24: Complete Paper Finder query intent taxonomy (part 1 of 3).

Intent	Description	Examples
Concept Definition and Explanation	The user seeks to understand the meaning, core principles, and fundamental aspects of a specific academic theory, concept, model, or term.	“ppf” “Definition of climate” “What is artificial intelligence?”
Comparative Analysis	The user wants to understand the similarities, differences, advantages, and disadvantages between two or more concepts, theories, models, or methods.	“find paper that comparing the American native or indigenous or village chicken egg and commercial chicken egg” “Copare the papers of learning analytics through UTAUT and TAM” “Comparison of the efficiency of in-vitro digestibility to in-vivo digestibility”
Tool and Resource Discovery	The user is searching for specific academic resources such as datasets, software tools, questionnaires, evaluation benchmarks, or code repositories relevant to their research.	“Papers introducing a dataset of an unscripted dialogue between 2 speakers…” “i need Research papers and open datasets on sports injuries… sensor data such as fitbit…” “Document Processing LLM Benchmarks”
Causal and Relational Inquiry	The user wants to understand the relationship, impact, effect, or influence of one variable or concept on another. These queries often explore cause-and-effect connections or correlations.	“How does fear of missing out influence the effect of mindfulness on academic procrastination…” “The Impact of Corporate Governance Practices on Financial Inclusion…” “what are sex differences in psychiatric comorbidities among individuals with migraine?”
Application Inquiry	User seeks research focused on solving a specific, practical problem or demonstrating the application of a theory or technology in a real-world context.	“The Effect of a Program Based on Using Zoom Application on Developing Level Two Students’ Business Vocabulary…” “Golf shot/swing analysis and feedback for self training” “research paper on benefit of construction machineries application”

Appendix

Table 25: Complete Paper Finder query intent taxonomy (part 2 of 3).

Intent	Description	Examples
Focused Academic Synthesis	The user requests a structured and constrained synthesis of academic literature, such as a systematic review, a meta-analysis, or a review limited to specific journals, timeframes, or methodologies.	“emerging trends in communications sectors that Ofcom deals with” “recent advances in nuclear fusion power” “Must read papers for LLM”
Content Generation and Expansion	User provides a piece of existing text (such as an outline, abstract, or draft) and explicitly requests for it to be expanded, rewritten, or to have components like references generated for it.	“My research topic is ‘The Effects of Virtual Reality Environments on Adult English Language Acquisition.’ Personalize the following questions…” “melhore essa analise…” “The method of chemical coprecipitation… select literary sources…”
Data Interpretation and Analysis Support	The user needs help with interpreting or analyzing specific data, results, or observations. This can include understanding statistical outputs, analyzing images/spectra, or making sense of experimental findings.	“analyze UV-Visible peak of HDPE at 281 nm” “What is the clinical and biological implication and meaning of the blood and nail Selenium levels…” “analyzed uv visible spectrum of Fe3O4 nanopartticles at 262 nm”
Ideation	The user presents a complex problem, a scenario, or a research objective and asks for potential solutions, innovative ideas, hypotheses, or new research directions.	“I want to do ‘quantization in MoE model inference’, find some related papers and recommend some ideas on this” “how can the anthropomorphization of artificial intelligence influence teaching practices…” “Has any research been done to more fully articulate the collaboration/cooperation required…”
Research Gap and Limitation Analysis	The user aims to identify the limitations, unsolved problems, or unexplored areas (gaps) within a specific field of research, often to justify a new study or find a novel research direction.	“Systematic reviews on AWE tools in EFL argumentative writing. What is the gap?” “Limitations: What limitations are present in the theoretical literature on spatial poverty?” “Review the SOTA and research gaps of organic neuromorphic computing in AI…”
Specific Factual Retrieval	The user is asking for a precise, factual piece of information, a specific data point, a statistic, or a historical detail. This intent is characterized by its specificity and aims for a direct, verifiable answer.	“what is the best wavelength to check the hemolysis in plasma samples?” “What was the first study examine sexual double standards? When was the term coined?” “find user statistics for UniProt”

Appendix

Table 26: Complete Paper Finder query intent taxonomy (part 3 of 3). Taxonomy was built through iterative human review and LLM-assisted analysis using Gemini-2.5-pro on a sample of 1000 queries.

Category	Description	Examples
Methodology-Specific Criteria	Queries where the user requires specific methods, approaches, analytical techniques, or experimental designs to be present in the search results. This can include demands for computational models, experimental paradigms, or meta-analyses.	“Fe dopped ZnO and its composite with zeolite, explain crystallite size by Scherrer and Williamson-Hall models and compare” “meta analysis on behavioral epigenetics dogs and cats…” “compare las membranas poliméricas de inclusión respecto a cada uno de los métodos convencionales…”
Publication Type/Quality Filters	Queries that require certain types of literature based on format, peer review status, or journal ranking, such as papers, theses, reviews, or guidelines, with or without requirements for high-impact or reputable sources.	“Review papers on SOTA and research gaps in manifold learning in AI.” “provide several examples from original research articles published post-2020, excluding review articles” “find articles suggesting a research gap on the influence of OFW parentage…”
Temporal Constraints	Queries that include requirements related to the publication date or period of the information (e.g., recent literature, post-2020, historical periods). This includes searching within specific timeframes or for recent advances.	“Find policies and events in European countries that demonstrate ‘strategic autonomy’ between 2016 and 2025.” “provide several examples from original research articles published post-2020…” “Latest research on social loneliness and perfectionism”
Metadata-Based Criteria	Queries specifying attributes of the publications or their creators, such as language, authorship, institutional affiliation, location/country, or publication venue (journal, conference, etc.).	“in welchen Studien wurde das R paket treeclim verwendet und was wurde damit gemacht?” “Shahinda Rezk” “search for papers published in NeurIPS about reinforcement learning”
Citation and Impact-Based Criteria	Queries that specify requirements related to citation counts, impact factors, highly-cited papers, or seminal works in a field. This can include searching for influential or foundational studies.	“find the most highly cited papers on deep learning optimization” “seminal works on contract-net protocols” “SOTA metric on the canonical benchmark(s) and the delta or change vs. prior best benchmark score”
Data/Resource Availability Criteria	Queries that ask for studies or papers with available datasets, code, experimental materials, or other resources for reproducibility or re-use. This category includes explicit requests for supplementary information or links to data/code.	“the code or data availability weblink or vendor for the method” “I want TB expression system papers with downloadable protocols” “review papers listing open datasets for computational linguistics”

Appendix

Table 27: Complete ScholarQA search criteria taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.

Category	Description	Examples
Methodology-Specific Criteria	Queries where the user requires specific methods, approaches, analytical techniques, or experimental designs to be present in the search results. This can include demands for computational models, experimental paradigms, or meta-analyses.	“papers which use Nobel laureates’ publication data to study interdisciplinary research” “Equilibrium analysis enables the assessment of whether observed outcomes reflect a stable strategic configuration…” “Find systematic reviews and/or clinical practice guidelines published in the last 10 years…”
Temporal Constraints	Queries that include requirements related to the publication date or period of the information (e.g., recent literature, post-2020, historical periods). This includes searching within specific timeframes or for recent advances.	“Find systematic reviews and/or clinical practice guidelines published in the last 10 years (2014–2025)…” “papers that have been published between 2023 and 2025” “papers about recent advances in communications sectors that Ofcom deals with”
Metadata-Based Criteria	Queries specifying attributes of the publications or their creators, such as language, authorship, institutional affiliation, location/country, or publication venue (journal, conference, etc.).	“Language: English or Portuguese.” “jurnal tentang Teori stimulus respons Neal E.Miller & John Dollard” “In Sri Lanka, when a government employee is promoted…”
Publication Type/Quality Filters	Queries that require certain types of literature based on format, peer review status, or journal ranking, such as papers, theses, reviews, or guidelines, with or without requirements for high-impact or reputable sources.	“Find systematic reviews and/or clinical practice guidelines published in the last 10 years…” “any paper or thesis about nostalgia and self construal” “por favor me das una tesis de maestría de control constitucional”
Citation and Impact-Based Criteria	Queries that specify requirements related to citation counts, impact factors, highly-cited papers, or seminal works in a field. This can include searching for influential or foundational studies.	“papers which use Nobel laureates’ publication data to study interdisciplinary research” “Find papers that refer to the relationships between degradation ability of PROTAC…” “Find papers that explore mechanisms that stimulate agent cooperation in MARL.”
Data/Resource Availability Criteria	Queries that ask for studies or papers with available datasets, code, experimental materials, or other resources for reproducibility or re-use. This category includes explicit requests for supplementary information or links to data/code.	“the code or data availability weblink or vendor for the method” “I want TB expression system papers with downloadable protocols” “review papers listing open datasets for computational linguistics”

Appendix

Table 28: Complete Paper Finder search criteria taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.

Category	Description	Examples
Keyword-style Query	Short, often fragmentary, queries resembling search engine keywords or subject headings. No verbs or complete sentences; typically a list of nouns or concepts separated by spaces or simple punctuation.	“sludge production MBBR” “obsessive-compulsive personality” “hydrogen production methods”
Natural Language Question	Fully formed, grammatically complete questions using natural language.	“What is the role of agile human resource management in strategic entrepreneurship? The moderating role of corporate culture…” “How can more effective research on Neuromarketing be conducted?” “What are the most common ways (recently) to correct ocr errors”
Explicit Instruction/Imperative	Direct commands or requests instructing the system to perform an action (e.g., ‘find…’, ‘review…’, ‘compare…’). May be simple or include detailed steps, but typically begins with an imperative verb.	“Review papers on SOTA and research gaps in manifold learning in AI.” “Find policies and events in European countries that demonstrate ‘strategic autonomy’ between 2016 and 2025.” “Compare las membranas poliméricas de inclusión respecto a cada uno de los métodos convencionales…”
Complex Contextual Narrative	Lengthy, detailed queries that provide substantial background context, motivations, definitions, or examples before asking a main question or issuing a command. These queries resemble a mini-narrative, sometimes including an abstract, data, citations, or technical context.	“In this experiment, various culture systems for the GF677 rootstock were compared… Is there an explanation in the scientific literature for why leaf number and leaf area are greater…” “I have four papers and I want to combine them all to be a doctoral dissertation…” “The performance of T-junction micromixers, as analysed in CFD models…”
Multi-part/Multi-step Query	Queries composed of multiple distinct sub-questions, tasks, or steps, often divided by letters, numbers, or separate sentences. The sub-queries are related but require separate or structured responses.	“Review papers on SOTA and research gaps in manifold learning in AI. For each of the top 9 methods extract: a) Key architectural or algorithmic ideas… b) Reported SOTA metric… c) the code or data availability…” “Describe what this cognitive ability encompasses… Please use academic papers… for each part of your response.” “Find me the research papers about interictal epileptiform discharges (IEDs)… Does STD in excitatory neurons…”
Boolean or Logical Operators	Queries using explicit logical or Boolean operators or patterns (AND, OR, NOT, +, -, parentheses, slashes, etc.) to combine multiple search terms, constrain results, or designate alternatives. Sometimes appears as ‘Topic A/Topic B’, ‘x, y AND z’.	“adulteration detection in milk AND spectroscopy” “machine learning OR deep learning applications in medical imaging” “neural network NOT convolutional”

Appendix

Table 29: Complete ScholarQA query phrasing taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.

Category	Description	Examples
Keyword-style Query	Short, often fragmentary, queries resembling search engine keywords or subject headings. No verbs or complete sentences; typically a list of nouns or concepts separated by spaces or simple punctuation.	“algorithmic trading” “ppf” “bayesian optimization applications”
Natural Language Question	Fully formed, grammatically complete questions using natural language.	“How does fear of missing out influence the effect of mindfulness on academic procrastination among students?” “Do a literature review on artificial intelligence in management accounting” “What cognitive strategies and techniques can improve focus and concentration for artists…”
Explicit Instruction/Imperative	Direct commands or requests instructing the system to perform an action (e.g., ‘find…’, ‘review…’, ‘compare…’). May be simple or include detailed steps, but typically begins with an imperative verb.	“Find papers that refer to the relationships between degradation ability of PROTAC and potency or affinity…” “Give papers on finetuning an llm for writing research papers” “Find systematic reviews and/or clinical practice guidelines published in the last 10 years…”
Complex Contextual Narrative	Lengthy, detailed queries that provide substantial background context, motivations, definitions, or examples before asking a main question or issuing a command. These queries resemble a mini-narrative, sometimes including an abstract, data, citations, or technical context.	“As restored ecosystems evolve towards the composition and structure characteristic of native vegetation…” “Equilibrium analysis enables the assessment of whether observed outcomes reflect a stable strategic configuration…” “Papers about innovation and hurdles within industries that prevent it…”
Boolean or Logical Operators	Queries using explicit logical or Boolean operators or patterns (AND, OR, NOT, +, -, parentheses, slashes, etc.) to combine multiple search terms, constrain results, or designate alternatives. Sometimes appears as ‘Topic A/Topic B’, ‘x, y AND z’.	“Find systematic reviews and/or clinical practice guidelines published in the last 10 years (2014–2025)…” “Papers introducing a dataset of an unscripted dialogue between 2 speakers…” “Stretching exercises are commonly integrated into physical education programs…”
Multi-part/Multi-step Query	Queries composed of multiple distinct sub-questions, tasks, or steps, often divided by letters, numbers, or separate sentences. The sub-queries are related but require separate or structured responses.	“Review papers on SOTA and research gaps in manifold learning in AI. For each of the top 9 methods extract: a) Key architectural or algorithmic ideas… b) Reported SOTA metric… c) the code or data availability…” “Describe what this cognitive ability encompasses… Please use academic papers… for each part of your response.” “Find me the research papers about interictal epileptiform discharges (IEDs)… Does STD in excitatory neurons…”

Appendix

Table 30: Complete Paper Finder query phrasing taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.

Field of Study	Field of Study
Agricultural Sciences	Law and Legal Studies
Anthropology	Linguistics
Arts and Design	Literature
Biomedical Sciences	Mathematics
Biology	Mechanical Engineering
Business and Management	Philosophy

Appendix

Table 31: The 28 fields of study used for query classification.

AI-Assisted Scientific Research

AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.

See all 253 papers

Oct 5, 2026

Learning Scientific Exploration from Human Research Decision Trajectories

A key challenge in building AI systems for scientific research is enabling

\textit{scientific exploration}

: the systematic process of investigating unknown phenomena or ideas to gain new knowledge through sequences of research decisions and actions. Yet this process is largely missing from existing scientific corpora; for example, research papers primarily record final outcomes rather than the trajectories that produced them. In this work, we introduce

\textbf{ResearchTrails}

, a dataset of

\textbf{human research trajectories constructed from Git repositories}

, where

\textbf{commit histories}

serve as proxies for research exploration. We develop an automated and scalable pipeline that extracts structured research trajectories from repository commits, capturing successive changes to methods, experiments, and ablations. We characterize the resulting dataset and show that these trajectories contain meaningful signals about intermediate research decisions beyond what final papers reveal. We further demonstrate utilities of ResearchTrails in multiple use cases, including retrieving human research experience as external skills at test time and training models on research trajectories to improve generalization to new research decisions. Our results suggest a path toward AI systems that learn not only from the products of science, but from the evolving process of discovery itself.

Xuchen Gong, Shane Gu, Haokun Liu +3

University of Chicago

AI-Assisted Scientific Research

Ask PDF

May 7, 2026

Intentmaking and Sensemaking: Human Interaction with AI-Guided Mathematical Discovery

Artificial intelligence offers powerful new tools for scientific discovery, but the interaction paradigms required to effectively harness these systems remain underexplored. In this paper, we present findings from a formative user study with 11 expert mathematicians who used AlphaEvolve, an evolutionary coding agent, to tackle advanced problems in their fields of expertise. We identify and characterize a distinct workflow we term intentmaking, the iterative process of discovering, defining, and refining one's experimental goals through active system interaction. We frame this as a natural extension to sensemaking, the cognitive process of building an understanding of complex or novel data. We suggest that users enter a cycle of intentmaking (defining and updating their experiment) and sensemaking (interpreting the results) which repeats many times during the course of an investigation. Our documentation of these themes suggests an approach to designing AI tools for scientific discovery that goes beyond the existing question/answer model of many current systems, treating them as collaborative instruments rather than opaque black-box assistants.

Alex Bäuerle, Adam Connors, Alexander Novikov +5

Google DeepMind

AI for Science AI Coding Agents Human-AI Interaction

Ask PDF

Oct 23, 2026

PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers

Scientific agents collaborate with researchers whose expertise and preferences differ, but benchmarks usually hold the collaborator fixed. We introduce PersonaScienceBench: 200 tasks from five benchmarks across four scientific domains. Simulated researchers provide context and feedback according to persona profiles, while task objectives and grading remain fixed. Agents encounter researcher preferences only through conversation. Across eighteen modelconfigurations, task performance, fact recovery, and self-repetition reveal different strengths. Detailed GPT 6 Astra analyses show distinct agent responses across researcher profiles; interaction traces illustrate targeted clar ification and workflow revision. These behaviors can differ even when final scores match. PersonaScienceBench therefore evaluates how agents collaborate with diverse researchers alongside whether they complete the task.

Siyang Wu, Weizhi Zhang, Chen Yang +29

University of Chicago · University of Illinois Chicago · Harvard University +15

Scientific Agent Benchmarks LLM Agent Evaluation Agent Evaluation AI Agent Benchmarks AI Agents for Scientific Discovery Human-AI Collaboration

Ask PDF

Abstract

AI-Assisted Scientific Research

Explore similar work

Learning Scientific Exploration from Human Research Decision Trajectories

Intentmaking and Sensemaking: Human Interaction with AI-Guided Mathematical Discovery

PersonaScienceBench: Evaluating Scientific Agents with Diverse Simulated Researchers