cs.HCFeb 26, 2026
SaveUnderstanding Usage and Engagement in AI-Powered Scientific Research Tools: The Asta Interaction Dataset
Organizations: Allen Institute for AI
Abstract
AI-powered scientific research tools are rapidly being integrated into research workflows, yet the field lacks a clear lens into how researchers use these systems in real-world settings. We present and analyze the Asta Interaction Dataset, a large-scale resource comprising over 200,000 user queries and interaction logs from two deployed tools (a literature discovery interface and a scientific question-answering interface) within an LLM-powered retrieval-augmented generation platform. Using this dataset, we characterize query patterns, engagement behaviors, and how usage evolves with experience. We find that users submit longer and more complex queries than in traditional search, and treat the system as a collaborative research partner, delegating tasks such as drafting content and identifying research gaps. Users treat generated responses as persistent artifacts, revisiting and navigating among outputs and cited evidence in non-linear ways. With experience, users issue more targeted queries and engage more deeply with supporting citations, although keyword-style queries persist even among experienced users. We release the anonymized dataset and analysis with a new query intent taxonomy to inform future designs of real-world AI research assistants and to support realistic evaluation.
Figures & tables
Figure 1: Screenshots of the two Asta interfaces used in our study ( Singh et al., 2025 ; Allen Institute for Artificial Intelligence, 2025 ) .
| Intent | Example Query |
| Broad Topic Exploration | GLP-1 and diabetes |
| Specific Factual Retrieval | What are the four core concepts of Rotter’s theory? |
| Concept Def. & Exploration | Summarize the concept of “Technoimagination” […] by Vilém Flusser |
| Comparative Analysis | Describe the trade-offs between HBr and Cl2 plasma gases for reactive ion etching of polysilicon |
| Causal & Relational Inquiry | Relation between nighttime digital device use and sleep quality […] |
| Method. & Proced. Guidelines | How often should I collect mosquitoes for dengue surveillance? |
Table 1: Representative examples of query intents illustrating of user information needs.
| Phrasing Style | Example Query |
| Keyword-style | piracetam efficacy |
| Natural Language Q. | How are emerging digital technologies reshaping sustainable development outcomes? |
| Explicit Instruction | Write an essay as a doctor of literary discourse analysis[…] |
| Complex Context Narrative | When training large language models, it’s essential to develop benchmarks that assess true problem-solving ability, not just factual recall. [detailed discussion about knowing Berlin’s train stations vs. planning a novel journey] [C]onduct a literature review on efforts to curate evaluations that explicitly test synthesis rather than memorization. |
| Multi-part Query | 9. Identify and summarise key international instruments (e.g., UN Charter, Budapest Convention, UNSC Resolutions). Discuss their relevance and application […] |
| Boolean/Logical Ops | (“Family Centered” OR “Family Centred”) AND (challeng* OR implantat*) |
Table 2: Representative examples of query phrasing styles observed across PF and SQA.
| Search Constraint | Example Query |
| Methodology-Specific | federated learning where the authors had coined the new terms for defining client types like homogenous clients, selfish clients… |
| Publication Type/Quality | Please find scientific references only from highly reputable and scopus-indexed international journals… |
| Temporal Constraints | Pharmacological Activities of Ganoderma applanatum. Use journals from 2020-2025 |
| Metadata-Based | Papers from SIG CHI that have use cases with an “adaptive task challenge” |
| Cit./Impact-Based | What is the most cited paper of Orit Hazzan ? |
| Data/Resource Availability | search for documents that provide data of the amount of comic books are purchased among different classes in the United States. |
Table 3: Representative examples of search constraints (criteria) specified in user queries across PF and SQA.
Figure 3: Query phrasing styles and intents (% of queries). Keyword queries dominate the distribution. Asta users initiate more natural language queries than S2 users. Most queries are broad exploration or concept explanation; S2 is skewed towards broad and specific paper retrieval queries. See Table 6 for complete intent distribution, and Table 7 for criteria distribution.
| Query Label | single-query | inexperienced | experienced |
| Broad Topic Exploration | |||
| Causal and Relational Inquiry | |||
| Citation & Evidence Finding | |||
| Methodology-Specific Criteria |
Table 5: Query label by experience stage, tracking the same user population over time (percentages with 95 % confidence intervals).
Appendix figures & tables36 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 4: Section expansion distribution (on a log scale) showing which sections users expand first in SQA responses. Section index 1 has the largest number of expansions. Users tend to start on section 0 or 1 and end on a section between 2–4.
Figure 5: Section transition heatmap showing in-order vs out-of-order reading patterns in SQA. Sequential expansion is the dominant behavior, but there is notable backward traversal behavior as well as return to 0 (the introduction) behavior. Users also close and then reopen a section (presumably to read the TL;DR which is only visible when a section is collapsed).
| Intent | % | Example |
| SQA | ||
| Broad Topic Expl. | 51.6 | What are the types of approaches to employer branding? |
| Concept Def. & Expl. | 28.2 | Digital Banking BI Maturity Models |
| Causal & Rel. Inquiry | 19.1 | why evaluation as a construct of metacognition is very important for writing competence to emerge? |
| Specific Factual Retr. | 12.6 | what do say paper of sharma2019 and kamga 2016 about resistace in chemotherapy in AML IN NOTCH pathway |
| Method. & Proc. Guid. | 9.1 | design non-stationnary wavelets trough […] bezout’s equation |
Table 6: Query intent distribution (% of queries) with random examples.
| Asta | S2 | |
| Methodology | 42 | 29 |
| Pub. Quality | 11 | 3 |
| Metadata | 9 | 12 |
| Temporal | 5 | 2 |
| Data Avail. | 1 | 0.4 |
| Cit. Impact | 0.8 | 0.1 |
Table 7: Query criteria distribution (% of queries). Methodology-related criteria are most common on Asta.
Figure 6: Distribution of coarse-grained fields of study across PF, SQA, and S2.
Figure 7: Reading order shown through section flow Sankey diagram, illustrating user navigation patterns through SQA response sections.
| Characteristic | Description |
| Keyword-style Query (phrasing style) | Short, often fragmentary, queries resembling search engine keywords or subject headings. No verbs or complete sentences; typically a list of nouns or concepts separated by spaces or simple punctuation. |
| Methodology-Specific Criteria (query criteria) | Queries where the user requires specific methods, approaches, analytical techniques, or experimental designs to be present in the search results. This can include demands for computational models, experimental paradigms, or meta-analyses. |
Table 8: Common query characteristics observed in Asta.
Figure 8: The fraction of queries from each field of study that have a given intent on Asta. The distribution reflects the research tasks common in the given fields.
Figure 9: Action engagement trends by query index showing how users perform different actions as they gain experience with the system. Shaded regions represent 95 % confidence intervals. PF reports tend to receive less click engagement over time compared to SQA reports which grow in engagement, the reason likely being that PF results can be consumed passively without interacting with the web content at all whereas most of the content generated by SQA can only be accessed after clicking on the web page.
Figure 10: Distribution of text feedback categories reflecting user expectations and experiences. The comprehensiveness of the response is the most common complaint ( Lack of Depth and Citation & Referencing Issues ).
| Category | Example Feedback |
| Response Quality | develop more the introduction and add more citations (references) |
| Response Quality | this is leaving out some key details from Magalhaes et al 2013 |
| Response Quality | Out of date |
| Query Refinement | Its very good but I need to add a historical component |
| Feature Requests | would be a great place to a diagram |
| Feature Requests | amazing! Coudl you create a PDF |
Table 9: Example user feedback by category. Some users seem to expect the feedback submission to provide a multiturn chat experience.
| Pattern | Tool | Example Query |
| Template Filling | PF | fill this tabel with 10 jurnal bellow:… [table template and citations] |
| Template Filling | SQA | for sacubitril find all: "IUPAC Name: CAS Number: Molecular Formula:…" [15+ fields] |
| Prompting | SQA | You are an expert research assistant specializing in computational geosciences and machine learning. |
| Prompting | PF | Find papers…The model must be capable of… |
| Persona Adoption | SQA | Think of yourself as experienced professor…Please write me a phd proposal…devour Turnitin detection bots |
| Collaborative Writing | SQA | I’m working on my paper… [LaTeX section] add papers from TSE, TOSEM, ICSE |
Table 10: Examples of interesting usage patterns showing how users probe the capabilities of LLM-powered research tools. Users go beyond basic IR tasks and expect Asta to work as a collaborative research assistant.
| Metric | SQA | PF |
| User Prevalence | ||
| Users with duplicate queries | 18.8 % | 14.8 % |
| Users who revisited reports | 50.5 % | 42.1 % |
| Temporal Patterns | ||
| Median time between duplicates | 15.9 min | 5.8 min |
| Median time between revisits | 3.8 hours | 5.9 hours |
Table 11: Comparison of duplicate query submission versus report revisiting behavior.
| Tool | Gap | Query Pair |
| SQA | 4.0 min | First: History of experiential learning of science |
| Second: History of experiential learning of science | ||
| SQA | 1.4 min | First: postbiotics food |
| Second: postbiotics in food industry | ||
| SQA | 6.4 min | First: [Long query requesting literature review] |
| Second: [Same query + “write this in future tense”] |
Table 12: Representative examples of duplicate queries showing exact duplicates and incremental refinements. Time gaps indicate minutes between submissions.
Figure 11: Fraction of reports on which the given action has been performed across PF and SQA, showing how users engage with different features. Section expansion actions are the most common, likely because users must click to expand the section text in the SQA response. Note that link clicks are much more common than thumbs up/down feedback; below we also show that link clicks are also strongly associated with use satisfaction.
Figure 12: Survival analysis showing the Kaplan-Meier curve associated with various actions that users performed during their first visit to Asta. Users who did not perform any action take longer to return to Asta than those who do.
Figure 13: Action return rates showing the probability of users returning after different action types. Actions of first time users are likely more affected by novelty effect.
| Query Type | Example Query |
| Citation & Evidence Finding | Can you find for me explicitly mentioned in a paper when an anti-reflection coating was used to improve the temporal contrast of a high power laser due to it having to be at s polarization. |
| Broad Topic Exploration | find me articles on AI implementation in SMEs. |
| Temporal Constraints | What are the drawbacks of Intelligent tutoring systems from before they used LLMs |
| Data/Resource Availability | dataset with image and sun azimuth and sun elevation |
| Content Generation and Expansion | Create an article with the title “Material Strength Design on Excavator Arm Model” with references of 40 articles or journals |
Table 13: Example PF queries by type. PF performs well on Citation & Evidence Finding and Broad Topic Exploration queries but struggles with queries having Temporal Constraints, Data/Resource requirements, or Content Generation requests.
| Query Type | Example Query |
| Concept Definition and Explanation | explain what multimodal and multisensor are and the differents betwen them |
| Complex Contextual Narrative | Recent evidence highlights the critical role of growth hormone secretagogue receptor signaling in hippocampal synaptic physiology, mediated through dopamine receptor activity… [describes detailed molecular mechanisms] |
| Citation Format Specification | Use APA style 7 for in-text citations, citing authors’ names, publication dates, and page numbers. Write a theoretical framework of 17,000 words about schooling of rural girls… |
Table 14: Example SQA queries by type. SQA excels at Concept Definition and Complex Contextual Narrative queries but struggles with Citation Format Specification requests.
Figure 14: Estimated coefficients for the linear models predicting clicks and return on PF and SQA. Coefficients are shown with 95% confidence intervals. Only coefficients significant after Benjamini-Hochberg correction are displayed.
| Field of Study | OR |
| Law & Legal Studies | |
| Sociology | |
| Agricultural Sciences | |
| Electrical Engineering | |
| Clinical Medicine | |
| Business & Management |
Table 17: Field of study odds ratios for S2 link click on PF (95% CI).
| LLM Assessment | Return Rate | 95% CI | |
| Negative | 1,275 | 54.7% | [52.0%, 57.5%] |
| Positive | 8,725 | 62.3% | [61.3%, 63.3%] |
Table 18: User return rate by LLM quality assessment of SQA reports. Users who received reports assessed as high-quality by an LLM are significantly more likely to return ( , ).
| LLM Assessment | Click Rate | 95% CI | |
| Negative | 1,373 | 3.9% | [3.0%, 5.0%] |
| Positive | 8,631 | 5.5% | [5.1%, 6.0%] |
Table 19: S2 link click rate by LLM quality assessment of SQA reports. Users who received reports assessed as high-quality by an LLM are more likely to click on citation links ( , ).
| LLM Assessment | ||
| User Feedback | Positive | Negative |
| Thumbs Up | 86 | 14 |
| Thumbs Down | 22 | 16 |
Table 20: Confusion matrix comparing user feedback (thumbs up/down) with LLM quality assessment. The LLM achieves 73.9% accuracy, with higher agreement on positive assessments (F1 = 0.83) than negative ones (F1 = 0.47), reflecting the challenge of identifying subtle quality issues.
| Intent | Description | Examples |
| Broad Topic Exploration | The user wants a general overview, literature review, or summary of a broad academic field or topic without highly specific constraints. | “Conduct a literature review on the use of deep learning in focussed ultrasound stimulation” “E-commerce” “Theories of internalization” |
| Specific Factual Retrieval | The user is asking for a precise, factual piece of information, a specific data point, a statistic, or a historical detail. This intent is characterized by its specificity and aims for a direct, verifiable answer. | “what is the best wavelength to check the hemolysis in plasma samples?” “What was the first study examine sexual double standards? When was the term coined?” “find user statistics for UniProt” |
| Concept Definition and Explanation | The user seeks to understand the meaning, core principles, and fundamental aspects of a specific academic theory, concept, model, or term. | “What are reasoning LLMs?” “overview of the Scholarly Primitive Theory by John unsworth” “What is ‘Administration’ in Public Financial Administration” |
| Comparative Analysis | The user wants to understand the similarities, differences, advantages, and disadvantages between two or more concepts, theories, models, or methods. | “employer brand vs employer branding” “What are the differences and similarity between information foraging theory and exploratory search?” “Compare the concept of world order between the theory of realism, liberalism, constructivism…” |
| Methodological and Procedural Guidance | The user is looking for instructions, protocols, frameworks, or best practices for carrying out a specific research task, experiment, analysis, or procedure. | “what can I do to parse effectively PDFs for RAG?” “how to extract dna from blood for sepsis molecular diagnosis” “What are the best research questions related to the following research Gap?…” |
Table 21: Complete ScholarQA query intent taxonomy (part 1 of 3).
| Intent | Description | Examples |
| Tool and Resource Discovery | The user is searching for specific academic resources such as datasets, software tools, questionnaires, evaluation benchmarks, or code repositories relevant to their research. | “Question answering datasets” “find the deepfake dataset which contains some real-world perturbations.” “What is the best questionnaire to measure mathematics anxiety?” |
| Research Gap and Limitation Analysis | The user aims to identify the limitations, unsolved problems, or unexplored areas (gaps) within a specific field of research, often to justify a new study or find a novel research direction. | “Systematic reviews on AWE tools in EFL argumentative writing. What is the gap?” “Limitations: What limitations are present in the theoretical literature on spatial poverty?” “Review the SOTA and research gaps of organic neuromorphic computing in AI…” |
| Causal and Relational Inquiry | The user wants to understand the relationship, impact, effect, or influence of one variable or concept on another. These queries often explore cause-and-effect connections or correlations. | “Does sleep consistency benefit overall health” “how do CEO power influence sustainability performance and financial performance of companies?” “drivers of public support for cost-intensive policies” |
| Focused Academic Synthesis | The user requests a structured and constrained synthesis of academic literature, such as a systematic review, a meta-analysis, or a review limited to specific journals, timeframes, or methodologies. | “Future of the Labor Market and Higher Education ‘Only use articles from Q1 and Q2 journals.”’ “Write a systematic review on soybean production challenges and opportunities in Ethiopia” “Generate a comprehensive, citation-rich literature review (2014–2025)…” |
| Academic Document Drafting | The user explicitly asks for the generation of a specific academic document or a section of one, such as a proposal, a chapter, an abstract, or a detailed review, often providing a structured template or detailed instructions. | “using APA 7 style with in text citation, write a literature review of about 2000 words…” “make me a study justification for Impacted canines and its association with dental anomalies…” “write a comprehensive theoretical framework for attention and sustained attention” |
Table 22: Complete ScholarQA query intent taxonomy (part 2 of 3).
| Intent | Description | Examples |
| Ideation | The user presents a complex problem, a scenario, or a research objective and asks for potential solutions, innovative ideas, hypotheses, or new research directions. | “I want to do ‘quantization in MoE model inference’, find some related papers and recommend some ideas on this” “how can the anthropomorphization of artificial intelligence influence teaching practices…” “Has any research been done to more fully articulate the collaboration/cooperation required…” |
| Application Inquiry | User seeks research focused on solving a specific, practical problem or demonstrating the application of a theory or technology in a real-world context. | “The Effect of a Program Based on Using Zoom Application on Developing Level Two Students’ Business Vocabulary…” “Golf shot/swing analysis and feedback for self training” “research paper on benefit of construction machineries application” |
| Data Interpretation and Analysis Support | The user needs help with interpreting or analyzing specific data, results, or observations. This can include understanding statistical outputs, analyzing images/spectra, or making sense of experimental findings. | “analyze UV-Visible peak of HDPE at 281 nm” “What is the clinical and biological implication and meaning of the blood and nail Selenium levels…” “analyzed uv visible spectrum of Fe3O4 nanopartticles at 262 nm” |
| Content Generation and Expansion | User provides a piece of existing text (such as an outline, abstract, or draft) and explicitly requests for it to be expanded, rewritten, or to have components like references generated for it. | “My research topic is ‘The Effects of Virtual Reality Environments on Adult English Language Acquisition.’ Personalize the following questions…” “melhore essa analise A Figura exibe dois picos expressivos…” “The method of chemical coprecipitation allows to obtain barium hexaferrite… select literary sources…” |
| Complex Cross-Paper Synthesis | The user requires complex reasoning that involves synthesizing information across multiple research papers to understand relationships, trace evolution of concepts, identify emergent patterns, or connect insights from different disciplines. | “How has the conditional Pesin entropy formula evolved in the literature on coupled dynamical systems…” “Has any research been done to more fully articulate the collaboration/cooperation required…” “topological structure of knowledge networks or knowledge maps…” |
| Citation & Evidence Finding | User provides a specific claim, statement, or fact and requests academic sources that can be used as a citation to support it. | “Please search for literature to prove that BAK is mainly distributed on the outer membrane of mitochondria” “it was shown that hydroxyl groups… can form intramolecular hydrogen bonds… find me a citation for that” “Media video dan televisi adalah media pembelajaran… carikan sumbernya” |
Table 23: Complete ScholarQA query intent taxonomy (part 3 of 3). Taxonomy was built through iterative human review and LLM-assisted analysis using Gemini-2.5-pro on a sample of 1000 queries.
| Intent | Description | Examples |
| Broad Topic Exploration | The user wants a general overview, literature review, or summary of a broad academic field or topic without highly specific constraints. | “Do a literature review on artificial intelligence in management accounting” “papers about the technological advances, debates and challenges in humanitarian aid” “Show some exploratory qualitative studies on ELT” |
| Specific Paper Retrieval | User attempts to locate a single, known academic paper, often using a DOI, a partial citation, or specific author and title information. | “10.1097/PRS.0000000000008882” “To improve the adaptability of control systems… Amiri et al…. find this paper” “Smith J, et al. The regulation of sperm motility… [citation]” |
| Complex Cross-Paper Synthesis | The user requires complex reasoning that involves synthesizing information across multiple research papers to understand relationships, trace evolution of concepts, identify emergent patterns, or connect insights from different disciplines. | “How has the conditional Pesin entropy formula evolved in the literature on coupled dynamical systems…” “Has any research been done to more fully articulate the collaboration/cooperation required…” “topological structure of knowledge networks…” |
| Citation & Evidence Finding | User provides a specific claim, statement, or fact and requests academic sources that can be used as a citation to support it. | “Please search for literature to prove that BAK is mainly distributed on the outer membrane of mitochondria” “it was shown that hydroxyl groups… find me a citation for that” “Media video dan televisi… carikan sumbernya” |
| Methodological and Procedural Guidance | The user is looking for instructions, protocols, frameworks, or best practices for carrying out a specific research task, experiment, analysis, or procedure. | “How to screen the downstream pathways of ERK” “How is alignment consistency ensured from part identification to measurement feedback…” “what is the importance of covering many different spatial configurations…” |
Table 24: Complete Paper Finder query intent taxonomy (part 1 of 3).
| Intent | Description | Examples |
| Concept Definition and Explanation | The user seeks to understand the meaning, core principles, and fundamental aspects of a specific academic theory, concept, model, or term. | “ppf” “Definition of climate” “What is artificial intelligence?” |
| Comparative Analysis | The user wants to understand the similarities, differences, advantages, and disadvantages between two or more concepts, theories, models, or methods. | “find paper that comparing the American native or indigenous or village chicken egg and commercial chicken egg” “Copare the papers of learning analytics through UTAUT and TAM” “Comparison of the efficiency of in-vitro digestibility to in-vivo digestibility” |
| Tool and Resource Discovery | The user is searching for specific academic resources such as datasets, software tools, questionnaires, evaluation benchmarks, or code repositories relevant to their research. | “Papers introducing a dataset of an unscripted dialogue between 2 speakers…” “i need Research papers and open datasets on sports injuries… sensor data such as fitbit…” “Document Processing LLM Benchmarks” |
| Causal and Relational Inquiry | The user wants to understand the relationship, impact, effect, or influence of one variable or concept on another. These queries often explore cause-and-effect connections or correlations. | “How does fear of missing out influence the effect of mindfulness on academic procrastination…” “The Impact of Corporate Governance Practices on Financial Inclusion…” “what are sex differences in psychiatric comorbidities among individuals with migraine?” |
| Application Inquiry | User seeks research focused on solving a specific, practical problem or demonstrating the application of a theory or technology in a real-world context. | “The Effect of a Program Based on Using Zoom Application on Developing Level Two Students’ Business Vocabulary…” “Golf shot/swing analysis and feedback for self training” “research paper on benefit of construction machineries application” |
Table 25: Complete Paper Finder query intent taxonomy (part 2 of 3).
| Intent | Description | Examples |
| Focused Academic Synthesis | The user requests a structured and constrained synthesis of academic literature, such as a systematic review, a meta-analysis, or a review limited to specific journals, timeframes, or methodologies. | “emerging trends in communications sectors that Ofcom deals with” “recent advances in nuclear fusion power” “Must read papers for LLM” |
| Content Generation and Expansion | User provides a piece of existing text (such as an outline, abstract, or draft) and explicitly requests for it to be expanded, rewritten, or to have components like references generated for it. | “My research topic is ‘The Effects of Virtual Reality Environments on Adult English Language Acquisition.’ Personalize the following questions…” “melhore essa analise…” “The method of chemical coprecipitation… select literary sources…” |
| Data Interpretation and Analysis Support | The user needs help with interpreting or analyzing specific data, results, or observations. This can include understanding statistical outputs, analyzing images/spectra, or making sense of experimental findings. | “analyze UV-Visible peak of HDPE at 281 nm” “What is the clinical and biological implication and meaning of the blood and nail Selenium levels…” “analyzed uv visible spectrum of Fe3O4 nanopartticles at 262 nm” |
| Ideation | The user presents a complex problem, a scenario, or a research objective and asks for potential solutions, innovative ideas, hypotheses, or new research directions. | “I want to do ‘quantization in MoE model inference’, find some related papers and recommend some ideas on this” “how can the anthropomorphization of artificial intelligence influence teaching practices…” “Has any research been done to more fully articulate the collaboration/cooperation required…” |
| Research Gap and Limitation Analysis | The user aims to identify the limitations, unsolved problems, or unexplored areas (gaps) within a specific field of research, often to justify a new study or find a novel research direction. | “Systematic reviews on AWE tools in EFL argumentative writing. What is the gap?” “Limitations: What limitations are present in the theoretical literature on spatial poverty?” “Review the SOTA and research gaps of organic neuromorphic computing in AI…” |
| Specific Factual Retrieval | The user is asking for a precise, factual piece of information, a specific data point, a statistic, or a historical detail. This intent is characterized by its specificity and aims for a direct, verifiable answer. | “what is the best wavelength to check the hemolysis in plasma samples?” “What was the first study examine sexual double standards? When was the term coined?” “find user statistics for UniProt” |
Table 26: Complete Paper Finder query intent taxonomy (part 3 of 3). Taxonomy was built through iterative human review and LLM-assisted analysis using Gemini-2.5-pro on a sample of 1000 queries.
| Category | Description | Examples |
| Methodology-Specific Criteria | Queries where the user requires specific methods, approaches, analytical techniques, or experimental designs to be present in the search results. This can include demands for computational models, experimental paradigms, or meta-analyses. | “Fe dopped ZnO and its composite with zeolite, explain crystallite size by Scherrer and Williamson-Hall models and compare” “meta analysis on behavioral epigenetics dogs and cats…” “compare las membranas poliméricas de inclusión respecto a cada uno de los métodos convencionales…” |
| Publication Type/Quality Filters | Queries that require certain types of literature based on format, peer review status, or journal ranking, such as papers, theses, reviews, or guidelines, with or without requirements for high-impact or reputable sources. | “Review papers on SOTA and research gaps in manifold learning in AI.” “provide several examples from original research articles published post-2020, excluding review articles” “find articles suggesting a research gap on the influence of OFW parentage…” |
| Temporal Constraints | Queries that include requirements related to the publication date or period of the information (e.g., recent literature, post-2020, historical periods). This includes searching within specific timeframes or for recent advances. | “Find policies and events in European countries that demonstrate ‘strategic autonomy’ between 2016 and 2025.” “provide several examples from original research articles published post-2020…” “Latest research on social loneliness and perfectionism” |
| Metadata-Based Criteria | Queries specifying attributes of the publications or their creators, such as language, authorship, institutional affiliation, location/country, or publication venue (journal, conference, etc.). | “in welchen Studien wurde das R paket treeclim verwendet und was wurde damit gemacht?” “Shahinda Rezk” “search for papers published in NeurIPS about reinforcement learning” |
| Citation and Impact-Based Criteria | Queries that specify requirements related to citation counts, impact factors, highly-cited papers, or seminal works in a field. This can include searching for influential or foundational studies. | “find the most highly cited papers on deep learning optimization” “seminal works on contract-net protocols” “SOTA metric on the canonical benchmark(s) and the delta or change vs. prior best benchmark score” |
| Data/Resource Availability Criteria | Queries that ask for studies or papers with available datasets, code, experimental materials, or other resources for reproducibility or re-use. This category includes explicit requests for supplementary information or links to data/code. | “the code or data availability weblink or vendor for the method” “I want TB expression system papers with downloadable protocols” “review papers listing open datasets for computational linguistics” |
Table 27: Complete ScholarQA search criteria taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.
| Category | Description | Examples |
| Methodology-Specific Criteria | Queries where the user requires specific methods, approaches, analytical techniques, or experimental designs to be present in the search results. This can include demands for computational models, experimental paradigms, or meta-analyses. | “papers which use Nobel laureates’ publication data to study interdisciplinary research” “Equilibrium analysis enables the assessment of whether observed outcomes reflect a stable strategic configuration…” “Find systematic reviews and/or clinical practice guidelines published in the last 10 years…” |
| Temporal Constraints | Queries that include requirements related to the publication date or period of the information (e.g., recent literature, post-2020, historical periods). This includes searching within specific timeframes or for recent advances. | “Find systematic reviews and/or clinical practice guidelines published in the last 10 years (2014–2025)…” “papers that have been published between 2023 and 2025” “papers about recent advances in communications sectors that Ofcom deals with” |
| Metadata-Based Criteria | Queries specifying attributes of the publications or their creators, such as language, authorship, institutional affiliation, location/country, or publication venue (journal, conference, etc.). | “Language: English or Portuguese.” “jurnal tentang Teori stimulus respons Neal E.Miller & John Dollard” “In Sri Lanka, when a government employee is promoted…” |
| Publication Type/Quality Filters | Queries that require certain types of literature based on format, peer review status, or journal ranking, such as papers, theses, reviews, or guidelines, with or without requirements for high-impact or reputable sources. | “Find systematic reviews and/or clinical practice guidelines published in the last 10 years…” “any paper or thesis about nostalgia and self construal” “por favor me das una tesis de maestría de control constitucional” |
| Citation and Impact-Based Criteria | Queries that specify requirements related to citation counts, impact factors, highly-cited papers, or seminal works in a field. This can include searching for influential or foundational studies. | “papers which use Nobel laureates’ publication data to study interdisciplinary research” “Find papers that refer to the relationships between degradation ability of PROTAC…” “Find papers that explore mechanisms that stimulate agent cooperation in MARL.” |
| Data/Resource Availability Criteria | Queries that ask for studies or papers with available datasets, code, experimental materials, or other resources for reproducibility or re-use. This category includes explicit requests for supplementary information or links to data/code. | “the code or data availability weblink or vendor for the method” “I want TB expression system papers with downloadable protocols” “review papers listing open datasets for computational linguistics” |
Table 28: Complete Paper Finder search criteria taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.
| Category | Description | Examples |
| Keyword-style Query | Short, often fragmentary, queries resembling search engine keywords or subject headings. No verbs or complete sentences; typically a list of nouns or concepts separated by spaces or simple punctuation. | “sludge production MBBR” “obsessive-compulsive personality” “hydrogen production methods” |
| Natural Language Question | Fully formed, grammatically complete questions using natural language. | “What is the role of agile human resource management in strategic entrepreneurship? The moderating role of corporate culture…” “How can more effective research on Neuromarketing be conducted?” “What are the most common ways (recently) to correct ocr errors” |
| Explicit Instruction/Imperative | Direct commands or requests instructing the system to perform an action (e.g., ‘find…’, ‘review…’, ‘compare…’). May be simple or include detailed steps, but typically begins with an imperative verb. | “Review papers on SOTA and research gaps in manifold learning in AI.” “Find policies and events in European countries that demonstrate ‘strategic autonomy’ between 2016 and 2025.” “Compare las membranas poliméricas de inclusión respecto a cada uno de los métodos convencionales…” |
| Complex Contextual Narrative | Lengthy, detailed queries that provide substantial background context, motivations, definitions, or examples before asking a main question or issuing a command. These queries resemble a mini-narrative, sometimes including an abstract, data, citations, or technical context. | “In this experiment, various culture systems for the GF677 rootstock were compared… Is there an explanation in the scientific literature for why leaf number and leaf area are greater…” “I have four papers and I want to combine them all to be a doctoral dissertation…” “The performance of T-junction micromixers, as analysed in CFD models…” |
| Multi-part/Multi-step Query | Queries composed of multiple distinct sub-questions, tasks, or steps, often divided by letters, numbers, or separate sentences. The sub-queries are related but require separate or structured responses. | “Review papers on SOTA and research gaps in manifold learning in AI. For each of the top 9 methods extract: a) Key architectural or algorithmic ideas… b) Reported SOTA metric… c) the code or data availability…” “Describe what this cognitive ability encompasses… Please use academic papers… for each part of your response.” “Find me the research papers about interictal epileptiform discharges (IEDs)… Does STD in excitatory neurons…” |
| Boolean or Logical Operators | Queries using explicit logical or Boolean operators or patterns (AND, OR, NOT, +, -, parentheses, slashes, etc.) to combine multiple search terms, constrain results, or designate alternatives. Sometimes appears as ‘Topic A/Topic B’, ‘x, y AND z’. | “adulteration detection in milk AND spectroscopy” “machine learning OR deep learning applications in medical imaging” “neural network NOT convolutional” |
Table 29: Complete ScholarQA query phrasing taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.
| Category | Description | Examples |
| Keyword-style Query | Short, often fragmentary, queries resembling search engine keywords or subject headings. No verbs or complete sentences; typically a list of nouns or concepts separated by spaces or simple punctuation. | “algorithmic trading” “ppf” “bayesian optimization applications” |
| Natural Language Question | Fully formed, grammatically complete questions using natural language. | “How does fear of missing out influence the effect of mindfulness on academic procrastination among students?” “Do a literature review on artificial intelligence in management accounting” “What cognitive strategies and techniques can improve focus and concentration for artists…” |
| Explicit Instruction/Imperative | Direct commands or requests instructing the system to perform an action (e.g., ‘find…’, ‘review…’, ‘compare…’). May be simple or include detailed steps, but typically begins with an imperative verb. | “Find papers that refer to the relationships between degradation ability of PROTAC and potency or affinity…” “Give papers on finetuning an llm for writing research papers” “Find systematic reviews and/or clinical practice guidelines published in the last 10 years…” |
| Complex Contextual Narrative | Lengthy, detailed queries that provide substantial background context, motivations, definitions, or examples before asking a main question or issuing a command. These queries resemble a mini-narrative, sometimes including an abstract, data, citations, or technical context. | “As restored ecosystems evolve towards the composition and structure characteristic of native vegetation…” “Equilibrium analysis enables the assessment of whether observed outcomes reflect a stable strategic configuration…” “Papers about innovation and hurdles within industries that prevent it…” |
| Boolean or Logical Operators | Queries using explicit logical or Boolean operators or patterns (AND, OR, NOT, +, -, parentheses, slashes, etc.) to combine multiple search terms, constrain results, or designate alternatives. Sometimes appears as ‘Topic A/Topic B’, ‘x, y AND z’. | “Find systematic reviews and/or clinical practice guidelines published in the last 10 years (2014–2025)…” “Papers introducing a dataset of an unscripted dialogue between 2 speakers…” “Stretching exercises are commonly integrated into physical education programs…” |
| Multi-part/Multi-step Query | Queries composed of multiple distinct sub-questions, tasks, or steps, often divided by letters, numbers, or separate sentences. The sub-queries are related but require separate or structured responses. | “Review papers on SOTA and research gaps in manifold learning in AI. For each of the top 9 methods extract: a) Key architectural or algorithmic ideas… b) Reported SOTA metric… c) the code or data availability…” “Describe what this cognitive ability encompasses… Please use academic papers… for each part of your response.” “Find me the research papers about interictal epileptiform discharges (IEDs)… Does STD in excitatory neurons…” |
Table 30: Complete Paper Finder query phrasing taxonomy. Taxonomy was built through iterative human review and LLM-assisted analysis using GPT-4.1 on a sample of 1000 queries.
| Field of Study | Field of Study |
| Agricultural Sciences | Law and Legal Studies |
| Anthropology | Linguistics |
| Arts and Design | Literature |
| Biomedical Sciences | Mathematics |
| Biology | Mechanical Engineering |
| Business and Management | Philosophy |
Table 31: The 28 fields of study used for query classification.
More in this topic
AI-Assisted Scientific Research
AI-assisted scientific research uses machine learning and AI agents to support discovery, analysis, experimentation, and scientific communication.
See all 253 papers