Scholarly sensemaking requires researchers to iteratively retrieve, collect, and synthesize literature as their understanding evolves. However, existing scholarly tools provide only partial support for this process. Recent artificial intelligence (AI)-based systems generally rely on opaque submit-and-wait pipelines that it users' ability to inspect, revise, and validate intermediate outputs. In this study, we present HiPaper, a human-AI collaborative literature review system informed by a formative study with academic experts. HiPaper organizes scholarly sensemaking as a three-stage interactive workflow spanning Retrieval, Collection, andSynthesis, centralized on the human-in-the-loop paradigm that exposes intermediate AI-generated content for user inspection and intervention through a persistent workspace called Knowledge Pocket. Through this workflow, users can iteratively refine search intent, curate and organize papers with AI-enriched metadata and clustering, and conduct evidence-traceable synthesis over selected literature. Our user study with 14 Ph.D. students showed that HiPaper efficiently supports scholarly sensemaking by enabling human-AI collaboration in both familiar and unfamiliar domains. Furthermore, the findings highlight how staged interactions with intermediate AI-generated content successfully scaffold user interactions throughout the sensemaking process.
Figures & tables
Figure 1. HiPaper : A system with three-stage human-AI collaborative workflow. Retrieval (Left): User express research intent, and the AI expands query terms via keyword suggestions; users decide to accept or refuse. Collection (Middle): AI organizes retrieved papers into clusters and extracts paper structures; users explore and curate evidence within a dedicated Knowledge Pocket workspace. Synthesis (Right): Users ask comparison questions over curated papers; AI generates grounded insights; users validate and iterate through follow-up questions.
Figure 2. HiPaper retrieval and collection stages interface is organized into five panels. (A) Search Panel: Users enter a query; the AI expands the query with keyword suggestions, which users can Accept or Refuse via dedicated buttons; (B)Cluster & Pocket Panel: Retrieved papers are grouped into topic clusters ( e.g. , Human Computer Interaction, Machine Learning); users can organize curated papers into named Pockets ( e.g. , Pocket 1, Pocket 2) and trigger Deep Analysis on selected collections ; (C) Paper List Panel: each paper entry displays its title, authors, venue, year, conference tier, relevance score, and two AI-generated summaries — Background and Methodology — enabling rapid triage without opening the full paper; (D) Action Panel: Each paper row with three UI buttons including Add to Pocket, Similar paper, and Dislike. (E) Knowledge Pocket Panel: Users continuously select papers from clusters and manage curated paper collections across multiple named Pockets.
Figure 3. HiPaper ’s synthesis interface, organized into four panels. (E) Knowledge Pocket Panel: Users manage curated paper collections across multiple named Pockets. Papers are organized by cluster, with checkboxes for selection, and options to Add to Chat; (F) AI Insight Dashboard: Displays an automatically generated visual summary of relationships ( e.g. , Different colored lines between nodes) among selected papers, including conceptual links such as Intersection, Interpretation, Expansion, Abstraction, Reification, and Analogy, helping users identify structural connections across literature; (G) Pocket Dashboard: analyzes papers in the Knowledge Pocket and presents a visual summary of structural connections among them. (H) Evidence-Traceable Chat Panel: Users trigger AI-generated synthesis actions, inlcuding Summarize , Compare , Timeline , Research Gaps , and Method on selected papers. The AI produces grounded, citation-based responses with inline references ( e.g. , [1], [2]) traceable to specific papers. Users can further continue iterating via free-form questions in the input field.
Figure 4. Phase Progression Over Task Time.
Figure 5. Action-to-Action Transition Heatmap.
Appendix figures & tables21 assets
Supplementary material from the paper’s appendix.
Appendix
Figure A.1. Backend workflow of HiPaper . The system moves from data fetching to retrieval, paper collection, and synthesis. (a) Papers from selected venues and years are fetched from metadata sources, normalized into a unified record format, and encoded from title and abstract for vector indexing. (b) A user’s rough query is rewritten into a retrieval-oriented academic query to retrieve semantically similar papers from the vector database. (c) Retrieved papers are collected as structured records and enriched with LLM-generated metadata. Users can also add similar papers through similarity-based browsing. (d) Selected papers are organized into Knowledge Pockets for streaming paper relationship graph construction and evidence-traceable QA.
Figure B.1. Prototype Retrieval Workspace.
Figure B.2. Prototype Collection Workspace.
Figure B.3. Prototype Similar-Paper Recommendation in the Collection Workspace.
Figure B.4. Prototype Synthesis Workspace for Paper Analysis.
Figure B.5. Prototype Synthesis Interface with Graph-Based Analysis and AI Assistant.
Raw event type
Phase label
Heatmap label
Note
search_query
Retrieval
Retrieval
User issues a search query to retrieve candidate papers or information in search page.
header_search
Retrieval
Retrieval
User issues a search query to retrieve candidate papers or information in the header of explore page.
action_add_to_pocket
Collection
Action Add To Pocket
User saves a paper or item to the pocket for later consideration.
action_dislike
Collection
Dislike
User marks a paper or item as irrelevant or undesirable.
action_remove_from_pocket
Collection
Action Remove From Pocket
User removes a previously saved paper or item from the pocket.
action_similar
Collection
Similar
User requests or opens similar items related to the current paper.
Appendix
Table 1. Event mapping used across the two behavioral analyses. The phase-progression analysis mapped raw events to high-level sensemaking phases, while the transition analysis used human-readable action labels for visualization.
Abbreviation
Venue Name
#Record
Domain
CORE
CCF
JCR
AAAI
AAAI Conference on Artificial Intelligence
11416
Conference
A*
A
–
AAMAS
International Conference on Autonomous Agents and Multiagent Systems
2076
Conference
A*
B
–
ACL
Annual Meeting of the Association for Computational Linguistics
8374
Conference
A*
A
–
CHI
ACM Conference on Human Factors in Computing Systems
4612
Conference
A*
A
–
CSCW
ACM Conference on Computer-Supported Cooperative Work and Social Computing
781
Conference
A
A
–
CVPR
IEEE/CVF Conference on Computer Vision and Pattern Recognition
11674
Conference
A*
A
–
Appendix
Table 2. Summary of Venues in the Database of HiPaper Systems.
Key
Value Type
Example
Meaning
doi
str
"10.1609/AAAI.V36I1.12345"
DOI of the paper.
title
str
"Anonymized Research Title on Learning Systems"
Paper title.
authors
list[str]
["Author A", "Author B", "Author C"]
Ordered list of authors.
venue
str
"AAAI"
Venue abbreviation.
year
int / str
2022
Publication year.
pdf_url
str
"https://publisher.org/article/view/12345/12000"
URL to the paper PDF.
Appendix
Table 3. Schema Summary of Paper Records (Examples are Anonymized).
Systematic Literature Reviews (SLRs) are essential for evidence-based research but remain time-consuming, requiring researchers to manage large volumes of publications across planning, screening, analysis, and reporting. Large language models (LLMs) can now produce fluent, well-structured review text, which makes it difficult to distinguish synthesis that was verified by a researcher from synthesis that merely appears authoritative. This raises the risk that unverified AI-generated synthesis enters the scholarly record carrying the credibility of a systematic review. We present CoSLR, a Human-AI collaborative multi-agent system that supports the SLR workflow through a modular three-phase pipeline using large language models and Retrieval-Augmented Generation (RAG), and that places explicit, mandatory human checkpoints on the path between generated output and its acceptance. In a survey-based study with 63 participants, the system was received positively: 27 of 63 participants (42.9 percent) rated its usability highly, indicating that the mandatory checkpoints did not come at the cost of a workable interface. However, a checkpoint safeguards the review only if researchers use it to verify: 22 of 63 participants (34.9 percent) reported that they would trust AI-generated summaries and reports without additional human checking after only a short interaction with the system. These findings indicate that Human-AI collaboration can support literature review work, but that the effectiveness of human oversight depends on whether users are willing to exercise it. This is a calibration problem that interface design must address directly, not assume.
MD Aidul Islam, Malik Abdul Sami, Muhammad Waseem +4
There are different goals for literature research, from understanding an unfamiliar topic to generate hypothesis for the next research project. The nature of literature research also varies according to user's familiarity level of the topic. For inexperienced researchers, identifying gaps in the existing literature and generating feasible hypothesis are crucial but challenging. While general deep research'' tools can be used, they are not designed for such use case, thus often not effective. In addition, the black box" nature and hallucination of Large Language Models (LLMs) often lead to distrust. In this paper, we introduce a human-agent collaborative visualization system AwesomeLit to address this need. It has several novel features: a transparent user-steerable agentic workflow; a dynamically generated query exploring tree, visualizing the exploration path and provenance; and a semantic similarity view, depicting the relationships between papers. It enables users to transition from general intentions to detailed research topics. Finally, a qualitative study involving several early researchers showed that AwesomeLit is effective in helping users explore unfamiliar topics, identify promising research directions, and improve confidence in research results.
Zefei Xie, Yuhan Guo, Kai Xu
School of Computer Science, University of Nottingham, UK · School of Intelligence Science and Technology, Peking University, China
ResearchPilot is an open-source, self-hostable multi-agent system for literature-review assistance. Given a natural-language research question, it retrieves papers from Semantic Scholar and arXiv, extracts structured findings from paper abstracts, synthesizes cross-paper patterns, and drafts a citation-aware related-work section. The system combines FastAPI, Next.js, DSPy, SQLite, and Qdrant in a local-first architecture that supports bring-your-own-key model access and remote-or-local embeddings. This paper describes the system design, typed agent interfaces, persistence and history-search mechanisms, and the engineering tradeoffs involved in building a transparent research assistant. Rather than claiming algorithmic novelty, we present ResearchPilot as a systems contribution and evaluate it through automated tests and end-to-end local runs. We discuss limitations including external API rate limits, abstract-only extraction, incomplete corpus coverage, and the lack of citation verification.