cs.HCApr 3, 2026
SaveLitPivot: Developing Well-Situated Research Ideas Through Dynamic Contextualization and Critique within the Literature Landscape
Organizations: University of Pennsylvania Philadelphia, USA · Allen Institute for AI Seattle, USA
Abstract
Developing a novel research idea is hard. It must be distinct enough from prior work to claim a contribution while also building on it. This requires iteratively reviewing literature and refining an idea based on what a researcher reads; yet when an idea changes, the literature that matters often changes with it. Most tools offer limited support for this interplay: literature tools help researchers understand a fixed body of work, while ideation tools evaluate ideas against a static, pre-curated set of papers. We introduce literature-initiated pivots, a mechanism where engagement with literature prompts revision to a developing idea, and where that revision changes which literature is relevant. We operationalize this in LitPivot, where researchers concurrently draft and vet an idea. LitPivot dynamically retrieves clusters of papers relevant to a selected part of the idea and proposes literature-informed critiques for how to revise it. A lab study () shows researchers produced higher-rated ideas with stronger self-reported understanding of the literature space; an open-ended study () reveals how researchers use LitPivot to iteratively evolve their own ideas.
Figures & tables
Figure 1. LitPivot supports the co-development of a research idea and its grounding in the literature through literature-initiated pivots (in purple). (1) The authors’ original idea (verifying hallucinations in AI-generated audio) has too little prior work to motivate it, so LitPivot suggests pivoting to verifying hallucinations in AI-generated text. (2) That idea is not novel, so LitPivot suggests traversing retrieved citations in scientific RAG systems. (3) The resulting idea builds on prior work while remaining differentiated from it. A diagram in two columns tracing three versions of a research idea. The right column shows three rounded boxes stacked vertically, labeled Idea 1, Idea 2, and Idea 3, each surrounded by small circular icons representing clusters of retrieved papers. Idea 1 reads "verify hallucinations in AI-generated audio" and has one nearby paper cluster. Idea 2 reads "verify hallucinations in AI-generated text" and is surrounded by five clusters. Idea 3 reads "verify scientific QA answers by traversing citations" and is bordered by three clusters. Purple curved arrows run from Idea 1 down to Idea 2, and from Idea 2 down to Idea 3, marking each literature-initiated pivot. The left column holds three numbered boxes aligned with the ideas. Box 1 reads "critique: problem lacks citable work" and "rewrite: verify attributed text." Box 2 reads "critique: idea matches prior work" and "rewrite: verify scientific QA answers." Box 3 reads "well-grounded, novel idea with respect to the literature" and ends with a check mark.
Figure 2. The LitPivot interface. The left pane is a document editor with a toolbar toggle to highlight facets. The right pane is a paper panel that surfaces literature clusters. The interface colors text by idea facet: problem (purple), contribution (green), and evaluation (orange; currently missing). Clusters most relevant to the selected facet (here, the contribution statement) are starred. After selecting papers, the researcher can generate an assessment to compare the idea against selected work. The LitPivot interface with explanatory callouts. Left Pane (Editor): A callout points to a toggle to "enable facet highlighting." The text below is color-coded: a purple sentence (problem), a green sentence marked "selected facet" (contribution), and an orange placeholder marked "missing facet" (evaluation). Right Pane (Literature): Displays a list of related paper topics. A callout for "salient literature clusters" points to two starred topics highly relevant to the selected green facet. At the bottom, a callout points to a blue "generate an assessment" button.
Figure 3. LitPivot pipeline for literature retrieval and faceted paper clustering. (1) The researcher’s idea is used to retrieve an initial set of relevant papers via Asta PaperFinder. (2) GPT-4o extracts idea-relevant facets from each paper. (3) Sonnet-4 clusters the literature by facet, surfacing groups of papers most relevant to each part of the idea. (4) The system extracts papers that are meaningfully cited within the extracted facets and includes this in the literature set. A four-step flowchart illustrating the LitPivot pipeline for literature retrieval and clustering: Step 1: Retrieve initial set of papers. A "researcher-written idea" is processed by Asta PaperFinder to output a stack of "relevant literature." Step 2: Extract facets from each paper. GPT-4o analyzes the literature and highlights color-coded facets within the text. Step 3: Cluster literature by facet. Sonnet-4 organizes the papers into distinct stacks grouped by their specific color-coded facets. Step 4: Extract salient cited literature. Asta Snippet Search pulls cited snippets from the faceted text to discover new papers. An arrow loops these new papers back to Step 2, indicating an iterative cycle.
Figure 4. Distribution of change in Likert scores (post task - pre task) for perceived idea novelty, feasibility, and utility across conditions. Horizontal bars indicate median and quartile values; the red dashed line marks no change. A violin plot compares self-reported change in idea quality between two conditions: LitPivot (blue) and Baseline (orange). Three metrics are shown on the x-axis: change in Novelty, Feasibility, and Utility (post–pre scores). For all three metrics, LitPivot distributions are wider and centered above zero, indicating greater positive change, whereas Baseline distributions are narrower and closer to or below zero, suggesting little to no improvement. Horizontal lines within each violin indicate the median and interquartile range, and a red dashed line at zero marks no change.
Appendix figures & tables4 assets
Supplementary material from the paper’s appendix.
Appendix
| Baseline | LitPivot | |
|---|---|---|
| Initial Idea 1: Current RAG QA systems are touted as more interpretable and explainable because they point to existing sources. But sometimes these sources aren’t even relevant to the generation. We propose a tool that helps users engage more with source texts and better discern the quality of source texts. | Recent research has highlighted a critical gap in RAG QA systems where retrieved documents that appear relevant based on surface-level similarity, don’t actually support the specific claims being made in the generation. While RAG systems theoretically provide transparency by surfacing source documents, the granularity of attribution is often too coarse (typically at the document or sentence level) to be truly useful for verification, as demonstrated in works on fine-grained attribution. The mismatch between retrieved sources and generated content not only undermines the system’s interpretability claims but also potentially misleads users who trust the system’s ability to ground its responses in reliable sources. We propose a tool that combines fine-grained attribution techniques with an interactive interface, where users can click on specific claims in the generated text to see the exact supporting quotes from source documents highlighted in context. The tool also employs a traffic light system to visually indicate the strength of attribution for each claim (green for direct quotes/strong support, yellow for partial/indirect support, red for unsupported claims), helping users quickly identify potential issues. We can evaluate our tool through a multi-faceted approach combining: (1) precision/recall metrics comparing system-identified attributions to human-annotated gold standards, (2) semantic similarity scores between generated claims and source text segments using established embedding models, (3) user studies measuring verification time and accuracy with and without the tool. | Current RAG QA systems are touted as more interpretable and explainable because they point to existing sources. However, these systems still tend to fail for certain tasks, e.g. they can hallucinate information, miss key references, and find wrong or unrelated sources. Existing literature focus on attribution (and how to use them for improving QA for LLMs, but not how to engage with these attributions, nor how to help users discern the quality of these attributions. Enabling users to do so empowers users to be more diligent in checking the answers and sources’ quality and improves the trustworthiness and transparency of such RAG QA systems. We propose a tool that helps users engage more with source texts and better discern the quality of source texts. We introduce a voice- and pointer-driven RAG interface in which (i) users can issue spoken follow-ups such as show me the sentence that supports that claim or read the most reliable paragraph aloud, and (ii) every cited span is accompanied by an automatically generated reliability badge that aggregates contradiction, sufficiency and source-authority scores computed with CiteEval-AUTO and AutoAIS. The system also supports LAQuer-style localized queries so that users can drill down into sub-sentences, with the interface highlighting the exact supporting tokens. Our tool provides 1) a novel way for users to interact with these sources through multimodal interaction paradigms including source navigation (via attribution) and voice interaction, rather than treating RAG as a blackbox component in the QA pipeline, and 2) a novel way for users to easily verify and discern source quality. We focus on textual content as the source, leaving multimodal content (images, videos, PDFs, voices) as future work. A controlled user study will measure whether the badge + drill-down workflow improves verification speed and accuracy compared to the baseline clickable-citation UI used in TruthReader. |
| Initial Idea 2: Existing methods for verifying scientific claims across research papers typically lack systematic checks for logical consistency. We want to develop an interactive semi-formal proof navigator that translates natural-language claims from research papers into structured logical propositions. | Existing methods for verifying scientific claims across research papers typically lack systematic checks for logical consistency. We want to develop an interactive, two-way “semi-formal proof navigation” system that 1) translates natural-language claims from research papers into structured logical propositions using various formal logic representations for flexibility, 2) enables visualizing and interacting with the claims and their logical relationships. The system automatically highlights parts with problematic claims and suggests remediations, and enables the user to modify visualized logical claims where the results will be updated in the original content, and vice versa. | Scientists currently lack automated tools that can detect when claims drawn from different papers form a logically inconsistent chain of reasoning. Existing scientific fact-checking resources focus on single-claim, single-paper links for example, SCIFACT targets abstract-level support or refutation, and CLAIM-BENCH evaluates claim-evidence reasoning within one paper leaving cross-paper logical consistency unchecked. Without such cross-document verification, meta-analyses and policy decisions risk propagating contradictions or unsupported citation leaps in an ever-growing literature, making a principled consistency checker both timely and necessary. We will build an interactive proof-navigation environment that (i) harvests claims and supporting sentences from multiple research papers, (ii) auto-formalises each statement into FOL while attaching citation metadata, (iii) dynamically chooses the most reliable symbolic solver per claim type, (iv) stitches the resulting formulas into a cross-paper proof graph, and (v) lets users step through, edit, and re-verify any node to expose contradictions or unsupported leaps. |
Table 1. Side-by-side comparison of baseline ideas and their LitPivot counterparts. Diagonal entries are from the same participant. Three columns (Initial Idea, Baseline, LitPivot) and two rows detailing different research concepts. Row 1 compares paragraph-length baseline and LitPivot expansions for "Initial Idea 1," which proposes a tool to help users engage with and verify the quality of source texts in RAG QA systems. Row 2 compares paragraph-length baseline and LitPivot expansions for "Initial Idea 2," which proposes an interactive semi-formal proof navigator to verify scientific claims across research papers using logical propositions. Note: The table contains text in the Baseline and LitPivot columns demonstrating how each participatn expands on the initial ideas.
Figure 5. When a change to one idea facet might require editing another idea facet, LitPivot highlights the corresponding affected idea facet in the text editor. Here, the problem statement facet is outlined. An editor view with the label
affected idea facet.'' The first sentence of the idea statement (Traditional planning tools are really inaccessible\ldots'') is outlined with a dashed purple box, indicating the specific facet of the idea that a critique or suggestion will modify.Figure 6. Baseline interface. The left pane contains an idea editor where researchers can write out their research idea. The right pane shows a list of retrieved papers. At the bottom, a chat box allows researchers to ask questions about their idea or the listed literature. A baseline interface with three main components. On the left, an idea editor pane lets researchers write out their research idea in free text. On the right, a list of papers displays retrieved literature relevant to the idea, each with a short description. At the bottom, a chat box invites researchers to ask questions about their idea or the listed papers, supporting interactive exploration.
Figure 7. Design probe used in the formative study. This is an LLM-powered prototype. The research hypothesis is displayed at the top. Verdict tags are assigned below. There are recommendations, supporting and negating viewpoints that use related literature to ground feedback. The design probe interface. At the top, the user can select a hypothesis file ID from a dropdown. The selected hypothesis is displayed with a description (
An interactive interface that helps programmers explore and compare program traces to enhance debugging'') and is tagged with the verdict tags Frontier'' and Blindspot.'' Buttons allow the user to expand or collapse a summary and view uncertainties. A green-highlighted recommendation suggests focusing on cognitive and performance challenges and conducting broader empirical validation. Below, a section titled Viewpoints and Supporting/Negating Snippets'' lists viewpoint entries that can be expanded to show related evidence from the literature.More in this topic
Research Idea Generation
Research idea generation uses computational methods, especially language models, to propose and assess new questions, hypotheses, and research directions.
See all 60 papers