cs.HCOct 2, 2026
SaveAsterism: Exploring and Synthesizing Scattered Observations into Literature-Grounded Hypotheses and Theories
Organizations: Allen Institute for AI Seattle, Washington, United States · University of Washington Seattle, Washington, United States · Allen Institute Seattle, Washington, United States · University of Maryland College Park, Maryland, United States
Abstract
A theory draws many independent observations into one framework with novel hypotheses. A researcher building such a theory must synthesize observations scattered across many papers, each describing related concepts but often in different terms. Which concepts matter most also depends on their preferences and research questions. Recent approaches scale theory synthesis with LLMs, but automate away choices and intuitions from researchers. We present Asterism, which extracts observations from hundreds of papers as concept-relation triples, with concepts unified in a hierarchical ontology. Researchers curate an evidence graph using the ontology and aggregate observations at different levels of granularity to focus theory formation on specific phenomena of interest. In a field deployment (n=10), researchers worked from observations to theories, and kept concepts and hypotheses fitting their preferences. In two case studies, teams of immunology and agriculture researchers discovered mechanisms outside their standard analyses and constructed hypotheses worth follow-up experiments.
Figures & tables
Figure 1. Asterism on a graph extracted from around 100 papers about how AI explanations affect human-AI teams. Left: a chat agent that searches the extracted evidence on the user’s behalf, answers questions about it, and proposes extracted concepts and relations to be added to the canvas (Figure 4 ). Middle: the canvas, holding the concepts and relations the user has curated for their inquiry, tinted by concept category and colored by relation, with a badge on every node and edge counting the unique papers behind it (Figures 6 and 5 ). Right: the hypothesis drawer, where hypotheses generated based on evidence from the canvas are reviewed by the user, marked interesting or dismissed, and selected for synthesis into a theory (Figure 9 ). Each hypothesis lists the concept-relation triples it was drawn from, hovering highlights them on the canvas (Figure 9 ). Screenshot of the \sysname{} web interface, divided into three vertical panels. The left panel is a chat conversation in which the user asks the system to help build up a graph for a research question and the agent replies with a ranked list of concepts to explore, each with a paper count and an Add or Expand button. The middle panel is a node-link canvas holding about a dozen labeled concept boxes, such as Feature-Based Explanation, Example-Based Explanation, Domain Expertise, Overreliance on AI, Underreliance on AI, and Decision-Making Accuracy, joined by curved colored edges. The right panel lists generated hypotheses; the open one is titled Asymmetric Reliance Calibration Law and shows its statement, its scope, and its supporting evidence with concept-relation triples.
Figure 2. Asterism extracts experiments and observations as concept-relation triples from a large set of papers, and infers a shared ontology that normalizes and generalizes concepts across papers (top). The researcher can steer theory development by selecting concepts across abstraction levels and relations between them to curate an evidence graph (left). Based on this shared representation, Asterism generates hypotheses for her to review and select (center), and draws the hypotheses she keeps into a unified theory that stays grounded in her evidence graph (right). A diagram of the \sysname{} process in four panels. The top panel shows a stack of paper icons with an arrow into a list of four extracted triples, each drawn as a flask icon, a lettered source concept, a green up arrow or a red down arrow, and a lettered target concept. Dashed lines run from each triple to the leaves of a tree on the right headed Shared Ontology, Concept Normalization, whose nodes are colored circles arranged over four levels. The lower left panel, outlined in green, is headed Zoom Nodes, Navigate abstraction levels in the ontology, and Select Interesting Triples, Curate an evidence graph of interest. It holds two controls reading Zoom out, more abstract, and Zoom in, more specific, a checklist of the same four triples with three of them ticked, and a laptop screen showing a node-link evidence graph over the eight lettered concepts. A researcher in a lab coat sits at a laptop beside it. An arrow leads down to a panel outlined in orange, headed Triage Hypotheses, Review generated hypotheses and choose which to keep, holding six cards; each card groups two or three triples under a lightbulb icon and is marked below with either a green tick or a red cross, four ticks and two crosses in total. An arrow leads right to a panel outlined in blue, headed Provisional Theory, Stitch together kept hypotheses, showing five lightbulbs on a ring around a central circle labelled Provisional theory. Below it a dashed arrow annotated outside scope.
Figure 3. Part of the ontology Asterism inferred for Dora’s corpus, shown in the outline panel. Each row carries the number of papers behind it: Overreliance on AI mentioned in 74 papers while Deception of Reliance only in 1. The ontology also powers the concept-zoom interaction, where zooming in on a concept offers the ones nested under it and zooming out merges it into its parent (Figures 5 and 6 ). A screenshot of a collapsible outline panel listing ontology concepts as an indented tree. Each row has a coloured dot, a concept name, and a badge on the right giving a paper count. Top-level rows with pink dots read AI Explanation Technique with 57 papers, Task Performance Outcome, AI Reliance Behavior, User Trust, User Experience Outcome, and User Characteristics. Indented under AI Explanation Technique, rows with green dots read Feature-Based Explanation with 41 papers, Example-Based Explanation with 16, Counterfactual Explanation, Contrastive Explanation, Natural Language Explanation, Rule-Based Explanation, Confidence Score Display, Attention-Based Explanation, and Concept-Based Explanation. A third level under Example-Based Explanation lists Nearest Neighbor Explanation, Gradient Similarity Explanation, and Representer Point Selection. Under AI Reliance Behavior the rows include Overreliance on AI with 74 papers, Under-Reliance on AI with 19, and Appropriate Reliance with 26.
Figure 4. The triage widget the agent returns instead of writing to the canvas itself, here for the 26 relations it found around Overreliance on AI . Rows are split by what they would cost: the first group connects concepts already on the canvas, the rest bring in new ones. Each row carries the relation, the number of papers behind it, and how much those papers agree, and the best-supported rows arrive pre-checked. A screenshot of a modal dialog inside the chat panel, headed SUGGESTED RELATIONS with a count of 26 and the line: Relations around Overreliance on AI. Rows are grouped under two orange headers, CONNECTS CONCEPTS YOU ALREADY HAVE with one row, and BRINGS IN NEW CONCEPTS with twenty-four. Each row has a checkbox, a source concept, an italic relation such as increases, decreases, or no effect, the target Overreliance on AI, and a small line reading a paper count and an agreement percentage. The first six rows are checked and the rest are not. A link at the bottom reads Show 5 more, 15 hidden, above an orange button reading: Add 5 concepts plus 6 relations, 6 to 11.
Figure 5. The concept panel for Example-Based Explanation . The two controls below move the concept along the ontology. Zooming in replaces it with the six specific techniques grouped under it; zooming out replaces it with its parent, AI Explanation , and the checkbox keeps the current concept nested inside that parent instead of folding it away, which is the state Figure 6 shows on the canvas. A vertical side panel. At the top, the concept name Example-Based Explanation, a category chip reading intervention, a one-sentence definition, and a long list of aliases. Below are two boxed sections. The first, Zoom-in to Specific Concepts, lists six concepts including Correspondence-Based Explanation, ExMatchina Explanation, and kNN Explanation. The second, Zoom out to Broader Concepts, shows the parent AI Explanation with a badge reading 55 nodes, a checkbox for keeping the current concept nested inside the new group, and a sample of the concepts in that group. A footer button reads: See 92 observations across 19 sources.
Figure 6. Nesting keeps a concept’s aggregated evidence without hiding the techniques beneath it. Feature-Based Explanation is drawn as a group with LIME and SHAP nested inside it, so the group’s merged edge to Overreliance on AI and the two techniques’ own edges are on the canvas at the same time. Figure 5 shows the controls that produce this state. On a node, the dark pill counts the unique papers behind the concept and the orange pill the number of sub-concepts folded into it. Hovering over the edge out of Example-Based Explanation shows the detailed distribution of reported relations from different papers. Clicking on the edge leads to Figure 7 . A screenshot of a small region of the canvas. On the left, a group node drawn with a dashed border is labelled Feature-Based Explanation and contains two child boxes, LIME Explanation and SHAP Explanation, each with its own source-count badge. Curved edges leave the group and each child and converge on a node labelled Overreliance on AI. Their labels read increases with 3 sources, no effect with 4 sources, and increases with 14 sources. Below, a separate node labelled Example-Based Explanation connects to the same target through an edge whose label is a stack of three rows reading increases, No Effect, and Decreases.
Figure 7. The evidence panel for the aggregated edge from Example-Based Explanation to Overreliance on AI . The bars break the 12 triples behind the edge down by relation as a share of the 6 unique papers reporting them; the shares add to more than 100% because a paper that reports two different relations counts toward both. Below, the panel unfolds the aggregate into the specific concept-relations it merged, each with the raw phrasing its paper used next to the canonical relation it was normalized to. Each of these unfolds one step further into the observation and experiment behind it (Figure 8 ). A vertical side panel headed Edge. At the top, two boxes joined by an arrow read Example-Based Explanation and Overreliance on AI. Beneath them, three labelled bars read Increases 83 percent, No Effect 17 percent, and Decreases 17 percent, followed by the line: 12 triplets, 6 sources. The lower part of the panel, headed Specific Concept-Relations, Normalized versus Raw Relations, is a stack of cards. Each card names a more specific source concept, such as Gradient Similarity Explanation or Representer Point Selection Explanation, with the same target, the normalized relation, the raw relation phrases in small chips, and a count of triplets and sources.
Figure 8. Every relation on the canvas traces back to sentences in a paper. This is the last step of the drill-down that starts at an edge in Figure 7 . Left: the evidence behind one triple, showing the observation as it was extracted, the triples drawn from that observation, the claim the authors made from it, and the experiment that produced it. Right: the same paper opened in the reader, with the sentence the observation was extracted from highlighted in place, so a user can check the extraction against the authors’ own wording and read around it. (Example paper ( Morrison et al., 2023 ) ). A two-pane screenshot. The left pane is a scrolling panel headed with a paper title, The Impact of Imperfect XAI on Human-AI Decision-Making, and the note 1 observation. It contains an OBSERVATION paragraph reporting relative self-reliance figures for expert participants, a TRIPLETS section with two extracted concept-relation triples each carrying an Add or Remove button, and CLAIM and EXPERIMENT sections describing what the authors concluded and how the study was run. The right pane shows the rendered PDF of that paper, open at a section headed Deception caused by Imperfect XAI, with one long sentence highlighted in yellow, the same sentence the observation was extracted from.
Figure 9. A saved theory, synthesized from hypotheses kept in the review drawer on the right of Figure 1 . The theory starts with a description that ties the hypotheses together under a single lens. The inline badges in the description reference the evidence behind each claim in the user-curated graph on the left, and hovering one highlights the corresponding edge. The hypotheses listed below can also be hovered to show the parts of the evidence graph they reference. The Conflicting Evidence and Unaccounted For sections report evidence on the canvas that runs against the theory or is not covered by it (based on ( Jansen et al., 2026 ) ). A two-pane screenshot. The right pane is the theory panel. It shows the theory name, Expertise-Modulated Explanation Reliance Theory, a paragraph describing the three interlocking claims it is built from, with small inline badges labelled e2.0 and e4.0 embedded in the text, and buttons reading Restore Full Canvas and Visualize Theory Nodes. Below that, a section headed Hypotheses lists three collapsed rows, each with a name and a tag reading mixed or qualitative. At the bottom, a section headed Conflicting Evidence contains two grey boxes describing findings that run against the theory. The left pane is the canvas, where five concept nodes joined by green and red edges are drawn at full contrast while the remaining nodes are faded out.
| Scope | Ours closer | Zhu et al. closer | Tie | Mean best-J (ours) | Mean best-J (Zhu et al.) |
|---|---|---|---|---|---|
| Pooled 288 concepts | 204 (71%) | 55 (19%) | 29 (10%) | 0.508 [0.484, 0.532] | 0.375 [0.354, 0.397] |
| Averaged 12 ontologies | 74.0% [64.3, 83.7] | 16.7% [8.2, 25.3] | 9.3% [3.1, 15.4] | 0.542 [0.477, 0.608] | 0.391 [0.350, 0.431] |
Table 1. Per-concept best-match Jaccard over the 288 non-trivial survey concepts, ours against the reference method of Zhu et al. ( Zhu et al., 2025 ) . The first row pools all 288 concepts; the second computes each statistic within each ontology and averages across the 12 ontologies (95% CIs over those 12 values).
| ID | Min. | Ses. | Theory question |
| P0 | 123 | 10 | How does the presence of external members (AI agents, new hires, new friends) affect group dynamics? |
| P1 | 109 | 10 | How do different context-management strategies in agentic AI systems affect task performance? (and 2 more) |
| P5 | 53 | 7 | How does different AI explanation technique affect users on cognitive load and task performance? (and 1 more) |
| P6 | 179 | 16 | How do different grounding strategies (retrieval, tool, human feedback) affect the novelty and rigor of AI-generated scientific discoveries? (and 3 more) |
| P7 | 90 | 5 | How do different AI interaction interface designs (conversational, embedded, agentic) affect user adoption, engagement, and task performance? (and 1 more) |
| P8 | 76 | 8 | How do different presentations of AI diagnostic advice […] affect clinician overreliance and diagnostic accuracy in clinical decision support settings? (and 1 more) |
Table 2. The theory questions each participant worked on. Some created additional graphs during the study (Table 3 ). A session ends when there was 15+ minutes of inactivity. The final four rows are the two case-study teams (§ 7 ), whose engagement ran through meetings and self-directed use outside the deployment protocol, so minutes and sessions are not shown.
| P0 | P1 | P5 | P6 | P7 | P8 | P13 | P15 | P18 | P19 | ||
| Canvases | 1 | 2 | 2 | 4 | 2 | 1 | 4 | 2 | 1 | 3 | 2.2 ± 1.1 |
| Concepts | |||||||||||
| Added | 60 | 38 | 32 | 287 | 51 | 33 | 70 | 91 | 33 | 47 | 74.2 ± 77.1 |
| Removed | 16 | 0 | 0 | 22 | 0 | 0 | 48 | 62 | 0 | 2 | 15.0 ± 22.7 |
| Zoomed | 59 | 0 | 0 | 0 | 0 | 0 | 75 | 0 | 0 | 2 | 13.6 ± 28.4 |
| Hypotheses | |||||||||||
Table 3. Engagement per participant, based on the interaction logs. Zoomed counts the child concepts revealed by zooming in or grouped by zooming out. Removed counts the distinct concepts a participant placed and later took off the canvas, by hand or by clearing it, and Canvases counts the canvases each participant saved.
Figure 10. Two common strategies interacting with Asterism. Top: how often they took different kind of action (smoothed over 3 minutes; with each curve scaled to its own peak.) Bottom: the number of concepts on their canvases. The horizontal axis is engaged time, concatenating multiple sessions. P0 added concepts more aggressively, and curate further by removing and zooming them; P7 added concepts more selectively, and never removed any. Both inspected evidence throughout, and P7 additionally went back to evidence on the canvas after theory creation, while P0 mostly read the evidence included in the theory report (not shown). Interaction traces of P0 and P7 demonstrate two common strategies using \sysname{} each with a horizontal axis of engaged minutes running from zero to about 120. Each panel holds an upper plot of six smooth colored curves, one per action type, and a lower plot with a dark step line for the number of concepts on the canvas. A colored strip along the top of each panel alternates between two shades marking building the evidence graph and generating and triaging hypotheses. In P0's panel the six curves rise and fall repeatedly through the first hundred minutes, and the concepts line climbs to a peak, dips, recovers, and falls again near the end. In P7's panel the curves are sparser, the concepts line rises early and then stays flat, and the amber Read Evidence curve peaks again near minute eighty-five, inside the stretch marked as hypothesis work. A legend below the two panels names the six action colors: Read Evidence, Chat, Add Concept, Remove Concept, Zoom Concept, and Hyp/Theory.
| Source | P0 | P1 | P5 | P6 | P7 | P8 | P13 | P15 | P18 | P19 | % |
| Browse List | 14 | 0 | 1 | 6 | 0 | 2 | 10 | 0 | 0 | 6 | 6 ± 8 |
| Concept-Neighbors | 10 | 0 | 0 | 9 | 1 | 0 | 28 | 1 | 1 | 0 | 7 ± 13 |
| Concept-Hypotheses | 9 | 0 | 0 | 192 | 0 | 0 | 0 | 35 | 22 | 13 | 21 ± 27 |
| Chat | 27 | 38 | 31 | 80 | 50 | 31 | 32 | 55 | 10 | 28 | 66 ± 29 |
| /Proposed | /50 | /80 | /52 | /124 | /61 | /69 | /74 | /99 | /27 | /34 |
Table 4. Participants used different methods to add concepts, ranging from browsing the full list of triples to find familiar concepts, to anchoring on a node to see its relational and ontological neighbors , to adding the concepts mentioned in a pre-generated hypothesis , and to asking the chat agent to propose concepts or triples.
| Term | Definition |
|---|---|
| Kept (outcome) | Hypothesis marked as interesting or added to a theory |
| Breadth | # of canvas relations cited |
| LeafConcepts | % of cited concepts at the ontology’s most specific level |
| AnchorPapers | # of papers behind the best-supported relation cited |
| AnchorAgreement | % of its relations in the majority direction |
Table 5. The model’s outcome and four hypothesis properties.
Appendix figures & tables2 assets
Supplementary material from the paper’s appendix.
Appendix
| Tool | Purpose |
|---|---|
| Finding concepts and relations | |
| search_suggested_triples | Keyword-search the ranked pool of triples that can be added to the canvas, matching against source and target labels, their aliases, and the relation text. Returns up to 50 triples ranked by how many papers back the relation and how far they agree on its label; called with no arguments it returns the whole addable pool. Triples conflicting with concepts already on the canvas are filtered out. |
| expand_around_node | Walk one hop outward from a single concept, returning the top-ranked pool triples whose other endpoint is not yet on the canvas. Restricted to incoming relations, outgoing relations, or both. |
| find_paths_between | Find directed chains of pool triples connecting two concepts through intermediates, for questions of the form “how does A lead to B” and for bridging disconnected regions of the canvas. |
| Proposing changes to the canvas | |
| propose_concepts | Render a card of up to eight concepts, one clickable row each with a one-line gloss from the agent; the label and paper count on each row are computed from the data rather than written by the model. Clicking a row adds that single concept and opens a relations widget around it. |
Table 6. The eleven tools available to Asterism’s chat agent. Relation labels in the ranked triple pool are derived from the extracted evidence, so a tool may choose which concept pairs to propose but never what relation holds between them.
| Metric | Ours | Zhu et al. | Per-ontology higher |
|---|---|---|---|
| NMI | 0.666 [0.597, 0.735] | 0.622 [0.557, 0.687] | 8 / 12 |
| ARI | 0.255 [0.183, 0.327] | 0.185 [0.101, 0.270] | 10 / 12 |
| Purity | 0.644 [0.560, 0.728] | 0.585 [0.506, 0.665] | 6 / 12 |
Table 7. Partition-level scores under Zhu et al.’s protocol (higher is closer to the survey ontology), with 95% CIs over the 12 ontologies.
More in this topic
Scientific Hypothesis Generation
Scientific hypothesis generation develops methods for proposing testable explanations and research directions from literature, data, and experiments.
See all 90 papers