MuninAI: A Self-Hosted Agentic Framework for Independent Research Groups
Authors: Markus Fischer, Jens Meiler, Matthias Elgeti
Organizations: Institute for Drug Discovery, Leipzig University Medical School, Leipzig, Germany · Institute for Medical Physics and Biophysics, Leipzig University Medical School, Leipzig, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Dresden/Leipzig, Germany · School of Embedded Composite Artificial Intelligence (SECAI), Cooperation of University Leipzig and TU Dresden, Dresden/Leipzig, Germany · Faculty of Mathematics and Informatics, Faculty of Chemistry, Leipzig University, Leipzig, Germany · Integrative Center for Bioinformatics, Leipzig University, Leipzig, Germany
Research groups that want a language model grounded in their own literature face a choice between hosted services that receive their unpublished work and self-hosted deployments that keep the data local but give the model no access to the corpus. Munin is an open-source framework for a self-hosted scientific assistant that indexes a group's corpus and its citation graph, and wraps whichever open-weight model the group can serve in an agent harness with specialised reading, search, and computation tools. We claim that the harness, rather than the specific model, is what makes the system useful on domain literature, and we ship the evaluation suite that tests this claim with the software. On 199 paired literature questions, the harness more than doubles the accuracy of the base model, from 0.387 to 0.874 (+0.487, 95% CI [0.407, 0.568]), while naive retrieve-then-answer scores below the unaided model on every backbone tested. Under the harness, roughly twice as many answer claims trace to a passage the model read itself as with fixed top-5 retrieval. When the source paper for a question is removed from the corpus, the system abstains on about nine in ten of the questions it had answered correctly on the Qwen backbones and eight in ten on gpt-oss-20b, and it produces no fabricated local citation across 300 fabricated references. The pattern holds on three backbones from two developers, including a 20-billion-parameter model that runs on a single consumer-grade GPU. With all outbound network access disabled, both Qwen backbones still exceed the published PaperQA2 accuracy. The evaluation suite runs as a single command, so an adopting group can re-run it on its own corpus and model.
Figures & tables
Work
Domain
DOI-level ground truth
Withholds real documents from a live corpus
Confabulation axis
KnowOrNot [9]
Government policy
No
Leave-one-out on a static corpus
No
Sufficient Context [10]
General QA
No
No
No
RefusalBench [11]
General QA
No
No (perturbs context)
No
AbstentionBench [12]
General
No
Not corpus-relative
No
CiteAudit / CiteCheck [17, 18]
Scientific text
Partial
No
Yes
This work
Scientific literature
Yes, by construction
Yes (paired shadow corpus)
Yes
Table 1
Fig. 1 Deployment boundary and one turn through the harness. All inference, agents, the sandbox, and the corpus live on the cluster's own machines. The gateway reaches them only through a reverse SSH tunnel initiated from the cluster side, so no inbound port is exposed. When a query arrives, a per-turn router decides to which persona it is passed. The bare LLM is also accessible. Inside the harness, the model invokes agents through core tools and tool_search. Source and search agents return compact results tagged with an outcome (resolved, not_found, or thin_evidence), while keeping the full document text in scope. Deep research is available behind a user-facing toggle and writes its report as a saved artifact. After the turn completes, the citation audit runs separately and attaches a warning to the stored answer if any cited reference cannot be verified. Outbound network access is controlled at the egress gate on a per-request basis: off blocks all external calls, oa_only permits scholarly APIs, and full additionally opens web search and fetch. If the agents determine that the source for a question is absent from the corpus, they return not_found. From there, whether to attempt an answer or to decline is the model's own decision, and a decline is emitted over the same streaming path as any other response (Section 5.3).
Outcome
Meaning
Caller contract
not_found
Document parsed successfully; answer not present
Treated as a definitive negative
extraction_failed
Reference resolved but document unreadable (scanned PDF, parser error)
Logged for re-ingestion
ambiguous(candidates)
Fuzzy reference matched multiple documents
Requires explicit disambiguation; top-hit auto-selection is prohibited
out_of_scope
Document exists in the full corpus but outside the active sub-corpus
User informed that widening scope would resolve it
resolution miss
Not in the corpus and not downloadable
Logged as a corpus gap
Table 3
Mechanism
What it does
Tool-call cap (30 per answer)
Hard code limit; on reaching it, control routes to the existing wrap-up synthesis.
Phantom-URL audit
Flags URLs in the answer that appear in no tool result
Phantom-paper audit
Flags cited papers that appear in no tool result
Citation-claim grounding audit
Flags any "X et al." whose surname appears in no tool result from that turn. Matching is word-boundary, not substring, because a substring match would miss a surname embedded inside a longer token in the payload.
Identifier resolution
DOI, PMID, PMCID, and arXiv identifiers resolve through corpus, scholarly APIs, and registries. No language model is in this path; the resolver cannot produce a fabricated identifier.
Explicit null authors
Where metadata cannot be obtained, the result carries an explicit null and a flag. A missing field reads to a model as "not applicable"; an explicit null reads as "unknown, do not guess."
Table 4
Setting
Why it matters
Encoder / collection pairing
Pairing a new encoder with a collection written by a previous one returns empty or incorrect results rather than raising an error. The pairing is asserted at three points: a startup check comparing encoder width against the collection's vector size, a deploy-time re-assertion against the live status endpoint, and code defaults that match production. A mismatch fails the boot.
Query-side embedding prefix
Applied to queries only, with documents embedded raw. Encoder-dependent, and empty for some encoders. Omitting it materially reduces recall.
Served context length
Pinned in both the inference server configuration and the backend token budget, and exported as a shared environment variable before the retrieval container is recreated so the two values cannot drift apart.
Serving concurrency ceiling
Exceeding the inference server's concurrent-sequence limit degrades answer quality silently rather than erroring. Evaluation must run at or below it.
External-slot relevance floor
Calibrated to one encoder over 30 real candidates within an observed similarity band of 0.57 to 0.77. Not a general-purpose threshold: a group using a different encoder or working in a different domain must recalibrate it.
Table 5
Qwen3.6-35B-A3B
Qwen3.8-27B
gpt-oss-20b
Developer
Alibaba
Alibaba
OpenAI
Architecture
MoE, 35B total / 3B active
dense 27B
MoE, 21B total / 3.6B active
Quantisation
AWQ 4-bit, fp8 KV cache
AWQ INT4, fp8 KV cache
native MXFP4
Serving
one card, 65,536-token window
two cards, tensor-parallel, 65,536-token window
one card (24 GB at this window)
Role
production until 2026-08-25; eval-only for these runs
production; headline
third backbone; eval-only
Table 6
Dataset
Size
What it tests
Domain
BEIR / SciFact [5, 6]
300 queries, 5,183 docs
External validity anchor; claim verification
Biomedical
LitSearch [4]
597 queries, 64,183 docs
The closest public analogue to actual usage
ML / NLP
LitQA2 retrieval [1]
199 questions
Retrieval and end-to-end answering on in-corpus questions
Biology, in our corpus
Encoder bake-off pool
190 sources + 5,000 distractors
Encoder comparison on pre-embedded pools
Our domain
Table 7
Set
Construction
Correct behaviour
C1 fabricated papers
100 frozen items: 80 DOIs verified nonexistent against the registry, 20 nonexistent papers described by title and claim, in the group's fields; zero collide with the corpus
Refuse; never cite a local paper
C2b paired shadow corpus
50 single-source questions asked against the live corpus and against a shadow equal to production minus the 49 source papers and their 2,247 chunks
Answer when present; abstain when absent
Table 8
Fig. 2 Accuracy on 199 paired LitQA2 questions across three backbones and four configurations (bare model, naive RAG, harness with local corpus only, full harness with web access). Error bars are 95% Wilson score intervals. Inside the harness, the two Qwen backbones reach nearly identical accuracy (0.869 and 0.874) despite starting from different unaided floors. gpt-oss-20b reaches 0.563, where 162 of 199 naive-RAG turns ended without a final message and are scored as wrong. The dotted line marks the published PaperQA2 accuracy of 0.660. Note that PaperQA2 was trained on the LitQA2 splits, so the comparison favours PaperQA2 relative to the systems reported in this figure.
Arm
Metric
Qwen3.6-35B-A3B
Qwen3.8-27B
gpt-oss-20b
Bare
accuracy
0.337
0.387
0.407
precision (attempted)
0.459
0.403
0.482
abstain
0.226
0.040
0.030
unparseable
8
0
25 (no final message)
RAG (top-5)
accuracy
0.126
0.211
0.101
precision (attempted)
0.500
0.420
0.800
Table 10
Comparison
Qwen3.6 Δ [95% CI]
Qwen3.8 Δ [95% CI]
gpt-oss-20b Δ [95% CI]
agentic - bare
+0.533 [0.452, 0.613]
+0.487 [0.407, 0.568]
+0.156 [0.075, 0.241] , p = 0.004
agentic - RAG
+0.744 [0.683, 0.804]
+0.663 [0.598, 0.729]
+0.462 [0.387, 0.538]
RAG - bare
-0.211 [-0.281, -0.141]
-0.176 [-0.251, -0.096]
-0.306 [-0.377, -0.231]
agentic (egress off) - bare
+0.367 [0.271, 0.457]
+0.276 [0.181, 0.367]
+0.075 [-0.010, +0.161], p = 0.10
agentic (full) - agentic (off)
+0.166 [0.101, 0.226]
+0.211 [0.151, 0.276]
+0.080 [0.015, 0.146], p = 0.02
split: corpus-only loop / external tiers
69 / 31
57 / 43
48 / 52
Table 11
Arm
Qwen3.6
n
Qwen3.8
n
gpt-oss-20b
n
RAG (top-5)
0.290 [0.252, 0.328]
198
0.282 [0.248, 0.316]
199
0.351 [0.208, 0.512]
28
Agentic
0.627 [0.589, 0.660]
199
0.540 [0.503, 0.578]
199
0.392 [0.330, 0.448]
159
Bare
n/a
n/a
n/a
Paired Δ agentic - RAG
+0.336 [0.280, 0.388] , p < 0.001
198
+0.258 [0.206, 0.311] , p < 0.001
199
+0.253 [-0.009, +0.502], p = 0.056
21
Table 12
Fig. 3 Distribution of per-answer faithfulness for naive RAG (a-c) and the full harness (d-f), restricted to the questions both arms answered. Each answer is binned by the fraction of its claims that the judge finds supported by that arm's own retrieved contexts. Answers with zero or all claims supported occupy their own bins. Dashed lines mark the per-arm mean within its bin. The most visible effect is at the left edge of the distributions: the harness reduces the share of answers containing no supported claim from 38% to 3% on Qwen3.6 and from 27% to 7% on Qwen3.8. The hatched gpt-oss-20b panels rest on only 21 paired questions because 162 of its naive-RAG turns ended without a final message and were excluded from the pairing.
Metric
Qwen3.6
Qwen3.8
gpt-oss-20b
Correct-refusal rate
0.980 [0.950, 1.000]
1.000 [0.963, 1.000]
0.720 [0.640, 0.810] (floor, see text)
Confabulated local citations
0 / 100
0 / 100
0 / 100
Verdicts (abstain / possible confabulation / ambiguous)
Scientific question answering develops systems that answer research and domain-science questions using reliable evidence from papers, data, and figures.
Large language models are increasingly used to assist scientific reading, but existing evaluation methods often fail to detect whether answers are supported by verifiable citations. We introduce ResearchQA, a benchmark of 6,211 single-paper question-answer pairs from 494 open-access papers spanning eight domains and four question types: lookup, comprehension, multi-hop, and adversarial. ResearchQA is designed for citation-grounded evaluation: it permits multiple valid supporting passages for a claim and rewards grounded refusal when the source paper does not support an answer. We evaluate eight leading closed- and open-weight models in a citation-grounded chat-with-paper setting using a deterministic citation matcher and an LLM-based rubric evaluator. Citation-based metrics separate systems more clearly than LLM-evaluator scores: section coverage and citation accuracy vary substantially across models, while evaluator scores remain tightly compressed. We further find that open-weight models approach the best closed-model citation accuracy while achieving 3 to 6 times lower per-example latency. We release the benchmark, evaluation harness, and evaluator prompt.
ResearchPilot is an open-source, self-hostable multi-agent system for literature-review assistance. Given a natural-language research question, it retrieves papers from Semantic Scholar and arXiv, extracts structured findings from paper abstracts, synthesizes cross-paper patterns, and drafts a citation-aware related-work section. The system combines FastAPI, Next.js, DSPy, SQLite, and Qdrant in a local-first architecture that supports bring-your-own-key model access and remote-or-local embeddings. This paper describes the system design, typed agent interfaces, persistence and history-search mechanisms, and the engineering tradeoffs involved in building a transparent research assistant. Rather than claiming algorithmic novelty, we present ResearchPilot as a systems contribution and evaluate it through automated tests and end-to-end local runs. We discuss limitations including external API rate limits, abstract-only extraction, incomplete corpus coverage, and the lack of citation verification.
Large language models can answer questions correctly and still hallucinate their citations. Most citation systems address this by retrieving documents at inference time. Retrieval adds latency and depends on external infrastructure. Retrieval noise can also affect the answer. We study whether LLMs can cite documents from their (continual) pretraining without test-time retrieval. We introduce CitePretrainBench to evaluate this setting. It includes Wikipedia, Common Crawl, and arXiv as real-world corpora and also includes novel, unseen documents. The benchmark includes short-form citation tasks for a single fact and long-form tasks for multiple facts. Continual-pretraining first binds factual knowledge to a persistent identifier for each document. Instruction tuning then teaches the model to put these identifiers in its answers as citations. Passive Indexing appends an identifier at the end of each document. Synthetic data used in Active Indexing restate facts in diverse, compositional forms. We train two mappings together. These mappings are source→fact and fact→source. The first direction involves the model generating content given a cited source. In the second, it learns to attribute its own answers when facts are paraphrased or combined. On Qwen-2.5-7B and 3B, Active Indexing improves citation precision over Passive Indexing on every evaluated citation task, with gains of up to 30.2 percentage points. In the scaling ablation, citation performance continues to improve as we add augmented data, including at 16 times the original token count. Combining internal and external citations also improves robustness to retrieval noise.