cs.CLOct 16, 2026
SaveMuninAI: A Self-Hosted Agentic Framework for Independent Research Groups
Organizations: Institute for Drug Discovery, Leipzig University Medical School, Leipzig, Germany · Institute for Medical Physics and Biophysics, Leipzig University Medical School, Leipzig, Germany · Center for Scalable Data Analytics and Artificial Intelligence (ScaDS.AI), Dresden/Leipzig, Germany · School of Embedded Composite Artificial Intelligence (SECAI), Cooperation of University Leipzig and TU Dresden, Dresden/Leipzig, Germany ·
Abstract
Research groups that want a language model grounded in their own literature face a choice between hosted services that receive their unpublished work and self-hosted deployments that keep the data local but give the model no access to the corpus. Munin is an open-source framework for a self-hosted scientific assistant that indexes a group's corpus and its citation graph, and wraps whichever open-weight model the group can serve in an agent harness with specialised reading, search, and computation tools. We claim that the harness, rather than the specific model, is what makes the system useful on domain literature, and we ship the evaluation suite that tests this claim with the software. On 199 paired literature questions, the harness more than doubles the accuracy of the base model, from 0.387 to 0.874 (+0.487, 95% CI [0.407, 0.568]), while naive retrieve-then-answer scores below the unaided model on every backbone tested. Under the harness, roughly twice as many answer claims trace to a passage the model read itself as with fixed top-5 retrieval. When the source paper for a question is removed from the corpus, the system abstains on about nine in ten of the questions it had answered correctly on the Qwen backbones and eight in ten on gpt-oss-20b, and it produces no fabricated local citation across 300 fabricated references. The pattern holds on three backbones from two developers, including a 20-billion-parameter model that runs on a single consumer-grade GPU. With all outbound network access disabled, both Qwen backbones still exceed the published PaperQA2 accuracy. The evaluation suite runs as a single command, so an adopting group can re-run it on its own corpus and model.
Figures & tables
| Work | Domain | DOI-level ground truth | Withholds real documents from a live corpus | Confabulation axis |
|---|---|---|---|---|
| KnowOrNot [9] | Government policy | No | Leave-one-out on a static corpus | No |
| Sufficient Context [10] | General QA | No | No | No |
| RefusalBench [11] | General QA | No | No (perturbs context) | No |
| AbstentionBench [12] | General | No | Not corpus-relative | No |
| CiteAudit / CiteCheck [17, 18] | Scientific text | Partial | No | Yes |
| This work | Scientific literature | Yes, by construction | Yes (paired shadow corpus) | Yes |
Fig. 1 Deployment boundary and one turn through the harness. All inference, agents, the sandbox, and the corpus live on the cluster's own machines. The gateway reaches them only through a reverse SSH tunnel initiated from the cluster side, so no inbound port is exposed. When a query arrives, a per-turn router decides to which persona it is passed. The bare LLM is also accessible. Inside the harness, the model invokes agents through core tools and tool_search. Source and search agents return compact results tagged with an outcome (resolved, not_found, or thin_evidence), while keeping the full document text in scope. Deep research is available behind a user-facing toggle and writes its report as a saved artifact. After the turn completes, the citation audit runs separately and attaches a warning to the stored answer if any cited reference cannot be verified. Outbound network access is controlled at the egress gate on a per-request basis: off blocks all external calls, oa_only permits scholarly APIs, and full additionally opens web search and fetch. If the agents determine that the source for a question is absent from the corpus, they return not_found. From there, whether to attempt an answer or to decline is the model's own decision, and a decline is emitted over the same streaming path as any other response (Section 5.3).
| Outcome | Meaning | Caller contract |
|---|---|---|
| not_found | Document parsed successfully; answer not present | Treated as a definitive negative |
| extraction_failed | Reference resolved but document unreadable (scanned PDF, parser error) | Logged for re-ingestion |
| ambiguous(candidates) | Fuzzy reference matched multiple documents | Requires explicit disambiguation; top-hit auto-selection is prohibited |
| out_of_scope | Document exists in the full corpus but outside the active sub-corpus | User informed that widening scope would resolve it |
| resolution miss | Not in the corpus and not downloadable | Logged as a corpus gap |
| Mechanism | What it does |
|---|---|
| Tool-call cap (30 per answer) | Hard code limit; on reaching it, control routes to the existing wrap-up synthesis. |
| Phantom-URL audit | Flags URLs in the answer that appear in no tool result |
| Phantom-paper audit | Flags cited papers that appear in no tool result |
| Citation-claim grounding audit | Flags any "X et al." whose surname appears in no tool result from that turn. Matching is word-boundary, not substring, because a substring match would miss a surname embedded inside a longer token in the payload. |
| Identifier resolution | DOI, PMID, PMCID, and arXiv identifiers resolve through corpus, scholarly APIs, and registries. No language model is in this path; the resolver cannot produce a fabricated identifier. |
| Explicit null authors | Where metadata cannot be obtained, the result carries an explicit null and a flag. A missing field reads to a model as "not applicable"; an explicit null reads as "unknown, do not guess." |
| Setting | Why it matters |
|---|---|
| Encoder / collection pairing | Pairing a new encoder with a collection written by a previous one returns empty or incorrect results rather than raising an error. The pairing is asserted at three points: a startup check comparing encoder width against the collection's vector size, a deploy-time re-assertion against the live status endpoint, and code defaults that match production. A mismatch fails the boot. |
| Query-side embedding prefix | Applied to queries only, with documents embedded raw. Encoder-dependent, and empty for some encoders. Omitting it materially reduces recall. |
| Served context length | Pinned in both the inference server configuration and the backend token budget, and exported as a shared environment variable before the retrieval container is recreated so the two values cannot drift apart. |
| Serving concurrency ceiling | Exceeding the inference server's concurrent-sequence limit degrades answer quality silently rather than erroring. Evaluation must run at or below it. |
| External-slot relevance floor | Calibrated to one encoder over 30 real candidates within an observed similarity band of 0.57 to 0.77. Not a general-purpose threshold: a group using a different encoder or working in a different domain must recalibrate it. |
| Qwen3.6-35B-A3B | Qwen3.8-27B | gpt-oss-20b | |
|---|---|---|---|
| Developer | Alibaba | Alibaba | OpenAI |
| Architecture | MoE, 35B total / 3B active | dense 27B | MoE, 21B total / 3.6B active |
| Quantisation | AWQ 4-bit, fp8 KV cache | AWQ INT4, fp8 KV cache | native MXFP4 |
| Serving | one card, 65,536-token window | two cards, tensor-parallel, 65,536-token window | one card (24 GB at this window) |
| Role | production until 2026-08-25; eval-only for these runs | production; headline | third backbone; eval-only |
| Dataset | Size | What it tests | Domain |
|---|---|---|---|
| BEIR / SciFact [5, 6] | 300 queries, 5,183 docs | External validity anchor; claim verification | Biomedical |
| LitSearch [4] | 597 queries, 64,183 docs | The closest public analogue to actual usage | ML / NLP |
| LitQA2 retrieval [1] | 199 questions | Retrieval and end-to-end answering on in-corpus questions | Biology, in our corpus |
| Encoder bake-off pool | 190 sources + 5,000 distractors | Encoder comparison on pre-embedded pools | Our domain |
| Set | Construction | Correct behaviour |
|---|---|---|
| C1 fabricated papers | 100 frozen items: 80 DOIs verified nonexistent against the registry, 20 nonexistent papers described by title and claim, in the group's fields; zero collide with the corpus | Refuse; never cite a local paper |
| C2b paired shadow corpus | 50 single-source questions asked against the live corpus and against a shadow equal to production minus the 49 source papers and their 2,247 chunks | Answer when present; abstain when absent |
Fig. 2 Accuracy on 199 paired LitQA2 questions across three backbones and four configurations (bare model, naive RAG, harness with local corpus only, full harness with web access). Error bars are 95% Wilson score intervals. Inside the harness, the two Qwen backbones reach nearly identical accuracy (0.869 and 0.874) despite starting from different unaided floors. gpt-oss-20b reaches 0.563, where 162 of 199 naive-RAG turns ended without a final message and are scored as wrong. The dotted line marks the published PaperQA2 accuracy of 0.660. Note that PaperQA2 was trained on the LitQA2 splits, so the comparison favours PaperQA2 relative to the systems reported in this figure.
| Arm | Metric | Qwen3.6-35B-A3B | Qwen3.8-27B | gpt-oss-20b |
|---|---|---|---|---|
| Bare | accuracy | 0.337 | 0.387 | 0.407 |
| precision (attempted) | 0.459 | 0.403 | 0.482 | |
| abstain | 0.226 | 0.040 | 0.030 | |
| unparseable | 8 | 0 | 25 (no final message) | |
| RAG (top-5) | accuracy | 0.126 | 0.211 | 0.101 |
| precision (attempted) | 0.500 | 0.420 | 0.800 |
| Comparison | Qwen3.6 Δ [95% CI] | Qwen3.8 Δ [95% CI] | gpt-oss-20b Δ [95% CI] |
|---|---|---|---|
| agentic - bare | +0.533 [0.452, 0.613] | +0.487 [0.407, 0.568] | +0.156 [0.075, 0.241] , p = 0.004 |
| agentic - RAG | +0.744 [0.683, 0.804] | +0.663 [0.598, 0.729] | +0.462 [0.387, 0.538] |
| RAG - bare | -0.211 [-0.281, -0.141] | -0.176 [-0.251, -0.096] | -0.306 [-0.377, -0.231] |
| agentic (egress off) - bare | +0.367 [0.271, 0.457] | +0.276 [0.181, 0.367] | +0.075 [-0.010, +0.161], p = 0.10 |
| agentic (full) - agentic (off) | +0.166 [0.101, 0.226] | +0.211 [0.151, 0.276] | +0.080 [0.015, 0.146], p = 0.02 |
| split: corpus-only loop / external tiers | 69 / 31 | 57 / 43 | 48 / 52 |
| Arm | Qwen3.6 | n | Qwen3.8 | n | gpt-oss-20b | n |
|---|---|---|---|---|---|---|
| RAG (top-5) | 0.290 [0.252, 0.328] | 198 | 0.282 [0.248, 0.316] | 199 | 0.351 [0.208, 0.512] | 28 |
| Agentic | 0.627 [0.589, 0.660] | 199 | 0.540 [0.503, 0.578] | 199 | 0.392 [0.330, 0.448] | 159 |
| Bare | n/a | n/a | n/a | |||
| Paired Δ agentic - RAG | +0.336 [0.280, 0.388] , p < 0.001 | 198 | +0.258 [0.206, 0.311] , p < 0.001 | 199 | +0.253 [-0.009, +0.502], p = 0.056 | 21 |
Fig. 3 Distribution of per-answer faithfulness for naive RAG (a-c) and the full harness (d-f), restricted to the questions both arms answered. Each answer is binned by the fraction of its claims that the judge finds supported by that arm's own retrieved contexts. Answers with zero or all claims supported occupy their own bins. Dashed lines mark the per-arm mean within its bin. The most visible effect is at the left edge of the distributions: the harness reduces the share of answers containing no supported claim from 38% to 3% on Qwen3.6 and from 27% to 7% on Qwen3.8. The hatched gpt-oss-20b panels rest on only 21 paired questions because 162 of its naive-RAG turns ended without a final message and were excluded from the pairing.
| Metric | Qwen3.6 | Qwen3.8 | gpt-oss-20b |
|---|---|---|---|
| Correct-refusal rate | 0.980 [0.950, 1.000] | 1.000 [0.963, 1.000] | 0.720 [0.640, 0.810] (floor, see text) |
| Confabulated local citations | 0 / 100 | 0 / 100 | 0 / 100 |
| Verdicts (abstain / possible confabulation / ambiguous) | 98 / 2 / 0 | 100 / 0 / 0 | 72 / 4 / 24 |
| Mean tool calls per item | 10.3 | 5.6 | 9.2 |
| Qwen3.6 present | Qwen3.6 absent | Qwen3.8 present | Qwen3.8 absent | gpt-oss present | gpt-oss absent | |
|---|---|---|---|---|---|---|
| Accuracy | 0.640 | 0.040 | 0.540 | 0.040 | 0.400 | 0.040 |
| Abstain rate | 0.300 | 0.860 | 0.420 | 0.920 | 0.340 | 0.800 |
| Accuracy drop on removal | -0.600 | -0.500 | -0.360 |
| Outcome | Qwen3.6 (n = 32) | Qwen3.8 (n = 27) | gpt-oss-20b (n = 20) |
|---|---|---|---|
| Correct abstention | 0.875 [0.750, 0.969] | 0.889 [0.777, 1.000] | 0.800 [0.600, 0.950] |
| Answered, still correct | 0.031 | 0.074 | 0.100 |
| Answered, now wrong | 0.094 | 0.037 | 0.000 |
| Not classified by the scorer | 0.000 | 0.000 | 0.100 |
| Population | Arm | Desired | Qwen3.6 | Qwen3.8 | gpt-oss-20b | Egress |
|---|---|---|---|---|---|---|
| litqa2-answerable | bare | answer | 0.734 / 0.541 | 0.960 / 0.597 | 0.844 / 0.518 | n/a |
| litqa2-answerable | RAG | answer | 0.251 / 0.500 | 0.502 / 0.580 | 0.126 / 0.200 | n/a |
| litqa2-answerable | agentic | answer | 0.935 / 0.070 | 0.925 / 0.054 | 0.628 / 0.104 | full |
| c2-present | agentic | answer | 0.700 / 0.086 | 0.580 / 0.069 | 0.500 / 0.200 | off |
| c2-absent | agentic | abstain | 0.140 / 0.714 | 0.080 / 0.500 | 0.060 / 0.333 | off |
| c1-fabricated | agentic | abstain | 0.020 / n.m. | 0.000 / n.m. | 0.280 / n.m. | full |
| Qwen3.6-35B-A3B | Qwen3.8-27B | gpt-oss-20b | |
|---|---|---|---|
| Lab / class | Alibaba, MoE 35B / 3B active | Alibaba, dense 27B | OpenAI, MoE 21B / 3.6B active |
| Bare accuracy | 0.337 | 0.387 | 0.407 |
| Agentic accuracy | 0.869 | 0.874 | 0.563 |
| Harness value | +0.533 [0.452, 0.613] | +0.487 [0.407, 0.568] | +0.156 [0.075, 0.241] |
| Corpus-only harness value (egress off) | +0.367 | +0.276 | +0.075 (n.s.) |
| Abstain outside the harness (bare / RAG) | 0.226 / 0.749 | 0.040 / 0.498 | 0.030 / 0.060, plus empty turns |
| Retriever | nDCG@10 [95% CI] |
|---|---|
| BM25 | 0.652 [0.61, 0.70] |
| SPECTER-dense | 0.479 [0.43, 0.53] |
| RRF [BM25, SPECTER] | 0.621 [0.58, 0.67] |
| SPECTER-dense, canonical [SEP] format | 0.494 (interval not recorded) |
| Retriever | nDCG@10 [95% CI] | R@10 | R@100 | MRR |
|---|---|---|---|---|
| BM25 | 0.378 [0.345, 0.413] | 0.511 | 0.699 | 0.349 |
| BGE-dense (production) | 0.485 [0.453, 0.516] | 0.637 | 0.829 | 0.451 |
| Citation re-rank 0.7/0.3 | 0.469 [0.437, 0.503] | 0.609 | 0.829 | 0.442 |
| RRF [BM25, BGE] | 0.490 [0.455, 0.526] | 0.628 | 0.830 | 0.462 |
| Qwen3.6 | Qwen3.8 | gpt-oss-20b | |
|---|---|---|---|
| Tool calls, total | 1,458 | 1,380 | 1,742 |
| Calls per query | 7.33 | 6.93 | 8.75 |
| Queries with at least one tool failure | 84 | 102 | 58 |
| Recovery rate | 1.000 | 1.000 | 1.000 |
| Wall-clock per query, agentic, egress = full | 67.0 s | 157.3 s | 29.3 s |
| Wall-clock per query, agentic, egress = off | 54.1 s | 96.0 s | 21.3 s |
| System | Accuracy | Precision | Note |
|---|---|---|---|
| PaperQA2 (Skarlinski et al. 2024) | 0.660 ± 0.012 | 0.852 ± 0.011 | trained on LitQA2 splits |
| Human experts (n = 9) | 0.677 ± 0.119 | 0.738 ± 0.096 | |
| Munin, Qwen3.6-35B-A3B | 0.874 [0.824, 0.920] | 0.951 [0.918, 0.978] | ablation arm 0.869 / 0.930 |
| Munin, Qwen3.8-27B | 0.884 [0.839, 0.925] | 0.926 [0.889, 0.963] | ablation arm 0.874 / 0.946 |
| Munin, gpt-oss-20b | 0.528 [0.462, 0.593] | 0.847 [0.774, 0.911] | ablation arm 0.563 / 0.896; 11 empty answers |
| Category | Tools |
|---|---|
| Agents | source, search, compute, deep_research |
| Retrieval | paper_search, semantic_scholar_search, paper_lookup, web_search, web_fetch, search_user_docs, list_documents, browse_tag_papers, check_papers_availability, get_paper_pdf |
| Citation graph | get_citations, get_references, s2_get_citations, s2_get_references, get_author_papers, export_citations |
| LLM utility | llm_summarize, transcribe_equation, view_attachment |
| Artifacts | create_artifact, read_artifact, update_artifact, list_artifacts, save_artifact_to_documents |
| Memory | remember, forget, recall, search_past_conversations |
More in this topic
Scientific QA
Scientific question answering develops systems that answer research and domain-science questions using reliable evidence from papers, data, and figures.
See all 62 papers