DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models

Alexander Sheppert

cs.CLMar 5, 2026

DepthCharge: A Domain-Agnostic Framework for Measuring Depth-Dependent Knowledge in Large Language Models

Authors: Alexander Sheppert

Organizations: Legacy Health · Capitol Technology University

Abstract

Large Language Models appear competent when answering general questions but often fail when pushed into domain-specific details. No existing methodology provides an out-of-the-box solution for measuring how deeply LLMs can sustain accurate responses under adaptive follow-up questioning across arbitrary domains. We present DepthCharge, a domain-agnostic framework that measures knowledge depth through three innovations: adaptive probing that generates follow-up questions based on concepts the model actually mentions, on-demand fact verification from authoritative sources, and survival statistics with constant sample sizes at every depth level. The framework can be deployed on any knowledge domain with publicly verifiable facts, without requiring pre-constructed test sets or domain-specific expertise. DepthCharge results are relative to the evaluator model used for answer checking, making the framework a tool for comparative evaluation rather than absolute accuracy certification. Empirical validation across four diverse domains (Medicine, Constitutional Law, Ancient Rome, and Quantum Computing) with five frontier models demonstrates that DepthCharge reveals depth-dependent performance variation hidden by standard benchmarks. Expected Valid Depth (EVD) ranges from 3.45 to 7.55 across model-domain combinations, and model rankings vary substantially by domain, with no single model dominating all areas. Cost-performance analysis further reveals that expensive models do not always achieve deeper knowledge, suggesting that domain-specific evaluation is more informative than aggregate benchmarks for model selection in professional applications.

Figures & tables

Depth	Tier	Pass
1	COMMON	P1
2	COMMON	P2
3	COMMON	P3
4	TEXTBOOK	P1
5	TEXTBOOK	P2
6	TEXTBOOK	P3

Table 1: Mapping from depth to difficulty level with Q=3 passes per tier.

Figure 1: Tree diagram showing adaptive drilling with cumulative survival calculation. Green nodes indicate correct answers (paths continue), red nodes indicate incorrect answers (paths terminate). Cumulative survival at each depth equals the product of all accuracies up to that point, reflecting that errors permanently reduce survival probability.

Figure 2: Example drilling path showing question-answer pairs at increasing depths. D1-D3 (COMMON tier) ask increasingly specific questions about foundational concepts. D4-D6 (TEXTBOOK tier) probe detailed knowledge requiring deeper familiarity with the subject. Each question builds on concepts from the previous answer.

Model	Cost/Domain	Relative Cost
Model D	$0.09	1.0x (baseline)
Model E	$0.61	6.8x
Model A	$0.96	10.7x
Model C	$4.34	48.2x
Model B	$5.80	64.4x

Table 2: Relative costs per model-domain evaluation with N=30 questions per depth.

Model	Domain	EVD	Max D	Acc.	COMMON	TEXT	PROF
A	Medicine	7.55	11	87%	100%	76%	77%
D	Constitutional Law	6.94	11	82%	97%	80%	49%
A	Constitutional Law	6.49	11	82%	97%	84%	49%
D	Quantum Computing	6.21	9	83%	100%	71%	62%
C	Quantum Computing	6.06	9	82%	97%	79%	46%
B	Ancient Rome	5.36	8	84%	100%	68%	40%

Table 3: DepthCharge results across all 20 model-domain combinations sorted by EVD. EVD = Expected Valid Depth. Max D = maximum depth reached. TEXT = TEXTBOOK, PROF = PROFESSIONAL. With N=30 questions per depth, 95% confidence intervals are approximately

\pm

12 percentage points at 50% accuracy (Wilson score intervals).

Figure 3: Comprehensive comparison across all model-domain combinations. Top left: EVD heatmap showing performance variation. Top right: Average EVD rankings. Bottom left: Survival scatter comparing top performers. Bottom right: Value analysis (EVD vs cost).

Observed Accuracy	95% CI Lower	95% CI Upper
50% (15/30)	33%	67%
70% (21/30)	52%	83%
80% (24/30)	62%	91%
90% (27/30)	74%	97%
100% (30/30)	88%	100%

Table 4: 95% Wilson score confidence intervals for accuracy estimates with N=30 questions per depth.

Evaluator Pair	Agreement	Cohen’s $\kappa$
Evaluator 1 vs Evaluator 2	90%	0.77 (substantial)
Evaluator 1 vs Evaluator 3	82%	0.58 (moderate)
Evaluator 3 vs Evaluator 2	82%	0.59 (moderate)
Unanimous (all three)	77%	–

Table 5: Inter-evaluator agreement across three LLM families on 100 sampled question-answer pairs. Positive rates are consistent across evaluators: 68%, 70%, and 66% respectively.

Tier	Errors	Percentage
PROFESSIONAL	187	52%
TEXTBOOK	142	39%
COMMON	32	9%

Table 6: Distribution of incorrect answers by specificity tier across all evaluations.

Questions/Depth	EVD	Max Depth	95% CI Width
10	7.32 $\pm$ 0.6	10	$\pm$ 26pp at 50%
30	7.55 $\pm$ 0.3	11	$\pm$ 17pp at 50%
50	7.61 $\pm$ 0.2	11	$\pm$ 14pp at 50%

Table 7: Effect of questions per depth on evaluation metrics and confidence interval width (pp = percentage points).

Passes/Tier	EVD	Max Depth	Depths per Tier
Q=1	2.51	4	1
Q=3	7.55	11	3
Q=5	12.18	17	5

Table 8: Effect of passes per tier on evaluation depth (Medicine, Model A).

Threshold	EVD	Max Depth	Final Survival
10%	8.12	12	11%
20%	7.55	11	18%
30%	6.89	9	27%

Table 9: Effect of survival threshold on evaluation depth (Medicine, Model A). Lower thresholds allow deeper probing.

Benchmark	Type	A	B	C	D
MMLU ( Hendrycks et al., 2021 )	Broad knowledge	1st	2nd	3rd	4th
GPQA ( Rein et al., 2023 )	Graduate reasoning	1st	2nd	3rd	4th
HumanEval ( Chen et al., 2021 )	Code generation	1st	3rd	4th	2nd
MedQA ( Jin et al., 2021 )	Medical QA	1st	2nd	4th	3rd
DepthCharge (ours)	Knowledge depth	1st	4th	3rd	2nd

Table 10: Model rankings across evaluation paradigms. Breadth-focused benchmarks produce consistent rankings; DepthCharge reveals a different ordering, suggesting depth is a distinct capability dimension.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

May 26, 2026

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.

Luca Giordano, Simon Razniewski

ScaDS.AI Dresden/Leipzig & TU Dresden, Germany

LLM Evaluation Factual Knowledge in Language Models

Ask PDF

Jun 3, 2026

Knowledge Index of Noah's Ark

Knowledge benchmarks for LLMs face three issues: scaling-driven designs that do not operationalize disciplinary representativeness; flat-payment annotation that permits lazy consensus; and unaudited ranking instability under bounded test budgets. We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines, with two formal results. First, we cast representativeness as a coverage-style objective over expert-elicited anchors and operationalize disciplinary representativeness through a proxy, yielding a (1-1/e) greedy approximation (Proposition 1); the guarantee applies to the proxy, not to population representativeness. Second, we prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality, with incentive-compatibility threshold B > Delta C / Delta p_min (Theorem 1). Evaluating 42 models from 13 labs, the top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%, leaving substantial headroom below saturation. The full leaderboard shows a tiered structure rather than a smooth total order: a small frontier tier lies above 48%, a dense strong-model tier spans roughly 38-45%, and low-performing models remain only modestly above the 10% chance baseline. Tool augmentation adds up to 5.17 points across the five tool-use evaluations, with gains varying substantially across models. We report bootstrap ranking-stability statistics to make bounded-budget variance explicit and to discourage over-interpretation of adjacent ranks.

Sheng Jin, Minghao Liu, Yunze Xiao +24

12077AI · 2M-A-P · 3Carnegie Mellon University +7

LLM Evaluation Human-in-the-Loop Annotation Factual Knowledge in Language Models

Ask PDF

Apr 19, 2026

KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models

Many real-world questions appear deceptively simple yet implicitly demand two capabilities: (i) systematic coverage of a bounded knowledge universe and (ii) compositional set-based reasoning over that universe, a phenomenon we term "the tip of the iceberg." We formalize this challenge through two orthogonal dimensions: knowledge width, the cardinality of the required universe, and reasoning depth, the number of compositional set operations. We introduce KnowledgeBerg, a benchmark of 4,800 multiple-choice questions derived from 1,183 enumeration seeds spanning 10 domains and 17 languages, with universes grounded in authoritative sources to ensure reproducibility. Representative open-source LLMs demonstrate severe limitations, achieving only 5.26-36.88 F1 on universe enumeration and 16.00-44.19 accuracy on knowledge-grounded reasoning. Diagnostic analyses reveal three stages of failure: completeness, or missing knowledge; awareness, or failure to identify requirements; and application, or incorrect reasoning execution. This pattern persists across languages and model scales. Although test-time compute and retrieval augmentation yield measurable gains -- up to 4.35 and 3.78 points, respectively -- substantial gaps remain, exposing limitations in how current LLMs organize structured knowledge and execute compositional reasoning over bounded domains. The dataset is available at https://huggingface.co/datasets/2npc/KnowledgeBerg

Xiao Zhang, Qianru Meng, Yongjian Chen +2

University of Groningen · LIACS, Leiden University

LLM Evaluation Compositional Reasoning Structured Reasoning Factual Knowledge in Language Models Knowledge-Intensive QA LLM Reasoning

Ask PDF

Abstract

LLM Evaluation

Explore similar work

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Knowledge Index of Noah's Ark

KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models