Diagnostic Analysis for Large Language Model Reasoning

Xinyu Liu; Dongqi Fu; Jun Wu

cs.CLOct 19, 2026

Diagnostic Analysis for Large Language Model Reasoning

Authors: Xinyu Liu, Dongqi Fu, Jun Wu

Organizations: Michigan State University · Meta AI

Abstract

Reasoning structures (e.g., chains and graphs) are increasingly used for probabilistic inference, factuality verification, and uncertainty estimation, but their structural reliability is rarely evaluated directly. We test whether this reliability holds through a proposed two-stage diagnostic evaluation in open-ended multi-hop QA. Stage 1 tests different capabilities required to construct reasoning graphs, showing that graph reliability varies with topology, prompt design, and model family. Stage 2 uses uncertainty estimation as a downstream test case. Generated-edge experiments show that graph-based computation is sensitive to upstream graph errors, with performance dropping relative to the gold-graph setting. Gold-graph experiments further show that graph structure provides little benefit when used only as prompt context, but improves the separation of correct from incorrect answers by confidence score and enables node-level diagnostics.

Figures & tables

Set.	Given to model	Predicted	Purpose
A	Gold sub-questions, answers, paragraphs	Edges	Dependency wiring
B	Gold sub-questions, paragraphs	Answers, evidence IDs, edges	Local node grounding
C	Question, paragraphs	Full graph	End-to-end generation

Table 1: Three controlled diagnostic settings. Difficulty increases from A to C as gold scaffolding is progressively removed.

			Setting A		Setting B			Setting C
Model	Family	Params	Edge F1	Exact Edge	Edge F1	Ans F1	Attr F1	HPS	Step F1	Edge HPS	Attr F1
Llama-3.1-8B	Llama	8B	0.797 †	0.388 †	0.803	0.386	0.567	0.688	0.679	0.344	0.676
Qwen3-8B	Qwen	8B	0.809 †	0.444 †	0.970	0.504	0.706	0.735	0.777	0.533	0.786
Qwen3-14B	Qwen	14B	0.963	0.913	0.957	0.465	0.763	0.801	0.837	0.567	0.812
Qwen3-32B	Qwen	32B	0.985	0.961	0.970	0.419	0.795	0.863	0.857	0.705	0.831
Qwen3-32B-Reasoning	Qwen	32B	0.992	0.976	0.973	0.579	0.821	0.878	0.894	0.745	0.850

Table 2: Stage 1 results on MuSiQue full dev. Bold marks the best completed value in each column. † Subject to schema-example artifact in the edge-only setting; see Appendix C.1 . ‡ DeepSeek V4-Flash is a mixture-of-experts model with 284B total parameters and 13B activated parameters.

Slice	N	Llama-3.1-70B			Qwen3-32B			Claude Sonnet 4.5
		HPS	Edge HPS	$\boldsymbol{\Delta}$ Step	HPS	Edge HPS	$\boldsymbol{\Delta}$ Step	HPS	Edge HPS	$\boldsymbol{\Delta}$ Step
2-hop	1252	0.918	0.780	0.049	0.921	0.820	0.133	0.973	0.920	0.159
3-hop	760	0.716	0.480	-0.622	0.842	0.638	-0.041	0.962	0.882	0.217
4-hop	405	0.590	0.340	-1.062	0.721	0.478	-0.397	0.917	0.814	0.165
simple_tree	1820	0.855	0.698	-0.162	0.897	0.780	0.069	0.968	0.910	0.169
v_structure	351	0.683	0.350	-0.723	0.792	0.454	-0.108	0.957	0.836	0.393

Table 3: Setting C end-to-end graph generation performance stratified by hop count and topology slices.

Tag	Description
Setting A:edge wiring failures
miss. terminal	Gold dependency into $q_{\mathrm{final}}$ is omitted.
shortcut repl.	Gold dependency replaced by a shortcut edge, often into $q_{\mathrm{final}}$ .
broad mixed	Multiple answers, evidence, and edge failures with no dominant cause.
Setting B:local grounding failures
ans.+evid.	Both generated answer and evidence attribution fail at a fixed node.

Table 4: Error tags appearing in Table 5 .

Model	Setting A	Setting B	Setting C
Llama-3.1-8B	miss. terminal 45.7%	broad mixed 35.0%	struct. gap 37.9%
Qwen3-8B	miss. terminal 46.5%	ans.+evid. 40.7%	under-decomp. 38.6%
Qwen3-14B	shortcut repl. 7.1%	ans.+evid. 34.0%	under-decomp. 33.7%
Qwen3-32B	shortcut repl. 3.2%	ans.+evid. 30.2%	struct. gap 19.5%
Qwen3-32B-Reasoning	shortcut repl. 1.8%	ans.+evid. 25.9%	under-decomp. 20.7%
Llama-3.1-70B	shortcut repl. 19.2%	ans.+evid. 23.7%	under-decomp. 34.7%

Table 5: Top non-aligned error type and rate per model per diagnostic setting (MuSiQue full dev set). Each entry is the most frequent error category excluding fully_aligned .

Method	EM $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$
Direct-SC	0.462	0.569	0.718	0.325	0.319
CoT-Conf	0.487	0.593	0.686	0.271	0.297
CoT-SC	0.572	0.680	0.766	0.161	0.220
Decomp-SC	0.550	0.659	0.770	0.234	0.247
Graph-CoT-SC	0.562	0.671	0.764	0.238	0.251
vPGM-Style	0.439	0.550	0.622	0.235	0.295

Table 6: Stage 2 confidence estimation on MuSiQue full dev set.

Method	N	EM			F1			AUROC			ECE $\downarrow$			Brier $\downarrow$
		Gen	Gold	$\boldsymbol{\Delta}$	Gen	Gold	$\boldsymbol{\Delta}$	Gen	Gold	$\boldsymbol{\Delta}$	Gen	Gold	$\boldsymbol{\Delta}$	Gen	Gold	$\boldsymbol{\Delta}$
Graph-CoT-SC	446	0.476	0.469	+0.007	0.620	0.622	-0.002	0.689	0.714	-0.025	0.313	0.324	-0.011	0.326	0.322	+0.004
Sampled BN	446	0.459	0.493	-0.034	0.599	0.657	-0.058	0.738	0.755	-0.017	0.300	0.283	+0.017	0.300	0.285	+0.015

Table 7: Generated-edge stress test on valid non-exact Setting A graphs (446 examples). Gen: performance under Llama-3.1-70B generated edges. Gold: performance using gold edges on the same examples.

\Delta

= Gen

-

Gold.

Figure 1: Confidence distributions for CoT-SC, vPGM-Style, and Sampled BN, separated by final-answer correctness.

Slice	F1		AUROC
	CoT-SC	Sampled BN	CoT-SC	Sampled BN
2-hop	0.717	0.742	0.753	0.795
3-hop	0.673	0.706	0.770	0.816
4-hop	0.577	0.499	0.780	0.752
simple_tree	0.713	0.737	0.761	0.789
v_structure	0.519	0.555	0.761	0.790

Table 8: Structure-stratified F1 and AUROC for CoT-SC and Sampled BN. Bold marks the higher value per row.

Appendix figures & tables24 assets

Supplementary material from the paper’s appendix.

Appendix

Model	Setting	Parse OK	Schema OK
Llama-3.1-8B	A	1.000	1.000
	B	0.998	0.998
	C	0.990	0.990
Llama-3.1-70B	A	1.000	1.000
	B	0.996	0.996
	C	0.995	0.995

Appendix

Table 9: Parse and schema validity across models and settings (2,417 examples per run).

Model	Var.	N	Parse OK	Schema OK	Disconn.	Edge F1	Exact Edge	Shortcut
Qwen3-8B	E1	2417	0.998	0.998	0.472	0.809	0.444	0.058
	E2	2417	0.998	0.998	0.260	0.856	0.611	0.097
	E3	2417	0.998	0.998	0.013	0.909	0.793	0.153
	E4	2417	0.998	0.998	0.008	0.902	0.785	0.169
	E5	2417	0.998	0.998	0.281	0.836	0.550	0.126
Qwen3-32B	E1	2417	0.999	0.999	0.003	0.985	0.961	0.028

Appendix

Table 10: Full Setting A schema-example ablation across model families on MuSiQue full development set (2,417 examples per model/variant). Variants differ only in the output-schema example; all other prompt text, dataset, metrics, and inference code are unchanged. E1 is the variant used in main experiments (Table 2 ).

Decoding	Edge F1	Exact Edge	Shortcut
Non-thinking (main)	0.985	0.961	0.028
Thinking-on	0.992	0.976	0.018

Appendix

Table 11: Qwen3-32B thinking-on vs. non-thinking under Setting A (2,417 examples). Thinking mode yields a further 0.7 pp Edge F1 gain and 1 pp shortcut reduction.

Model	Edge HPS	Strict Edge HPS
Llama-3.1-70B	0.612	0.597
Qwen3-32B	0.705	0.672
Claude Sonnet 4.5	0.890	0.841

Appendix

Table 12: Setting C Edge HPS under the main reachability-based criterion and a strict direct-edge criterion on the MuSiQue full development set.

Backbone	$\boldsymbol{\tau=0.75}$	$\boldsymbol{\tau=0.85}$	$\boldsymbol{\tau=0.90}$	$\boldsymbol{\tau=0.95}$
e5-large-v2	0.913	0.836	0.714	0.504
MiniLM-L6-v2	0.703	0.523	0.378	0.176
BGE-large-v1.5	0.787	0.621	0.474	0.230

Appendix

Table 13: Mean Setting C HPS across the nine main-table models under different semantic-alignment thresholds and embedding backbones.

Figure 2: Schematic error modes across diagnostic settings. Red solid edges indicate incorrect dependencies, red dashed edges or nodes indicate missing structure, and shaded nodes indicate incorrect local payloads.

Tag	Operational Detection Criterion
fully_aligned	Predicted nodes, edges, answers, and evidence all match the gold graph under the setting-specific evaluator.
shortcut_replacement	A gold dependency is missing and a direct or transitive shortcut edge connects an earlier prerequisite to a later node, often q_final .
spurious_final_shortcut	A non-parent node has an extra direct edge into q_final .
spurious_internal_shortcut	A non-parent node has an extra shortcut edge into an internal node.
mixed_shortcut	Final and internal shortcut behavior appear in the same fixed-node output.
missing_terminal_edge	A gold dependency into q_final is omitted.

Appendix

Table 14: Stage 1 diagnostic failure taxonomy. Tags are automatic labels over the evaluated graph outputs, intended to localize failure modes within each diagnostic setting.

Model	Full	Shortcut repl.	Missing term.	Missing gold	Other edge	Final shortcut	Internal shortcut	Mixed shortcut	Parse fail.
Claude Sonnet 4.5	2382	32	0	2	0	1	0	0	0
DeepSeek V4 Flash	2372	45	0	0	0	0	0	0	0
GPT-5 Chat	2337	67	0	0	2	10	0	1	0
Llama-3.1-70B	1880	465	3	48	8	9	4	0	0
Llama-3.1-8B	937	89	1105	23	245	5	0	13	0
Qwen3-8B	1071	173	1123	11	20	7	0	6	6

Appendix

Table 15: Full-dev Setting A error counts by model (2,417 examples). Full denotes fully_aligned . Rates are reported in the source analysis; this table gives counts to avoid rounding ambiguity.

Model	Full	Ans+Ev	Ev only	Ans only	Interm. ans	Edge only	Broad	Ans+Edge	Edge+Ev	Subq drift	Final synth	Final+Ev	Final+Edge	Final mixed	Parse	Schema
Claude Sonnet 4.5	1195	449	390	145	142	21	9	8	5	36	7	6	4	0	0	0
DeepSeek V4 Flash	1018	591	371	170	168	24	21	25	9	4	7	4	3	0	2	0
GPT-5 Chat	839	584	498	177	172	28	29	28	12	0	23	19	6	2	0	0
Llama-3.1-70B	756	572	290	160	108	152	157	101	38	54	3	6	10	1	9	0
Llama-3.1-8B	54	630	60	37	58	210	847	175	144	168	6	12	9	2	5	0
Qwen3-8B	543	983	317	183	130	15	111	42	9	51	7	4	2	1	16	3

Appendix

Table 16: Full-dev Setting B error counts by model (2,417 examples). Full denotes fully_aligned ; Ans+Ev denotes answer_and_evidence ; Interm. ans denotes intermediate_answer_only_recovered_final .

Model	Full	Under	Structure gap	Evidence gap	Over	Extra benign	Minor mixed	Parse fail.	Schema fail.
Claude Sonnet 4.5	1172	99	228	452	221	226	18	1	0
DeepSeek V4 Flash	900	506	393	306	164	91	19	9	29
GPT-5 Chat	873	153	423	421	291	221	21	0	14
Llama-3.1-70B	701	838	454	278	72	30	33	11	0
Llama-3.1-8B	134	639	917	203	408	62	31	23	0
Qwen3-8B	602	932	424	354	30	11	39	22	3

Appendix

Table 17: Full-dev Setting C error counts by model (2,417 examples). Full denotes fully_aligned ; Under and Over denote under_decompose and over_decompose .

Model	Hop	Topology	N	HPS	Edge HPS	Evidence HPS	$\boldsymbol{\Delta}$ Step
Llama-3.1-70B	3	simple_tree	568	0.715	0.517	0.667	-0.626
Llama-3.1-70B	3	v_structure	192	0.717	0.370	0.568	-0.609
Llama-3.1-70B	4	v_structure	159	0.642	0.326	0.566	-0.861
Llama-3.1-70B	4	complex_tree	246	0.556	0.349	0.535	-1.193
Qwen3-32B	3	simple_tree	568	0.842	0.692	0.742	-0.073
Qwen3-32B	3	v_structure	192	0.842	0.477	0.696	0.052

Appendix

Table 18: Within-hop Setting C graph generation performance by topology on MuSiQue. MuSiQue does not contain all hop–topology combinations: 2-hop examples are all simple_tree, 3-hop examples compare simple_tree and v_structure, and 4-hop examples compare v_structure and complex_tree.

\Delta

Step is predicted minus gold node count; negative values indicate under-decomposition.

			Setting A		Setting B			Setting C
Model	Family	Params	Edge F1	Exact Edge	Edge F1	Ans F1	Attr F1	HPS	Step F1	Edge HPS	Attr F1
Llama-3.1-8B	Llama	8B	0.803	0.402	0.863	0.665	0.643	0.818	0.727	0.361	0.769
Qwen3-8B	Qwen	8B	0.923	0.774	0.863	0.807	0.724	0.866	0.903	0.722	0.901
Qwen3-14B	Qwen	14B	0.972	0.952	0.799	0.723	0.756	0.956	0.960	0.784	0.920
Qwen3-32B	Qwen	32B	0.988	0.976	0.932	0.691	0.765	0.981	0.967	0.923	0.939
Llama-3.1-70B	Llama	70B	0.823	0.573	0.748	0.859	0.707	0.848	0.862	0.641	0.797

Appendix

Table 19: 2WikiMultiHopQA Stage 1 diagnostics on 498 stratified development examples, five models, all three settings. Bold marks the best value per column. Ans F1 in Setting B is token-level F1 over intermediate sub-answers under the fixed-decomposition interface. Metric names follow the main Stage 1 table (Table 2 ).

Question Type	N	Edge HPS $\uparrow$
inference †	80	0.634
compositional †	80	0.634
comparison	170	0.976
bridge_comparison	168	0.308

Appendix

Table 20: 2WikiMultiHopQA Stage 1 Setting C Edge HPS by official question type (498 examples, Llama-3.1-70B). Bold marks the highest. † Inference and compositional are both chain-shaped types; the Edge HPS value is reported for both combined. Comparison questions involve two independent fact retrievals merged at a comparison step and reach near-ceiling. Bridge_comparison questions require 4-step parallel chain merging and produce the lowest Edge HPS.

Method	Calls / example	Completions / example	Full-dev calls
Direct-SC	1	10	2,417
CoT-SC	1	10	2,417
Decomp-SC	1	10	2,417
Graph-CoT-SC	1	10	2,417
vPGM-Style	1	10	2,417
CoT-Conf	1	1	2,417

Appendix

Table 21: API call budget for Stage 2 methods on MuSiQue full development set (2,417 examples). SC-family runners use batch sampling (

K{=}10

per request); Sampled BN cost scales with graph structure.

Model	Mean	Median	Max
Qwen3-32B	3.568	3	71
Llama-3.1-70B	4.098	3	54

Appendix

Table 22: Sampled BN API calls per example on MuSiQue full dev.

Model	2-hop	3-hop	4-hop
Qwen3-32B	2.280	3.974	6.788
Llama-3.1-70B	2.568	4.546	7.990

Appendix

Table 23: Mean Sampled BN API calls per example by hop count.

Figure 3: Selective prediction curves for Stage 2 methods.

	CoT-SC Correct	CoT-SC Wrong
Sampled BN Correct	1117 (46.2%)	255 (10.5%)
Sampled BN Wrong	266 (11.0%)	779 (32.2%)

Appendix

Table 24: Pairwise correctness contingency between Sampled BN and CoT-SC on MuSiQue full development set (2,417 examples;

\kappa{=}0.560

). The near-symmetric one-method-only split (10.5% BN-only correct vs. 11.0% CoT-SC-only correct) indicates complementary rather than redundant error patterns.

$\alpha$	EM	F1	AUROC	ECE $\downarrow$	Brier $\downarrow$	High-conf. wrong $\downarrow$
0.0	0.568	0.691	0.795	0.206	0.225	11.0%
0.1	0.568	0.690	0.795	0.194	0.221	8.0%
0.5	0.569	0.692	0.790	0.168	0.212	7.3%
1.0	0.568	0.690	0.790	0.147	0.207	7.0%

Appendix

Table 25: Sensitivity of Sampled BN to additive CPT smoothing on the Llama-3.1-70B MuSiQue gold-graph setting. High-conf. wrong denotes examples with an incorrect final prediction and confidence

\geq 0.9

Comparison	Metric	Difference	$p$ -value
Decomp-SC vs. CoT-SC	AUROC	$+0.004$	$0.79$
Graph-CoT-SC vs. CoT-SC	AUROC	$-0.002$	$0.82$
Sampled BN vs. CoT-SC	AUROC	$+0.029$	$0.017$
Sampled BN vs. CoT-SC	F1	$+0.011$	$0.16$
Sampled BN vs. CoT-SC, 4-hop	F1	$-7.7$ pp	$<0.001$
Sampled BN vs. CoT-SC, complex_tree	F1	$-12.3$ pp	$<0.001$

Appendix

Table 26: Paired bootstrap significance tests for the main comparative claims in Stage 2. Differences are computed as the first condition minus the second.

Method	Hop	$N$	Ans. EM $\uparrow$	Ans. F1 $\uparrow$	AUROC $\uparrow$
Direct-SC	2	1252	0.491	0.602	0.717
CoT-Conf	2	1252	0.522	0.636	0.688
CoT-SC	2	1252	0.611	0.717	0.753
Decomp-SC	2	1252	0.601	0.713	0.763
Graph-CoT-SC	2	1252	0.617	0.727	0.762
vPGM-Style	2	1252	0.466	0.587	0.608

Appendix

Table 27: Hop-stratified performance on MuSiQue full development set (2,417 examples, Llama-3.1-70B,

K{=}10

). Bold marks per-hop per-column best. Sampled BN leads on 2-hop and 3-hop but drops to 0.393 EM on 4-hop, where CoT-SC retains 0.504 EM.

Method	Topology	$N$	Ans. EM $\uparrow$	Ans. F1 $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$	Brier $\downarrow$
Direct-SC	simple_tree	1820	0.491	0.599	0.718	0.305	0.306
CoT-Conf	simple_tree	1820	0.518	0.628	0.671	0.257	0.289
CoT-SC	simple_tree	1820	0.597	0.713	0.761	0.160	0.220
Decomp-SC	simple_tree	1820	0.587	0.697	0.752	0.221	0.245
Graph-CoT-SC	simple_tree	1820	0.599	0.711	0.757	0.224	0.242
vPGM-Style	simple_tree	1820	0.465	0.577	0.609	0.233	0.298

Appendix

Table 28: Topology-stratified Answer EM, F1, and EM-based calibration metrics on MuSiQue full dev. Bold marks per-topology per-column best. Sampled BN leads on simple_tree and v_structure for answer quality, while CoT-SC remains strongest on complex_tree .

	Qwen3-32B				Llama-3.1-70B
Method	EM $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$	EM $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$
Direct-SC	0.425	0.540	0.678	0.429	0.462	0.569	0.718	0.325
CoT-SC	0.537	0.654	0.776	0.176	0.572	0.680	0.766	0.161
CoT-Conf	0.386	0.507	0.672	0.432	0.487	0.593	0.686	0.271
Decomp-SC	0.534	0.653	0.722	0.327	0.550	0.659	0.770	0.234
Graph-CoT-SC	0.536	0.656	0.717	0.331	0.562	0.671	0.764	0.238

Appendix

Table 29: Cross-model Stage 2 comparison on MuSiQue full dev (

K{=}10

, temperature 0.7; CoT-Conf uses one temperature-0 call). Bold marks per-model per-column best.

	Llama-3.1-70B				Qwen3-32B
Method	EM $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$	EM $\uparrow$	F1 $\uparrow$	AUROC $\uparrow$	ECE $\downarrow$
Direct-SC	0.647	0.698	0.668	0.244	0.582	0.653	0.619	0.359
CoT-SC	0.843	0.894	0.715	0.076	0.821	0.883	0.723	0.118
CoT-Conf	0.801	0.854	0.721	0.134	0.769	0.854	0.636	0.165
Decomp-SC	0.799	0.853	0.715	0.139	0.803	0.859	0.627	0.172
Graph-CoT-SC	0.805	0.857	0.730	0.133	0.797	0.851	0.621	0.183

Appendix

Table 30: 2WikiMultiHopQA Stage 2 diagnostic on the same 498-example stratified sample used in Stage 1 evaluation. Bold marks per-model best values.

Language Model Reasoning Evaluation

See all 93 papers

Argument Graph Uncertainty: Quantifying Uncertainty from the Logical Structure of Reasoning Chains
Oct 14, 2026Simon Baur, Sumire Honda, Wojciech Samek +1LLM Uncertainty Estimation Uncertainty Calibration
Fraunhofer Heinrich-Hertz-Institut 10587 Berlin, Germany
Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework
Jul 9, 2026Riccardo Revalor, Jalees Rehman, Debjit PalCoT Faithfulness LLM Reasoning with Graphs
¶Department of Electrical and Computer Engineering · ‡Department of Biochemistry and Molecular Genetics
LCoT-GV: Graph Attention Networks for Verifying Long Reasoning Chains in Large Language Models
Aug 31, 2026Bérénice Jaulmes, Mehwish AlamLLM Reasoning with Graphs LLM Reasoning
1T´el´ecom Paris, Institut Polytechnique de Paris, France · 2BNP Paribas, IT Group, France

Oct 14, 2026J/K move · Enter open · S save

Argument Graph Uncertainty: Quantifying Uncertainty from the Logical Structure of Reasoning Chains

Simon Baur, Sumire Honda, Wojciech Samek, Jackie Ma

Fraunhofer Heinrich-Hertz-Institut 10587 Berlin, Germany

LLM Uncertainty Estimation Uncertainty Calibration

Open paper AskPDF arXiv

Reliable uncertainty estimation is essential for deploying large language models in high-stakes reasoning tasks, yet most existing approaches rely on expensive sampling strategies or shallow token-level signals that ignore the internal logical structure of model outputs. We introduce Argument Graph Uncertainty, a post-hoc framework for multiple choice question tasks, that estimates model confidence directly from the logical structure of a single chain-of-thought reasoning trace. Our method uses either a lightweight instruction-tuned model or a simple deterministic rule to segment the reasoning chain into propositional units, then applies a NLI model to infer relational edges and construct two complementary argument graphs over these segments. The resulting edge scores are aggregated into a Dirichlet posterior over answer options, whose concentration yields calibrated uncertainty estimates reflecting both local step-to-step coherence and global logical consistency across the reasoning trace. Evaluated on GPQA, MMLU-Pro and TruthfulQA, our method consistently surpasses token-based and consistency-based uncertainty baselines across multiple reasoning models. We further demonstrate through extensive combination analyses that argument graph uncertainty captures an orthogonal signal to existing approaches, making it a complementary component in uncertainty ensembles. Crucially, our framework requires no additional calls to the evaluated model and relies entirely on small, efficient auxiliary models, producing fully interpretable, per-node uncertainty attributions at low computational cost.

Abstract

Language Model Reasoning Evaluation

Explore similar work

Argument Graph Uncertainty: Quantifying Uncertainty from the Logical Structure of Reasoning Chains

Can We Trust LLM's Logic? Quantifying Uncertainty, Coherence, and Robustness via a Graph-Based Framework

LCoT-GV: Graph Attention Networks for Verifying Long Reasoning Chains in Large Language Models