We investigate how provision of in-context examples based on discourse similarity affects quality and hallucination in diagram code generation from educational materials. Using a sample of texts as sources for diagram generation, we test two methods using a conventional text-to-diagram task format, instruction-based zero-shot generation and ICL with text-to-diagram examples, and an unconventional ICL method with a discourse analysis-to-diagram example, which requires a model to generate a diagram from a discourse analysis. We evaluate AI-generated diagrams for faithfulness to provided context and their quality outcomes using both human and automated evaluation. Bayesian data analysis of the human diagram evaluation shows that there is significant evidence that the unconventional task format improves context faithfulness in diagram generation compared to the zero-shot instruction with reducing the odds of context faithfulness hallucination by nearly 70%; however, we do not find the ICL having a significant effect on the diagram aesthetic presentation and superficial coherence. Evaluation with the LLM-as-a-judge G-Eval metric shows nearly 4.5-fold increased odds of the perfect faithfulness outcome for the novel ICL task. Our study provides a novel insight into ICL format effect on superficial fluency and faithfulness in context-grounded structured diagram code generation.
Figures & tables
C1
C2
C3
IRR
α^= 0.83 (0.75, 0.89)
α^= 0.68 (0.44, 0.82)
α^= 0.60 (0.46, 0.71)
W= 0.91 (<0.01)
W= 0.79 (<0.01)
W= 0.82 (<0.01)
Table 1: Human Evaluation Results for Logical Organization C1 , Connectivity C2 , Layout Aesthetic C3 . Q1, Q2, and Q3 stand for quartiles. Krippendorff’s α^ with 95% CI and Kendall’s W with p− values are provided for IRR.
GPT-4o
o3
0-shot
RST1
RST2
0-shot
RST1
RST2
Ffact
0.2
0.2
0.04
0.12
0.04
0.04
Fc
0.68
0.52
0.48
0.6
0.36
0.28
G
0.64
0.6
0.4
0.76
0.84
0.8
Table 2: Hallucination occurrence and the ratio of G (high quality diagrams).
Figure 1: Distribution and correlation between BERTScore, G-Eval, ROUGE-L, Faithfulness metric. Asterisks denote statistical significance.
Figure 2: Distributions of BERTScore, ROUGE-l, G-Eval, Faithfulness Metric per each method.
model
reasoning
method
G-Eval
Faithfulness
BERTS
ROUGE-L
GPT-4.1
-
0-SHOT
0.89
1.00
0.25
0.37
GPT-4.1
-
RST1
0.94
0.98
0.30
0.46
GPT-4.1
-
RST2
0.96
1.00
0.46
0.65
GPT-4.1-mini
-
0-SHOT
0.89
0.97
0.24
0.40
GPT-4.1-mini
-
RST1
0.93
0.99
0.35
0.51
GPT-4.1-mini
-
RST2
0.96
1.00
0.54
0.75
Table 3: Mean score results across the model configurations
Figure 3: Mean and median of G-Eval scores across the model configurations per each method.
Figure 4: Mean and median of ROUGE-L F1 scores across the model configurations per each method.
Figure 5: Mean and median of BERTScore (F1) across the model configurations per each method.
Figure 6: Results of Bayesian Data Analysis: Odds Ratios for Hallucination and Quality Outcomes. (1) stands for posterior estimates obtained with Pr1a , (2) for Pr2a .
Figure 7: Predicted G-Eval scores for the model configurations and the methods with Pr1b .
Figure 8: Predicted G-Eval scores for the model configurations and the methods with Pr2b .
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Diagram 1 for Text 1
Figure 10: Diagram 2 for Text 2
Figure 11: Diagram 3 for Text 3
Figure 12: Diagram 4 for Text 4
Figure 13: Posterior predictive checks for factual hallucination Ffact , context inconsistency Fc , quality outcome G .
Figure 14: Posterior density and trace plots.
Fit
max R^
min bulk ESS
divergent C
Fc (1)
1.01
1015.31
0
Fc (2)
1.01
955.96
0
Ffact (1)
1.00
872.10
0
Ffact (2)
1.01
684.24
0
G (1)
1.00
933.78
0
G (2)
1.00
878.40
0
Appendix
Table 4: Convergence diagnostics for the Brms fits. Ffact stands for factual hallucination, Fc — context faithfulness, G —quality outcome. (1) is a fit with Pr1 , (2) — Pr2
Fit
ELPDLOO
SEELPD
PLOO
Rbayes2
Pareto k^
Ffact (1)
-47.07
7.60
13.11
0.21
0
Ffact (2)
-47.29
7.48
12.35
0.19
0
Fc (1)
-93.50
5.27
18.75
0.27
0
Fc (2)
-93.89
4.83
18.27
0.25
0
G (1)
-87.79
5.99
16.43
0.23
0
G (2)
-87.58
5.71
15.65
0.22
0
Appendix
Table 5: Diagnostics for the Brms fits. Ffact stands for factual hallucination, Fc — context faithfulness, G —quality outcome. (1) is a fit with Pr1 , (2) — Pr2
Effect
Term
Estimate
std.error
conf.low
conf.high
Fit
fixed
(Intercept)
-1.94
0.69
-3.46
-0.75
Ffact (1)
fixed
o3
-1.02
0.61
-2.27
0.15
Ffact (1)
fixed
RST1
-0.32
0.65
-1.61
0.92
Ffact (1)
fixed
RST2
-1.61
0.83
-3.35
-0.06
Ffact (1)
varying
sd(Intercept)
1.47
0.65
0.28
2.95
Ffact (1)
fixed
(Intercept)
-2.04
0.64
-3.43
-0.94
Ffact (2)
Appendix
Table 6: Posterior Estimates for the Brms fits. Ffact stands for factual hallucination, Fc — context faithfulness, G —quality outcome. (1) is a fit with Pr1 , (2) — Pr2
Figure 15: Posterior predictive checks for G-Eval.
G-Eval(1)
G-Eval(2)
R^
1.00
1.00
min ESS
1403.77
2525.27
div C
0
0
ELPDLOO
433.57
434.79
SEELPD
31.23
31.15
PLOO
59.22
57.63
Appendix
Table 7: Convergence diagnostics for the Brms fits. G-Eval, (1) is a fit with Pr1b , (2) — Pr2b
Figure 16: Posterior density and trace plots for G-Eval.
Effect
Term
Estimate
std.error
conf.low
conf.high
fixed
(Intercept)
2.24
0.16
1.92
2.55
fixed
zoi (Intercept)
-2.79
0.65
-4.11
-1.56
fixed
coi (Intercept)
6.24
1.38
4.14
9.53
fixed
RST1
0.18
0.08
0.02
0.33
fixed
RST2
0.53
0.09
0.36
0.71
fixed
GPT-4.1
0.07
0.14
-0.20
0.33
Appendix
Table 8: Posterior Estimates for the Brms fits for automated assessment.G-Eval—quality outcome. (1) is a fit with Pr1b
Effect
Term
Estimate
std.error
conf.low
conf.high
fixed
(Intercept)
2.26
0.16
1.95
2.57
fixed
zoi (Intercept)
-2.68
0.61
-3.95
-1.53
fixed
coi (Intercept)
6.23
1.37
4.12
9.43
fixed
RST1
0.17
0.08
0.01
0.33
fixed
RST2
0.52
0.09
0.35
0.70
fixed
GPT-4.1
0.06
0.13
-0.20
0.32
Appendix
Table 9: Posterior Estimates for the Brms fits for automated assessment.G-Eval—quality outcome. (1) is a fit with Pr2b
Figure 17: Diagram Generation Pipelines: RST1 and RST2 include Steps 1:5, 0-shot: Steps 4:5.
H
Where
Metric
Type
Description
Ffact
Irefi
{0,1}
factual
Factual fabrication or contradiction
Fae
Irefi
{0,1}
X
Layout issues described in 3
with R3∨R4
the model should avoid and correct
Flog
Irefi
{0,1}
logical
Internal logical inconsistency
inconsistency
Fc
Irefi
{0,1}
X
Deviation from a given context
Appendix
Table 10: Hallucination analysis description. X means incompliance, Ii is a first diagram version, Irefi is its refined version.
GPT-4o
o3
0-shot
RST1
RST2
0-shot
RST1
RST2
Ffact
0.2
0.2
0.04
0.12
0.04
0.04
Fae
0.6
0.72
0.8
0.44
0.44
0.36
Fc
0.68
0.52
0.48
0.6
0.36
0.28
Flog
0.28
0.28
0.48
0.2
0
0.12
Finh
0.28
0.52
-
0.08
0.24
Appendix
Table 11: Hallucination occurrence and the ratio of G (high quality diagrams).
Precision
Recall
F1 Score
Mean
0.9503
0.9393
0.9447
SD
0.0603
0.0838
0.0685
Min
0.7748
0.6484
0.7432
25%
0.9313
0.9054
0.9030
50%
0.9764
0.9752
0.9784
75%
1.0000
1.0000
1.0000
Appendix
Table 12: BERTScore for GPT-5.6 Luna Performance for RST analysis to original text reconstruction
model
reasoning
method
min
q1
q3
SD
mean
q2
max
gpt-4.1
0-SHOT
0.66
0.84
0.92
0.08
0.89
0.90
1.00
gpt-4.1
RST1
0.80
0.90
1.00
0.07
0.94
0.98
1.00
gpt-4.1
RST2
0.78
0.92
1.00
0.06
0.96
1.00
1.00
gpt-4.1-mini
0-SHOT
0.69
0.83
0.98
0.09
0.89
0.87
1.00
gpt-4.1-mini
RST1
0.79
0.87
1.00
0.08
0.93
0.97
1.00
gpt-4.1-mini
RST2
0.76
0.92
1.00
0.06
0.96
0.99
1.00
Appendix
Table 13: G-Eval descriptive results across the model configurations
model
reasoning
method
min
q1
q3
SD
mean
median
max
gpt-4.1
0-SHOT
0.92
1.00
1.00
0.02
1.00
1.00
1.00
gpt-4.1
RST1
0.75
1.00
1.00
0.06
0.98
1.00
1.00
gpt-4.1
RST2
1.00
1.00
1.00
0.00
1.00
1.00
1.00
gpt-4.1-mini
0-SHOT
0.50
1.00
1.00
0.11
0.97
1.00
1.00
gpt-4.1-mini
RST1
0.83
1.00
1.00
0.03
0.99
1.00
1.00
gpt-4.1-mini
RST2
0.90
1.00
1.00
0.02
1.00
1.00
1.00
Appendix
Table 14: Faithfulness descriptive results across the model configurations
model
reasoning
method
min
q1
q3
SD
mean
median
max
gpt-4.1
0-SHOT
0.03
0.19
0.31
0.11
0.25
0.24
0.51
gpt-4.1
RST1
0.10
0.24
0.38
0.11
0.30
0.32
0.50
gpt-4.1
RST2
0.31
0.37
0.55
0.11
0.46
0.45
0.71
gpt-4.1-mini
0-SHOT
0.01
0.16
0.30
0.13
0.24
0.24
0.52
gpt-4.1-mini
RST1
0.07
0.28
0.40
0.12
0.35
0.34
0.57
gpt-4.1-mini
RST2
0.30
0.40
0.65
0.16
0.54
0.50
0.86
Appendix
Table 15: BERTScore F1 descriptive results across the model configurations
model
reasoning
method
min
q1
q3
SD
mean
median
max
gpt-4.1
0-SHOT
0.18
0.29
0.43
0.12
0.37
0.36
0.66
gpt-4.1
RST1
0.19
0.38
0.57
0.13
0.46
0.48
0.69
gpt-4.1
RST2
0.44
0.58
0.73
0.12
0.65
0.68
0.84
gpt-4.1-mini
0-SHOT
0.17
0.32
0.48
0.12
0.40
0.41
0.63
gpt-4.1-mini
RST1
0.27
0.37
0.59
0.14
0.51
0.51
0.77
gpt-4.1-mini
RST2
0.43
0.68
0.86
0.14
0.75
0.76
0.96
Appendix
Table 16: ROUGE-L descriptive results across the model configurations
ids
min
q1
q3
SD
mean
median
max
1
0.80
0.81
0.92
0.07
0.88
0.86
1.00
2
0.87
1.00
1.00
0.03
0.99
1.00
1.00
3
0.80
0.90
0.98
0.06
0.93
0.94
1.00
4
0.90
0.99
1.00
0.03
0.98
1.00
1.00
5
0.84
0.95
1.00
0.05
0.97
1.00
1.00
6
0.91
0.99
1.00
0.02
0.99
1.00
1.00
Appendix
Table 17: G-Eval descriptive results across individual texts
Metric
Value
Accuracy estimate, 95% CI
0.7733 (0.6979, 0.8376)
Kappa
0.4032
No Information Rate (NIR)
0.6733
P-Value [Accuracy > NIR]
0.0048
McNemar’s Test P-Value
0.0000
Sensitivity (Recall)
0.3878
Appendix
Table 18: GPT-5.6 Sol Model Performance Statistics for G : the positive class is 0 .