In-Context Learning for Diagram Code Generation: What is the Effect on Faithfulness and Quality?
Organizations: Aalto University
Abstract
We investigate how provision of in-context examples based on discourse similarity affects quality and hallucination in diagram code generation from educational materials. Using a sample of texts as sources for diagram generation, we test two methods using a conventional text-to-diagram task format, instruction-based zero-shot generation and ICL with text-to-diagram examples, and an unconventional ICL method with a discourse analysis-to-diagram example, which requires a model to generate a diagram from a discourse analysis. We evaluate AI-generated diagrams for faithfulness to provided context and their quality outcomes using both human and automated evaluation. Bayesian data analysis of the human diagram evaluation shows that there is significant evidence that the unconventional task format improves context faithfulness in diagram generation compared to the zero-shot instruction with reducing the odds of context faithfulness hallucination by nearly ; however, we do not find the ICL having a significant effect on the diagram aesthetic presentation and superficial coherence. Evaluation with the LLM-as-a-judge G-Eval metric shows nearly 4.5-fold increased odds of the perfect faithfulness outcome for the novel ICL task. Our study provides a novel insight into ICL format effect on superficial fluency and faithfulness in context-grounded structured diagram code generation.
Figures & tables
| IRR | 0.83 (0.75, 0.89) | 0.68 (0.44, 0.82) | 0.60 (0.46, 0.71) |
|---|---|---|---|
| 0.91 | 0.79 | 0.82 |
| GPT-4o | o3 | |||||
|---|---|---|---|---|---|---|
| 0-shot | RST1 | RST2 | 0-shot | RST1 | RST2 | |
| 0.2 | 0.2 | 0.04 | 0.12 | 0.04 | 0.04 | |
| 0.68 | 0.52 | 0.48 | 0.6 | 0.36 | 0.28 | |
| 0.64 | 0.6 | 0.4 | 0.76 | 0.84 | 0.8 | |
| model | reasoning | method | G-Eval | Faithfulness | BERTS | ROUGE-L |
|---|---|---|---|---|---|---|
| GPT-4.1 | - | 0-SHOT | 0.89 | 1.00 | 0.25 | 0.37 |
| GPT-4.1 | - | RST1 | 0.94 | 0.98 | 0.30 | 0.46 |
| GPT-4.1 | - | RST2 | 0.96 | 1.00 | 0.46 | 0.65 |
| GPT-4.1-mini | - | 0-SHOT | 0.89 | 0.97 | 0.24 | 0.40 |
| GPT-4.1-mini | - | RST1 | 0.93 | 0.99 | 0.35 | 0.51 |
| GPT-4.1-mini | - | RST2 | 0.96 | 1.00 | 0.54 | 0.75 |
Appendix figures & tables24 assets
Supplementary material from the paper’s appendix.
Appendix
| Fit | max | min bulk ESS | divergent C |
|---|---|---|---|
| (1) | 1.01 | 1015.31 | 0 |
| (2) | 1.01 | 955.96 | 0 |
| (1) | 1.00 | 872.10 | 0 |
| (2) | 1.01 | 684.24 | 0 |
| (1) | 1.00 | 933.78 | 0 |
| (2) | 1.00 | 878.40 | 0 |
| Fit | Pareto | ||||
|---|---|---|---|---|---|
| (1) | -47.07 | 7.60 | 13.11 | 0.21 | 0 |
| (2) | -47.29 | 7.48 | 12.35 | 0.19 | 0 |
| (1) | -93.50 | 5.27 | 18.75 | 0.27 | 0 |
| (2) | -93.89 | 4.83 | 18.27 | 0.25 | 0 |
| (1) | -87.79 | 5.99 | 16.43 | 0.23 | 0 |
| (2) | -87.58 | 5.71 | 15.65 | 0.22 | 0 |
| Effect | Term | Estimate | std.error | conf.low | conf.high | Fit |
|---|---|---|---|---|---|---|
| fixed | (Intercept) | -1.94 | 0.69 | -3.46 | -0.75 | (1) |
| fixed | o3 | -1.02 | 0.61 | -2.27 | 0.15 | (1) |
| fixed | RST1 | -0.32 | 0.65 | -1.61 | 0.92 | (1) |
| fixed | RST2 | -1.61 | 0.83 | -3.35 | -0.06 | (1) |
| varying | sd(Intercept) | 1.47 | 0.65 | 0.28 | 2.95 | (1) |
| fixed | (Intercept) | -2.04 | 0.64 | -3.43 | -0.94 | (2) |
| G-Eval(1) | G-Eval(2) | |
| 1.00 | 1.00 | |
| min ESS | 1403.77 | 2525.27 |
| div C | 0 | 0 |
| 433.57 | 434.79 | |
| 31.23 | 31.15 | |
| 59.22 | 57.63 |
| Effect | Term | Estimate | std.error | conf.low | conf.high |
|---|---|---|---|---|---|
| fixed | (Intercept) | 2.24 | 0.16 | 1.92 | 2.55 |
| fixed | zoi (Intercept) | -2.79 | 0.65 | -4.11 | -1.56 |
| fixed | coi (Intercept) | 6.24 | 1.38 | 4.14 | 9.53 |
| fixed | RST1 | 0.18 | 0.08 | 0.02 | 0.33 |
| fixed | RST2 | 0.53 | 0.09 | 0.36 | 0.71 |
| fixed | GPT-4.1 | 0.07 | 0.14 | -0.20 | 0.33 |
| Effect | Term | Estimate | std.error | conf.low | conf.high |
|---|---|---|---|---|---|
| fixed | (Intercept) | 2.26 | 0.16 | 1.95 | 2.57 |
| fixed | zoi (Intercept) | -2.68 | 0.61 | -3.95 | -1.53 |
| fixed | coi (Intercept) | 6.23 | 1.37 | 4.12 | 9.43 |
| fixed | RST1 | 0.17 | 0.08 | 0.01 | 0.33 |
| fixed | RST2 | 0.52 | 0.09 | 0.35 | 0.70 |
| fixed | GPT-4.1 | 0.06 | 0.13 | -0.20 | 0.32 |
| H | Where | Metric | Type | Description |
| factual | Factual fabrication or contradiction | |||
| X | Layout issues described in 3 | |||
| with | the model should avoid and correct | |||
| logical | Internal logical inconsistency | |||
| inconsistency | ||||
| X | Deviation from a given context |
| GPT-4o | o3 | |||||
| 0-shot | RST1 | RST2 | 0-shot | RST1 | RST2 | |
| 0.2 | 0.2 | 0.04 | 0.12 | 0.04 | 0.04 | |
| 0.6 | 0.72 | 0.8 | 0.44 | 0.44 | 0.36 | |
| 0.68 | 0.52 | 0.48 | 0.6 | 0.36 | 0.28 | |
| 0.28 | 0.28 | 0.48 | 0.2 | 0 | 0.12 | |
| 0.28 | 0.52 | - | 0.08 | 0.24 | ||
| Precision | Recall | F1 Score | |
|---|---|---|---|
| Mean | 0.9503 | 0.9393 | 0.9447 |
| SD | 0.0603 | 0.0838 | 0.0685 |
| Min | 0.7748 | 0.6484 | 0.7432 |
| 25% | 0.9313 | 0.9054 | 0.9030 |
| 50% | 0.9764 | 0.9752 | 0.9784 |
| 75% | 1.0000 | 1.0000 | 1.0000 |
| model | reasoning | method | min | q1 | q3 | SD | mean | q2 | max |
|---|---|---|---|---|---|---|---|---|---|
| gpt-4.1 | 0-SHOT | 0.66 | 0.84 | 0.92 | 0.08 | 0.89 | 0.90 | 1.00 | |
| gpt-4.1 | RST1 | 0.80 | 0.90 | 1.00 | 0.07 | 0.94 | 0.98 | 1.00 | |
| gpt-4.1 | RST2 | 0.78 | 0.92 | 1.00 | 0.06 | 0.96 | 1.00 | 1.00 | |
| gpt-4.1-mini | 0-SHOT | 0.69 | 0.83 | 0.98 | 0.09 | 0.89 | 0.87 | 1.00 | |
| gpt-4.1-mini | RST1 | 0.79 | 0.87 | 1.00 | 0.08 | 0.93 | 0.97 | 1.00 | |
| gpt-4.1-mini | RST2 | 0.76 | 0.92 | 1.00 | 0.06 | 0.96 | 0.99 | 1.00 |
| model | reasoning | method | min | q1 | q3 | SD | mean | median | max |
|---|---|---|---|---|---|---|---|---|---|
| gpt-4.1 | 0-SHOT | 0.92 | 1.00 | 1.00 | 0.02 | 1.00 | 1.00 | 1.00 | |
| gpt-4.1 | RST1 | 0.75 | 1.00 | 1.00 | 0.06 | 0.98 | 1.00 | 1.00 | |
| gpt-4.1 | RST2 | 1.00 | 1.00 | 1.00 | 0.00 | 1.00 | 1.00 | 1.00 | |
| gpt-4.1-mini | 0-SHOT | 0.50 | 1.00 | 1.00 | 0.11 | 0.97 | 1.00 | 1.00 | |
| gpt-4.1-mini | RST1 | 0.83 | 1.00 | 1.00 | 0.03 | 0.99 | 1.00 | 1.00 | |
| gpt-4.1-mini | RST2 | 0.90 | 1.00 | 1.00 | 0.02 | 1.00 | 1.00 | 1.00 |
| model | reasoning | method | min | q1 | q3 | SD | mean | median | max |
|---|---|---|---|---|---|---|---|---|---|
| gpt-4.1 | 0-SHOT | 0.03 | 0.19 | 0.31 | 0.11 | 0.25 | 0.24 | 0.51 | |
| gpt-4.1 | RST1 | 0.10 | 0.24 | 0.38 | 0.11 | 0.30 | 0.32 | 0.50 | |
| gpt-4.1 | RST2 | 0.31 | 0.37 | 0.55 | 0.11 | 0.46 | 0.45 | 0.71 | |
| gpt-4.1-mini | 0-SHOT | 0.01 | 0.16 | 0.30 | 0.13 | 0.24 | 0.24 | 0.52 | |
| gpt-4.1-mini | RST1 | 0.07 | 0.28 | 0.40 | 0.12 | 0.35 | 0.34 | 0.57 | |
| gpt-4.1-mini | RST2 | 0.30 | 0.40 | 0.65 | 0.16 | 0.54 | 0.50 | 0.86 |
| model | reasoning | method | min | q1 | q3 | SD | mean | median | max |
|---|---|---|---|---|---|---|---|---|---|
| gpt-4.1 | 0-SHOT | 0.18 | 0.29 | 0.43 | 0.12 | 0.37 | 0.36 | 0.66 | |
| gpt-4.1 | RST1 | 0.19 | 0.38 | 0.57 | 0.13 | 0.46 | 0.48 | 0.69 | |
| gpt-4.1 | RST2 | 0.44 | 0.58 | 0.73 | 0.12 | 0.65 | 0.68 | 0.84 | |
| gpt-4.1-mini | 0-SHOT | 0.17 | 0.32 | 0.48 | 0.12 | 0.40 | 0.41 | 0.63 | |
| gpt-4.1-mini | RST1 | 0.27 | 0.37 | 0.59 | 0.14 | 0.51 | 0.51 | 0.77 | |
| gpt-4.1-mini | RST2 | 0.43 | 0.68 | 0.86 | 0.14 | 0.75 | 0.76 | 0.96 |
| ids | min | q1 | q3 | SD | mean | median | max |
|---|---|---|---|---|---|---|---|
| 1 | 0.80 | 0.81 | 0.92 | 0.07 | 0.88 | 0.86 | 1.00 |
| 2 | 0.87 | 1.00 | 1.00 | 0.03 | 0.99 | 1.00 | 1.00 |
| 3 | 0.80 | 0.90 | 0.98 | 0.06 | 0.93 | 0.94 | 1.00 |
| 4 | 0.90 | 0.99 | 1.00 | 0.03 | 0.98 | 1.00 | 1.00 |
| 5 | 0.84 | 0.95 | 1.00 | 0.05 | 0.97 | 1.00 | 1.00 |
| 6 | 0.91 | 0.99 | 1.00 | 0.02 | 0.99 | 1.00 | 1.00 |
| Metric | Value |
|---|---|
| Accuracy estimate, 95% CI | 0.7733 (0.6979, 0.8376) |
| Kappa | 0.4032 |
| No Information Rate (NIR) | 0.6733 |
| P-Value [Accuracy NIR] | 0.0048 |
| McNemar’s Test P-Value | 0.0000 |
| Sensitivity (Recall) | 0.3878 |
In-Context Learning
In-context learning studies how models adapt to new tasks from examples or instructions supplied at inference time, without changing their weights.
See all 432 papers