Multimodal large language models (LLMs) are increasingly used for structured extraction from financial documents, but their behavior under visual ambiguity is poorly understood. We study CARLAR (Courtesy Amount Recognition and Legal Amount Recognition) influence: a model returns the same amount for both cheque fields even though their annotations differ, including cases with a blank annotation encoded as zero. Across ten visual prompting conditions, we evaluate GPT-4.1 and Claude 4.5 Sonnet on 243 primary cheques (124 with two observed unequal amounts) and 598 strict mismatches from two additional datasets. Under courtesy redaction, GPT-4.1 CARLAR reaches 73.7 percent (95 percent bootstrap interval: 68.3 to 79.0; p below 0.001 versus baseline). Model-reported confidence can remain high when predictions agree; expected calibration error reaches 0.93 among CARLAR cases. We also test a compressed-image few-shot intervention that presents examples as one visual table without model fine-tuning. On the 243-case cohort, Claude's Control condition increases paired joint extraction accuracy by 11.5 percentage points (95 percent interval: 6.2 to 16.9; adjusted p = 0.00054), but its strict-124 gain is smaller and not significant after correction. The intervention's effect on predicted-field agreement is model-dependent: it decreases for Claude and increases for GPT-4.1. These results motivate validation methods that account for visual context and dependencies between extracted fields.
Figures & tables
Figure 1. Whitespace in a CARLAR example
Figure 2. Handwritten "7" deviating from typical digit representation
Figure 3. Digit morphology
Figure 4. Unreadable courtesy amount from thick handwriting.
Experiment
Claude 4.5
95% CI
GPT-4.1
95% CI
Exp 1 (Baseline: Full image)
5.8%
[2.9, 9.1]
5.3%
[2.9, 8.2]
Exp 2 (Annotated Both)
9.9%
[6.2, 13.6]
14.4%
[10.3, 18.9]
Exp 3 (Annotated Separate)
4.9%
[2.5, 7.8]
14.8%
[10.3, 19.3]
Exp 4 (Cropped Separate)
5.3%
[2.9, 8.2]
2.9%
[0.8, 5.3]
Exp 5 (Cropped Courtesy + Full Legal)
1.6%
[0.4, 3.3]
0.4%
[0.0, 1.2]
Exp 6 (Cropped Legal + Full Courtesy)
15.2%
[10.7, 19.8]
16.0%
[11.5, 20.6]
Table 1. Predicted-field agreement on 243 primary cheques (124 strict two-field mismatches and 119 blank-annotation cases), with 95% bootstrap confidence intervals (10,000 resamples). The strict rows isolate two observed unequal values; external rows use separate Indian cheque datasets. The displayed primary rates use the blank-annotation-as-zero convention, and paired-condition comparisons use McNemar’s test.
Figure 5. Distribution of Relative Deviation Asymmetry (RDA) for courtesy predictions in Experiment 1 (Full Image baseline). RDA = 0 indicates no drift from courtesy ground truth; RDA = 1 indicates full drift to the legal ground truth value. Black vertical line marks the median. Sample sizes differ between models because the RDA filter excludes CARLAR cases (exact agreement) and zero predictions, which occur at different rates for each model.
Figure 6. CARLAR transition probabilities: P(CARLAR in column∣CARLAR in row) . Higher values (darker cells) indicate stronger predictive relationships between experimental conditions.
Experiment
Model
Court. Acc.
Legal Acc.
Both Acc.
CARLAR
CARLAR-NZ
Prompt-v1 Baseline
GPT-4.1
85.2%
75.3%
64.6%
2.9%
2.9%
Control Set
GPT-4.1
82.7%
88.9%
73.7%
9.5%
9.1%
Prediction-Guided (Type 1)
GPT-4.1
79.8%
77.0%
63.4%
13.6%
11.9%
Comprehensive Set
GPT-4.1
81.5%
85.2%
69.1%
9.1%
8.2%
Prompt-v1 Baseline
Claude-4.5
54.7%
72.4%
41.2%
7.0%
4.1%
Control Set
Claude-4.5
67.5%
75.7%
52.7%
0.4%
0.4%
Table 2. Prompt-v1 baseline and three verified compressed-image example sets: Prediction-Guided Type 1 ( n=7 ), Control Type 2 ( n=10 ), and Comprehensive Type 3 ( n=5 ). All columns use the same 243 matched cheque IDs; nine non-cheque rows from the historical 252-row selection were excluded. These example sets differ from the confidence-elicitation prompts in Table 5 . Prompt-v1 and the full-image visual baseline use different prompts.
Example Set
Model
Joint Accuracy Gain (pp) [95% CI]
BH-adjusted p
Prediction-Guided (Type 1)
GPT-4.1
−1.2 [ −7.0 , 4.5]
0.780
Control (Type 2)
GPT-4.1
9.1 [3.3, 14.8]
0.0077
Comprehensive (Type 3)
GPT-4.1
4.5 [ −0.8 , 9.9]
0.173
Prediction-Guided (Type 1)
Claude 4.5
7.0 [1.6, 12.3]
0.0279
Control (Type 2)
Claude 4.5
11.5 [6.2, 16.9]
0.00054
Comprehensive (Type 3)
Claude 4.5
9.9 [4.9, 14.8]
0.00081
Table 3. Paired joint-extraction accuracy differences against the model’s prompt-v1 baseline on the same 243 cheque IDs. Confidence intervals are percentile bootstraps of paired per-cheque differences (10,000 resamples); exact two-sided McNemar p -values are Benjamini–Hochberg-adjusted over these six few-shot comparisons. Intervals themselves are not multiplicity-adjusted.
Example Set
Model
Strict Joint Gain (pp) [95% CI]
BH-adjusted p
Prediction-Guided (Type 1)
GPT-4.1
5.6 [ −3.2 , 14.5]
0.332
Control (Type 2)
GPT-4.1
23.4 [14.5, 32.3]
<0.00001
Comprehensive (Type 3)
GPT-4.1
14.5 [5.6, 23.4]
0.0088
Prediction-Guided (Type 1)
Claude 4.5
6.5 [ −0.8 , 13.7]
0.201
Control (Type 2)
Claude 4.5
8.1 [0.8, 15.3]
0.105
Comprehensive (Type 3)
Claude 4.5
4.0 [ −2.4 , 10.5]
0.332
Table 4. Exploratory paired joint-accuracy sensitivity on the same 124 cheques with two nonblank unequal annotations. Percentile bootstrap intervals use 10,000 paired resamples; exact two-sided McNemar p -values are BH-adjusted over this separate six-test strict family. Intervals themselves are not multiplicity-adjusted and need not have the same decision boundary as the adjusted tests.
Method
Field
ECE
Brier
GPT
Claude
GPT
Claude
All Rows ( n=243 )
LLM-as-Judge
Courtesy
0.43
0.65
0.43
0.62
LLM-as-Judge
Legal
0.28
0.27
0.31
0.34
Azure OCR
Courtesy
0.31
0.30
0.32
0.33
Azure OCR
Legal
0.48
0.37
0.42
0.31
Table 6. Calibration metrics for LLM-as-Judge and Azure OCR confidence on 243 cheques. Overall ECE uses 10 bins; CARLAR-only ECE uses five bins on separate run-specific subsets (GPT: 55 LLM, 22 OCR; Claude: 44 LLM, 40 OCR). Lower values indicate better calibration within each setting; ECE across different bin counts should not be compared as a paired change.
Method
Model
Overall Acc.
Coverage
Sel. Acc.
LLM-as-Judge
GPT-4.1
67.9%
85.6%
63.5%
LLM-as-Judge
Claude 4.5
38.3%
66.3%
35.4%
Azure OCR
GPT-4.1
92.2%
58.0%
98.6%
Azure OCR
Claude 4.5
93.0%
60.9%
95.9%
Table 7. Courtesy-field accuracy before and after filtering at the 0.9 confidence threshold. Overall accuracy, coverage, and selective accuracy are computed within each separate confidence experiment on the same 243 cheques; these are not the prompt-v1 predictions in Table 2 .
Figure 7. Risk-coverage curves for courtesy (left) and legal (right) amount extraction. LLM-as-Judge confidence (solid lines) yields nearly flat risk curves, while OCR confidence (dashed) shows a clear risk–coverage tradeoff.