Data Quality in RVL-CDIP: A Systematic Audit

Stefan Larson; Kevin Leach

cs.CVOct 23, 2026

Data Quality in RVL-CDIP: A Systematic Audit

Authors: Stefan Larson, Kevin Leach

Organizations: Department of Computer Science, Vanderbilt University, Nashville, TN 37235, USA

Abstract

RVL-CDIP is the de facto standard benchmark for document classification, and reported accuracy on it is now 98.8%. In this paper, we audit RVL-CDIP for data quality issues and find that approximately 10.5% of its 400,000 documents have erroneous labels and that 35.2% of the test set has a duplicate or near-duplicate in the training set. Identification codes on the pages are sufficient to classify at 69% accuracy, and a state-of-the-art model loses 11 points without them. Image-only models trained on RVL-CDIP lose 16 to 28 points on documents from other sources, and the choice of OCR engine changes accuracy by 4.1 to 8.3 points. We find that nearly 16,000 RVL-CDIP documents contain sensitive personal data, and produce de-identified versions of these documents. We re-benchmark 25 models, and with all defects remediated, the 13 that we retrain are 0.9 to 2.9 points more accurate than in the original condition; the most accurate of them, LayoutLMv3, is 96.5% accurate. We release the corrected corpus, the annotations and the out-of-distribution data.

Figures & tables

Figure 1: The defects of RVL-CDIP examined in this paper, each with an example page and the corresponding result. The addressee of the letter in (a) is masked, and the image models in (c) are those trained on the original labels.

Figure 2: Example pages from twelve of the 16 RVL-CDIP categories, taken from the de-identified test split.

Category	Train	Val.	Test
advertisement	19,963	2,522	2,515
budget	20,010	2,485	2,505
email	19,954	2,530	2,516
file_folder	20,022	2,451	2,527
form	19,957	2,537	2,506
handwritten	20,034	2,434	2,532

Table 1: RVL-CDIP per-category train, validation, and test counts.

Model (Reported by)	Mod.	Acc.	Model (Reported by)	Mod.	Acc.
LayoutLLM ( Fujitake, 2024 )	I, T, L	98.80	MGDoc ( Wang et al., 2022b )	I, T, L	93.64
EAML ( Bakkali et al., 2021 )	I, T	97.70	Dessurt ( Davis et al., 2022 )	I	93.60
Bi-VLDoc ( Luo et al., 2025 )	I, T, L	97.17	Bigbird-base ( Pham et al., 2022 )	T	93.48
ERNIE-Layout-large ( Peng et al., 2022 )	I, T, L	96.27	Pramanik et al. (2022)	I, T, L	93.36
UDOP-Dual ( Tang et al., 2023 )	I, T, L	96.22	Bigbird-large ( Pham et al., 2022 )	T	93.34
DocFormer-base ( Appalaraju et al., 2021 )	I, T, L	96.17	Multimodal Ensemble ( Dauphinee et al., 2019 )	I, T	93.07

Table 2: Reported accuracy on RVL-CDIP by model and modality (I: image, T: text, L: layout), ordered by reported score.

Entity type
US Social Security number
Age
Date of birth
Sex/gender
Place of birth
Marital/parental/spousal status

Table 3: The eleven sensitive entity types recorded during annotation.

Dataset size	400,000
US Social Security no.	2,342
Date of birth	12,800
Place of birth	6,125
Age	219
Home address	3,908
Home telephone	1,801

Table 4: Sensitive entity counts in RVL-CDIP. Counts for all five IIT-CDIP-derived datasets are given in Table D.1 .

Figure 3: Documents from RVL-CDIP containing sensitive personal information. Sensitive fields have been obscured by us; they appear unobscured in the corpus as distributed.

Figure 4: Synthetic replacement of non-PII dates for illustrative purposes, top to bottom: August 18 $\rightarrow$ March 25 , 7/31/84 $\rightarrow$ 5/31/55 , July 6, 1987 $\rightarrow$ July 2, 1954 .

Figure 5: Check images labeled budget (top row) and invoice (bottom row) in RVL-CDIP.

Figure 6: A placeholder page from the advertisement category.

Figure 7: Non-English examples in RVL-CDIP, with their language and category label.

Category	Description	Notes
advertisement	Advertisements from print media such as newspapers and magazines; a smaller number of scripts for television or radio advertisements; a small number of advertisement order and insertion documents.	Excludes the recurring “IMAGE NOT AVAILABLE” page, which appears predominantly in this category.
budget	Expense, spending, sales, cash, and accounting reports and forecasts; budgets; quotes and estimates; income and bank statements. Also receipt-like documents including political campaign contribution requests, other receipts, and checks and check stubs.	Checks are assigned here rather than to invoice : 8.8% of reviewed budget documents were checks, against 3.8% of invoice .
email	Scanned images of printed email messages.
file_folder	Scanned images of folders and binders, often characterized by vertically oriented text indicating a folder label.	A moderate number contain handwritten notes. Some scans are indistinguishable from blank pages.
form	Documents with ruled lines or boxes indicating fillable fields; also documents composed largely of key–value pair elements, such as “Size: Large”, whether or not the fields are ruled.	Substantial overlap with questionnaire , many of which contain form-like elements.
handwritten	Handwritten documents, including handwritten letters and scientific notes.	Takes precedence over the content-based categories: a handwritten letter is handwritten .

Table 5: Operational criteria for the 16 RVL-CDIP categories, with notes recording boundary cases encountered during annotation.

Figure 8: Examples of each error type. Top row: unknown , documents that do not satisfy the criteria for any RVL-CDIP category. Middle row: wrong , where a different category is correct. Bottom row: mixed , where more than one category legitimately applies.

Category	Unknown	Wrong	Mixed	Any Error	Unsure	Any + Unsure
advertisement	5.89%	0.60%	1.62%	8.12%	1.50%	9.61%
budget	4.93%	8.48%	3.94%	17.35%	1.66%	19.00%
email	8.12%	1.27%	0.12%	9.51%	0.50%	10.02%
file_folder	3.33%	0.16%	0.52%	4.01%	0.08%	4.09%
form	4.54%	3.60%	6.10%	14.25%	1.04%	15.29%
handwritten	9.50%	2.12%	1.03%	12.66%	0.00%	12.66%

Table 6: Label error rates by category and error type. Any Error is the sum of unknown , wrong , and mixed ; the final column additionally counts unsure cases as errors.

Figure 9: Example test–train pairs: the first is a duplicate, the second a template-match near-duplicate. Near-duplicates need not be pixel-identical: they share an underlying document template while differing in filled fields or scanning conditions.

Figure 10: Matched SuperPoint keypoints of a test document (left) and its duplicate in the training split (right), a random sample of 120 of the 599 matches.

Figure 11: Further test–train pairs detected as duplicates, each annotated with its CLIP similarity ( Image Similarity ), text sequence similarity ( Text Similarity ) and SuperPoint match count ( Feature Match ).

Category	Test		Validation
	# Dup.	% Dup.	# Dup.	% Dup.
advertisement	856	34.04%	843	33.43%
budget	1,479	59.04%	1,450	58.35%
email	175	6.96%	190	7.51%
file_folder	50	1.98%	56	2.28%
form	1,458	58.18%	1,455	57.35%

Table 7: Test–train and validation–train duplication by category. The percentage is the proportion of that split having a duplicate or near-duplicate counterpart in the training split.

Figure 12: Identification codes from RVL-CDIP, illustrating the range of position, orientation, typeface, and format.

Figure 13: Document classifiers are sensitive to the removal of identification codes. After the code (marked in red) is erased from the page, a DiT classifier changes its prediction to an incorrect label.

Figure 14: Position of the detected identification codes on the page for five RVL-CDIP categories, as the share of the category’s pages with a code at each position; all 16 categories are in Figure H.1 .

Model	RVL-CDIP	Tob.3482
Random chance	6.25	17.8
Zero-shot CLIP	39.2	49.5
Code feats.	69.2	56.5
Code feats. +CNN	69.9	57.8
File name only	68.8	55.6

Table 8: Accuracy of an XGBoost classifier on identification-code features, alone and with a CNN on the cropped code, and on the file name of the page. Zero-shot CLIP from Siddiqui et al. (2021) .

Figure 15: Documents from RVL-CDIP- N , shown prior to grayscale conversion. These belong to the RVL-CDIP categories but are largely born-digital, post-2006, and unrelated to the tobacco industry.

Figure 16: Documents from RVL-CDIP- O , shown prior to grayscale conversion. Each is an unremarkable document that belongs to none of the 16 RVL-CDIP categories.

Paper	Engine	Paper	Engine
LayoutLLM ( Fujitake )	EasyOCR	UDoc ( Gu et al. )	EasyOCR a
EAML ( Bakkali et al. )	Tesseract 4.0	LayoutLM ( Xu et al. )	– b
Bi-VLDoc ( Luo et al. )	Tesseract	MATrIX ( Delteil et al. )	Textract
ERNIE-Layout ( Peng et al. )	Microsoft OCR	MGDoc ( Wang et al. )	EasyOCR
UDOP ( Tang et al. )	–	Pramanik et al.	Tesseract
DocFormer ( Appalaraju et al. )	Textract	Dauphinee et al.	Tesseract

Table 9: OCR engine used for the RVL-CDIP text, as stated in each surveyed paper with a text or layout model. No paper reports a measurement of OCR quality on RVL-CDIP, and only UDoc compares engines.

Category	Textract	Tesseract	Textract	Tesseract	distance	overlap
	Words per page		Empty (%)		Char.	Word
advertisement	97	51	3.1	11.9	0.76	0.23
budget	153	107	0.7	1.6	0.58	0.46
email	124	118	0.0	0.2	0.11	0.85
file_folder	4	6	54.3	22.6	0.86	0.17
form	164	139	0.3	0.8	0.35	0.61

Table 10: Agreement between the Textract and Tesseract 4.1.1 text layers on all RVL-CDIP pages (25,000 per category, 400,000 in all). Empty pages do not have any text. Character distance is the edit distance normalized by the Textract length, and word overlap is the

F_{1}

between the two bags of lowercased, punctuation-stripped words; both are medians over the 374,439 pages on which neither engine is empty.

Condition	Test pages	Test labels	Trained on	$n$
All test pages
Orig.	all	original	original labels, with codes	40,000
Drop.	wrong labels removed	original	original labels, with codes	35,280
Fixed	uncorrectable removed	corrected	original labels, with codes	36,446
Fixed+RT	uncorrectable removed	corrected	corrected labels, with codes	36,446
De-duplicated test pages

Table 11: The eleven conditions of Table 12 , under the column names used there. Trained on lists the labels and pages of the training and validation splits, and

n

the number of test pages.

Image
	All test pages				De-duplicated				Codes removed
Model	Orig.	Drop.	Fixed	Fixed+RT	Orig.	Drop.	Fixed	Fixed+RT	Test	+RT	Full
$n$	40,000	35,280	36,446	36,446	25,923	22,530	23,132	23,132	40,000	40,000	23,132
AlexNet	89.3	92.5	90.2	92.8	85.8	90.0	88.5	90.4	84.9	88.5	90.2
GoogLeNet	89.2	92.6	90.3	93.2	86.0	90.3	88.7	91.0	85.7	88.5	90.8
SqueezeNet	88.7	92.3	90.1	92.6	85.8	90.3	88.9	90.8	83.8	87.7	90.5

Table 12: Accuracy (%) on the RVL-CDIP test split under the eleven conditions of Table 11 , all on the de-identified corpus of Section 3 . Retrained models are means over three seeds (one run for the linear baselines), and values marked

\dagger

apply the test-side changes only. Models marked

\ddagger

are scored on the 38,377 test pages that have OCR text.

Figure 17: Test accuracy with and without identification codes, one line per model. Left: models trained on pages with codes and the original labels, tested on pages with the codes and with the codes removed. Middle: the same models against models trained and tested on pages without codes. Right: the left panel for models trained on the corrected labels. Retrained models are means over three seeds (one run for the linear baselines).

Figure 18: Test accuracy before and after the other remediations, each measured against the original condition (original labels, all test pages, codes present), one line per model: corrected test labels without retraining, corrected labels with retraining, test pages with a detected training duplicate removed, and all three remediations together. Values are those of Table 12 .

Figure 19: Fully remediated test accuracy (corrected labels, codes removed from the training and test pages, test pages with a detected training duplicate left out) of the retrained models, with the linear SVMs on frozen features set against the fine-tuned models. The SVMs are those of Table K.1 without the handwriting-token and Tesseract variants.

Model	Original	Fixed	No codes	Both	$\Delta$
Image
AlexNet	61.4	73.1	59.9	70.2	$+$ 8.8
GoogLeNet	64.0	76.1	61.7	72.9	$+$ 8.8
SqueezeNet	66.4	76.4	63.0	73.7	$+$ 7.3
VGG-16	66.8	79.6	67.0	77.7	$+$ 10.9
ResNet-50	66.6	78.1	63.6	76.7	$+$ 10.1

Table 13: Accuracy (%) on RVL-CDIP- N . Left, the retrained models by training variant (means over three seeds, one run for the linear baselines); right, the public checkpoints and zero-shot models, which were not retrained. Models marked

\ddagger

are scored on the 1,001 pages that have OCR text.

\Delta

is the difference between Both and Original .

Figure 20: Test accuracy against accuracy on RVL-CDIP- N , one point per model; points below the diagonal are less accurate on RVL-CDIP- N than on the test split. Left: models trained and tested with the original labels. Right: models retrained on the corrected labels; the public checkpoints and the zero-shot models are scored against the corrected test labels and keep their single RVL-CDIP- N value. The four large language models (Maverick, gpt-oss, Qwen, Mistral) are scored on the pages that have OCR text.

		T-O		N-O
Input	Model	MSP	Energy	MSP	Energy
Image	AlexNet	0.880	0.914	0.645	0.673
	GoogLeNet	0.837	0.845	0.604	0.594
	ResNet-50	0.874	0.875	0.624	0.600
	ResNeXt-50	0.882	0.889	0.618	0.612
	VGG-16	0.889	0.930	0.661	0.668

Table 14: Separability (AUC) of in-domain from out-of-scope documents under maximum softmax probability and the energy score, for models trained on the original labels; Section 11 compares their corrected-label counterparts. T-O separates the RVL-CDIP test split (40,000 documents) from RVL-CDIP- O (2,915); N-O separates RVL-CDIP- N (1,002) from RVL-CDIP- O . The zero-shot models are never trained on RVL-CDIP. Appendix L shows FPR95 and two further scores.

Image
	T-O				N-O
Model	Original	Fixed	No codes	Both	Original	Fixed	No codes	Both
AlexNet	0.877	0.905	0.872	0.902	0.643	0.731	0.645	0.717
GoogLeNet	0.860	0.884	0.855	0.876	0.620	0.702	0.619	0.688
SqueezeNet	0.877	0.899	0.873	0.901	0.648	0.744	0.642	0.738
VGG-16	0.899	0.924	0.887	0.918	0.671	0.763	0.656	0.743

Table 15: Out-of-scope separability (AUC, maximum softmax probability) by training variant: T-O is the RVL-CDIP test split against RVL-CDIP- O , N-O is RVL-CDIP- N against it. The public checkpoints and the zero-shot models were not retrained and have one value per setting, placed under Original ; Table L.1 shows the energy score.

	Original labels			Errors fixed
Model	Tesseract	Textract	$\Delta$	Tesseract	Textract	$\Delta$
TF-IDF SVM	81.96	90.23	+8.27	86.21	94.07	+7.86
BERT-base	86.66	92.54	+5.88	90.23	95.97	+5.75
RoBERTa-base	86.50	92.47	+5.97	90.38	95.97	+5.59
LayoutLM-base	89.39	93.50	+4.11	92.43	96.99	+4.56

Table 16: Accuracy by OCR engine, holding architecture fixed, on the RVL-CDIP test split. Tesseract is our own 4.1.1 run. Original uses the labels as released; errors fixed uses the corrected labels of Section 5 .

\Delta

is Textract minus Tesseract, in accuracy points. Every

\Delta

is significant under a paired McNemar test at

p<10^{-200}

. The first three models read text alone; LayoutLM also reads the coordinates the engine assigns.

Label	$n$	Dupl. (%)	OR	OR, strat.	$n$	Dupl. (%)	OR	OR, strat.
	Test				Validation
Correct	35,280	36.1	–	–	35,291	36.2	–	–
Error or unsure	4,719	28.1	0.69	0.67	4,709	29.5	0.74	0.75
relabeled	1,166	48.4	1.66	0.98	1,089	49.0	1.70	0.96
not relabeled	3,553	21.4	0.48	0.57	3,620	23.6	0.55	0.68

Table 17: Label status of the test and validation pages against the detection of a training duplicate. Dupl. is the share of pages with a training duplicate, OR the odds ratio of having one against the pages with a correct label, and OR, strat. the Mantel–Haenszel odds ratio stratified by original category. Error or unsure are the pages with a label error (unknown, wrong or mixed) or an unsure label in the review of Section 5 ; the review assigned a corrected category to the relabeled pages.

Model	Accuracy N	N	O	MSP	Energy	MSP	Energy
		Mean MSP		$\Delta$ AUC N-O		$\Delta$ AUC T-O
AlexNet	62.1 $\to$ 72.9	0.884 $\to$ 0.930	0.821 $\to$ 0.837	$+0.096$	$+0.078$	$+0.007$	$+0.005$
GoogLeNet	63.1 $\to$ 73.5	0.900 $\to$ 0.933	0.861 $\to$ 0.870	$+0.080$	$+0.067$	$+0.005$	$-0.015$
ResNet-50	62.1 $\to$ 75.5	0.902 $\to$ 0.932	0.858 $\to$ 0.876	$+0.066$	$+0.083$	$-0.006$	$-0.014$
ResNeXt-50	65.2 $\to$ 77.9	0.893 $\to$ 0.944	0.846 $\to$ 0.865	$+0.106$	$+0.090$	$-0.006$	$-0.016$
VGG-16	67.0 $\to$ 80.3	0.933 $\to$ 0.963	0.884 $\to$ 0.892	$+0.098$	$+0.096$	$-0.008$	$-0.006$

Table 18: Retraining on the corrected labels and out-of-scope separability. Each row pairs a model trained on the original labels with the same architecture retrained on the corrected labels. Accuracy on RVL-CDIP- N (%) and the mean maximum softmax probability on RVL-CDIP- N and RVL-CDIP- O are given as original

\to

corrected;

\Delta

AUC is corrected minus original.

Image
	Alone						Pairs
Model	Labels	Codes	De-dup.	Sum	All three	Difference	L, C	L, D	C, D
AlexNet	$+$ 3.5	$-$ 0.8	$-$ 3.5	$-$ 0.8	$+$ 0.9	$+$ 1.7	$+$ 0.7	$+$ 1.1	$-$ 0.3
GoogLeNet	$+$ 3.9	$-$ 0.7	$-$ 3.3	$-$ 0.1	$+$ 1.5	$+$ 1.7	$+$ 0.6	$+$ 1.1	$-$ 0.3
SqueezeNet	$+$ 3.9	$-$ 1.0	$-$ 2.9	$+$ 0.1	$+$ 1.9	$+$ 1.8	$+$ 0.8	$+$ 1.1	$-$ 0.3
VGG-16	$+$ 3.4	$-$ 0.5	$-$ 3.2	$-$ 0.3	$+$ 1.0	$+$ 1.3	$+$ 0.3	$+$ 1.1	$-$ 0.2

Table 19: Change in test accuracy (points) of the retrained models against the original condition: under each remediation alone (corrected labels with retraining, codes removed from the training and test pages, de-duplicated test pages), their sum, and all three together, with the difference between the last two. The last three columns show the interaction of each pair of remediations (L labels, C codes, D de-duplication), which is the amount by which the change under the pair differs from the sum of its two single changes. Values are means over three seeds (one run for the linear baselines).

Model	Backbone	RVL-CDIP classifier	Availability
DiT ( Li et al., 2022 )	authors	authors	Hugging Face, live
Donut ( Kim et al., 2022 )	authors	authors	Hugging Face, live
DocXClassifier ( Saifullah et al., 2024a )	authors	authors	institutional host, live
StrucTexTv2 ( Yu et al., 2023 )	authors	authors	host refuses access
Stacked CNN ( Das et al., 2018 )	authors	authors	2018 cloud link, unverified
LayoutLMv1 ( Xu et al., 2020 )	authors	third party	fidelity unverified

Table 20: Availability of model weights for the 34 papers of Table B.1 , as of September 2026. Only the fourteen papers that release weights of any kind are listed, named by the model each reports on RVL-CDIP; the other twenty release neither a backbone nor an RVL-CDIP classifier. “Backbone” is the pre-trained model before task fine-tuning; “classifier” is a checkpoint that performs the 16-way RVL-CDIP task and so permits re-scoring against corrected labels without retraining. Papers below the rule release a backbone, and a classifier for them is not available from any source.

Appendix figures & tables37 assets

Supplementary material from the paper’s appendix.

Appendix

Artifact	Contents	Section
Category criteria	16 operational definitions	Section 4
Label annotations	400,000 judgments; error type; corrected	Section 5
	label where determined
Corrected corpus variants	the conditions of Section 10	Section 10
Duplicate pairs	test–train and validation–train pairs	Section 6
Duplicate calibration set	1,528 pairs judged by hand, 403 matches	Section 6

Appendix

Table A.1: Artifacts released with this paper.

Paper	Tobacco3482	Other classif.	Other tasks	Pre-training
Csurka et al. (2016)	—	NIST-2, MARG, CLEF-IP, IH1–3 $\dagger$	—	—
Tensmeyer and Martinez (2017)	—	ANDOC $\dagger$	—	—
Afzal et al. (2017)	$\bullet$	—	—	RVL-CDIP
Das et al. (2018)	—	—	—	—
Sarkhel and Nandi (2019)	$\bullet$	NIST-6, MARG	—	—
Dauphinee et al. (2019)	—	—	—	—

Appendix

Table B.1: Datasets used by the 34 papers of Table 2 , all of which evaluate document classification on RVL-CDIP. “Other tasks” counts datasets used for tasks other than classification (IE: information extraction, VQA: question answering, LA: layout analysis, TSR: table structure, OCR: text detection and recognition, NLI: natural language inference).

\dagger

: not publicly available. “—” in the last column: no document-corpus pre-training.

Entity type	Covers
US Social Security number	A full Social Security number, or a part of one such as the last four digits.
Date of birth	A person’s date of birth, in any format.
Age	A person’s age, stated as a number.
Place of birth	The city, state or country in which a person was born.
Marital/parental/spousal status	A person’s marital status and the number of their children, which are often in separate fields.
Home address	A residential address, in full or in its parts (street, city, state, ZIP code). The address of an employer or other organization is not included.

Appendix

Table C.1: The sensitive entity types recorded during annotation.

Entity type	Presidio	Google DLP	Azure	Amazon
US Social Security number	0.97	0.93	0.70	0.77
Date of birth	–	0.82	–	–
Age	1.00	1.00	1.00	1.00
Sex/gender	–	1.00	–	–
Place of birth	–	–	–	–
Marital/parental/spousal status	–	–	–	–

Appendix

Table C.2: Document-level recall of four PII detectors on 1,773 annotated documents, by entity type. A dash marks a type that the detector does not support.

Figure C.1: Reconstructed examples, with invented values, of the layouts in which detectors missed Social Security numbers. In the top example the label is in a column heading above the number, and in the bottom example the number has no label.

Figure C.2: A reconstructed citizenship field. The text next to the label is the printed option U.S. , and the entered value is Sweden , after Other .

	RVL-CDIP	Tob.3482	Tob.800	FUNSD	DocVQA	Total
Dataset size	400,000	3,482	1,290	199	12,767	417,738
US Social Security no.	2,342	9	5	2	70	2,428
Date of birth	12,800	62	7	—	232	13,101
Place of birth	6,125	29	7	—	123	6,284
Age	219	—	—	—	44	263
Home address	3,908	6	1	1	276	4,192

Appendix

Table D.1: Sensitive entity counts across the five IIT-CDIP-derived datasets. “Documents affected” counts documents containing at least one sensitive entity, so it is not the column sum.

Category	Replacement
Social Security number	Nine digits in the form the Social Security Administration issues: area 001 to 899 other than 666, group 01 to 99, serial 0001 to 9999. A partial number, such as the last four digits, is filled digit by digit in the layout of the original.
Date of birth	A date between 1930 and 1970, in the original’s format (numeric or with the month written out, its separators, and a two- or four-digit year).
Age	A number between 29 and 59.
Place of birth	A real US city, with the state written as in the original (postal code, traditional abbreviation, full name, or omitted).
Marital/parental/spousal status	One status (married, divorced, widowed or single) and one number of children (none to four) per document, written as in the original (a word, an initial, a coded entry such as “c. 2”, or a count of children).
Home address	A street line in the usual US form, named after a tree, a president, a surname, a US place or a numbered street, with a real US city and a ZIP code from the range of the state it is written beside.

Appendix

Table E.1: Value generators of the de-identification pipeline. Each replacement is written in the format of the original where the format can be detected, and in a format chosen at random otherwise.

Figure E.1: The typefaces bundled with doc-redact : (a) print, (b) handwriting, drawn here without the per-letter changes.

Image
Model	Test acc.	Estimate	Flagged	Precision	Recall	P at 4,719
AlexNet	89.3	4.5	5.3	39.7	17.8	37.2
GoogLeNet	89.2	5.9	6.6	40.4	22.6	38.1
SqueezeNet	88.7	3.4	4.3	44.9	16.4	39.9
VGG-16	91.3	6.0	6.3	42.6	22.8	38.6
ResNet-50	90.9	5.2	5.7	44.3	21.5	39.6

Appendix

Table F.1: Confident learning on the RVL-CDIP test split, scored against the label review, which finds a label error or an unsure label on 11.8% of the 40,000 test pages. Estimate is the share of wrong labels estimated by the method and Flagged the share of pages that it flags; precision and recall are those of the flagged pages; P at 4,719 is the precision among the 4,719 pages with the lowest label-quality score. All values are in percent and are means over three seeds.

Category	$n$	Wrong labels	Flagged	Precision	Recall
letter	2,464	30.7	5.6	68.5	12.3
budget	2,505	21.3	5.2	63.5	14.9
form	2,506	16.5	10.6	47.7	30.1
questionnaire	2,435	15.2	5.9	55.8	20.1
invoice	2,477	14.8	4.6	55.6	16.6
scientific_report	2,498	13.2	10.3	31.9	23.8

Appendix

Table F.2: Confident learning by original category on the RVL-CDIP test split: number of test pages, share of wrong labels according to the review, and the share of pages flagged, with the precision and recall of the flags. Values are in percent and are means over the eleven models and three seeds of Table F.1 .

	adv	bud	ema	fol	frm	hw	inv	let	mem	news	pres	qst	res	pub	rep	spec	Total
adv	–	1	1	38	22	16		14	17	39	1				1	1	151
bud		–		156	53	370	1,499	10	16	6	7					3	2,120
ema	2	15	–	3	7	2		52	138	32	49	1	2	15			318
fol	2	1		–	12	10	1	2	2	1	4			3	1	1	40
frm		62	6	100	–	74	188	82	90		1	21	6		20	255	905
hw	39	2	1	45	78	–	1	234	19	6		106		3		2	536

Appendix

Table G.1: Original label (rows) against corrected label (columns) for the 15,366 relabeled documents of RVL-CDIP, all splits. Empty cells are zero. adv = advertisement , bud = budget , ema = email , fol = file_folder , frm = form , hw = handwritten , inv = invoice , let = letter , mem = memo , news = news_article , pres = presentation , qst = questionnaire , res = resume , pub = scientific_publication , rep = scientific_report , spec = specification .

Figure H.1: Position of the identification codes detected on all 400,000 RVL-CDIP pages, by original category, as the share of the category’s pages with a code at each position; the number under each panel is the share of its pages with at least one detected code.

Category	0	1	2	3	4	5	6	7	8	9
advertisement	3.5	4.2	34.4	2.7	1.4	30.4	8.1	2.0	3.8	5.4
budget	1.5	0.7	37.6	0	0	2.2	4.5	0.7	7.0	9.1
email	0	0.2	54.4	0	0.4	16.4	0.2	14.0	0.2	0.2
file_folder	5.8	0.6	76.4	0	1.1	2.3	1.3	1.9	5.9	1.7
form	2.4	0.7	64.4	0.1	1.0	18.0	2.6	0.4	4.1	5.6
handwritten	0	0	0	0	0.1	99.9	0	0	0	0

Appendix

Table H.1: Frequency of the leading character of identification codes, by category, for the digits 0–9. Percentages within a row; rows do not sum to 100 because leading characters other than digits are omitted.

Group	Features	N
Bounding box	position, height, width, area, and center of the main code’s box, the area and the center both in pixels and as shares of the page, and whether the box is wider than it is tall	11
All codes on the page	number of codes, the boxes and orientations of up to three codes, the box enclosing all of them, and the numbers of horizontal and vertical codes	23
Length and characters	length of the main code in characters and in tokens, the numbers and shares of digits and of letters, and its first character, also as indicators for a digit and for a letter	9
String structure	number of space-separated groups and the lengths of the first two, the leading run of digits as a number and its logarithm, the number of leading letters, and indicators for a trailing letter and for a hyphen	8
Format and prefixes	for the pattern of digits and letters of the main code and for each of its first one to five characters, the distribution over categories of the training codes that share it and the logarithm of their number	102
Nearest numbers	among training codes with the same pattern, the distribution over categories of the five nearest in numeric value, the logarithm of the distance to the nearest, and its category	18

Appendix

Table I.1: Features of the code-only classifier of Section 7.3 , by group, with the number of features in each group for RVL-CDIP (sixteen categories). The main code is the first code on the page that has a transcription.

Figure J.1: Accuracy of the code-only classifier on the RVL-CDIP test split as groups of features are added cumulatively (XGBoost, 320,000 training documents). The dashed line marks zero-shot CLIP applied to the whole page ( Siddiqui et al., 2021 ) , and the dotted line a linear SVM trained on frozen ImageNet ResNet-50 features of the whole page (Table K.1 ).

Figure J.2: Recall of the code-only classifier by category on the 40,000 documents of the RVL-CDIP test split.

Category	No code	Vertical	Two-part	Ten digits	Letters
advertisement	4	73	30	39	5
budget	0	60	1	39	36
email	0	63	22	36	40
file_folder	11	86	0	76	3
form	3	88	18	65	2
handwritten	0	94	98	0	1

Appendix

Table J.1: Format of identification codes by category in the 11,200 annotated RVL-CDIP documents, in percent. No code is the share of documents without a code; the remaining columns are shares of the documents that have one: codes whose box is taller than it is wide, two-part codes of five and four digits, ten-digit numbers, and codes containing at least one letter.

Figure K.1: Accuracy on RVL-CDIP- N by training variant, one line per model, from Table 13 . Left: models trained on the original and on the corrected labels, both with codes. Middle and right: models trained with and without codes, under the original and under the corrected labels.

Figure K.2: Test accuracy lost when the identification codes are removed from the test pages, for every model. Filled markers are models trained with codes on the original labels, hollow markers the same architectures trained with codes on the corrected labels; the public checkpoints and the zero-shot models have one value.

Text
	All test pages				De-duplicated				Codes removed
Model	Orig.	Drop.	Fixed	Fixed+RT	Orig.	Drop.	Fixed	Fixed+RT	Test	+RT	Full
$n$	40,000	35,280	36,446	36,446	25,923	22,530	23,132	23,132	40,000	40,000	23,132
TF-IDF	88.9	93.0	91.2	93.4	84.7	90.0	89.1	90.3	88.8	88.8	90.3
TF-IDF + HW	89.8	94.1	92.3	94.6	86.0	91.6	90.8	92.2	89.7	89.7	92.2
TF-IDF, Tesseract text	79.8	84.0	82.5	84.7	74.0	79.1	78.5	80.2	79.6	79.7	80.0

Appendix

Table K.1: Accuracy (%) of the twelve linear SVMs on the RVL-CDIP test split under the conditions of Table 12 , one run each. HW adds two tokens that give the number of words on the page and the share that Textract marks as handwriting; the image features come from frozen encoders.

Text
Model	Original	Fixed labels	Codes removed	Both
TF-IDF	84.6	85.7	84.5	86.1
TF-IDF + HW	90.5	92.1	90.3	92.1
Image features
ResNet-50 (ImageNet)	56.0	62.1	54.0	60.5
CLIP	81.1	88.3	80.8	88.7

Appendix

Table K.2: Accuracy (%) of the linear SVMs on RVL-CDIP- N by training variant, one run each. The SVM trained on Tesseract text is left out, because the RVL-CDIP- N text is Textract output.

Model	Orig.	Drop.	Fixed	Orig.	Drop.	Fixed	Codes removed	Full	RVL-CDIP- N
	All test pages			De-duplicated
$n$	40,000	35,280	36,446	25,923	22,530	23,132	40,000	23,132	1,002
Jev 1.13	67.0	72.9	72.3	66.5	73.0	72.5	67.4	73.3	84.8
Bespoke-Nimble-9B	63.0	68.3	67.7	62.6	68.8	68.4	63.3	69.0	80.4
clef-flash	59.3	64.0	63.5	57.5	62.8	62.3	59.4	62.3	82.2
Strands Decider 2B	53.5	57.3	56.4	50.8	54.9	54.3	53.5	54.2	80.3

Appendix

Table K.3: Accuracy (%) of zero-shot decision models that read Textract text, under the test-side conditions of Table 12 and on RVL-CDIP- N . Full applies corrected labels, code removal and de-duplicated test pages together. Each model is run once.

Model	Prompt	Orig.	Drop.	Fixed	Orig.	Drop.	Fixed	RVL-CDIP- N
		All test pages			De-duplicated
$n$		38,377	33,796	34,922	24,324	21,062	21,624	1,001
Llama 4 Maverick	1, names only	67.6	73.5	72.8	65.3	72.0	71.6	87.8
	2, corpus-grounded	73.8	80.4	79.9	69.3	76.6	76.4	88.3
	3, generic	67.8	73.6	73.0	65.2	72.0	71.4	86.4
gpt-oss-20b	1, names only	64.6	70.2	69.7	63.8	70.1	69.7	84.3

Appendix

Table K.4: Accuracy (%) of the zero-shot large language models by prompt, on the pages that have OCR text, under the test-side conditions of Table 12 without code removal and on RVL-CDIP- N . Each model is run once per prompt.

Image
	Test split			RVL-CDIP- N
Model	Orig.	Fixed+RT	Full	Original	Fixed	No codes	Both
AlexNet	89.3	92.7	87.0	62.9	71.7	63.4	70.1
GoogLeNet	89.2	93.0	87.5	66.2	74.2	65.1	72.2
SqueezeNet	88.6	92.5	87.3	66.2	74.9	64.8	73.1
VGG-16	91.2	94.5	89.5	69.4	77.8	70.3	77.1

Appendix

Table K.5: Macro accuracy (%, mean per-category recall) on the RVL-CDIP test split, in three of the conditions of Table 11 , and on RVL-CDIP- N by training variant, as in Table 13 .

Image
	T-O				N-O
Model	Original	Fixed	No codes	Both	Original	Fixed	No codes	Both
AlexNet	0.912	0.931	0.894	0.923	0.676	0.741	0.661	0.723
GoogLeNet	0.874	0.879	0.860	0.856	0.627	0.691	0.626	0.660
SqueezeNet	0.885	0.882	0.852	0.869	0.644	0.719	0.665	0.722
VGG-16	0.924	0.943	0.907	0.936	0.683	0.770	0.664	0.751

Appendix

Table L.1: Out-of-scope separability (AUC under the energy score) by training variant, as Table 15 , for the models that produce logits.

	T-O					N-O
Model	MSP	Energy	MaxLogit	GEN	MCM	MSP	Energy	MaxLogit	GEN	MCM
AlexNet	0.880	0.914	0.914	0.912	–	0.645	0.673	0.673	0.673	–
AlexNet (corrected)	0.886	0.919	0.919	0.917	–	0.740	0.751	0.751	0.755	–
GoogLeNet	0.837	0.845	0.845	0.849	–	0.604	0.594	0.595	0.599	–
GoogLeNet (corrected)	0.842	0.829	0.829	0.840	–	0.684	0.661	0.662	0.674	–
ResNet-50	0.874	0.875	0.875	0.884	–	0.624	0.600	0.601	0.611	–

Appendix

Table L.2: AUC under each confidence function. (corrected): retrained on the corrected labels. MCM needs cosine similarities, so is defined for the zero-shot models only.

	T-O					N-O
Model	MSP	Energy	MaxLogit	GEN	MCM	MSP	Energy	MaxLogit	GEN	MCM
AlexNet	69.0	51.0	51.3	50.8	–	90.5	91.6	91.2	90.5	–
AlexNet (corrected)	66.2	49.8	50.1	50.6	–	86.3	86.7	85.6	85.5	–
GoogLeNet	74.5	62.8	63.1	62.8	–	93.3	92.8	93.0	92.9	–
GoogLeNet (corrected)	73.8	65.5	65.9	65.7	–	90.7	87.2	86.8	86.6	–
ResNet-50	67.1	56.3	56.0	55.4	–	92.5	93.6	93.4	93.2	–

Appendix

Table L.3: FPR95 (%) under each confidence function: the share of RVL-CDIP- O accepted at the threshold that accepts 95% of in-domain documents. Lower is better. (corrected): retrained on the corrected labels. MCM needs cosine similarities, so is defined for the zero-shot models only.

Figure L.1: Out-of-scope separability (AUC, maximum logit) of the RVL-CDIP test split against RVL-CDIP- O with the identification codes on the test pages and with the codes removed, for models trained with and without codes under the original and the corrected labels. One line per model; retrained models are means over three seeds (one run for the linear baselines).

Test set	Model	Prompt 1	Prompt 2	Prompt 3	$Q$	$p$
RVL-CDIP	Llama 4 Maverick	67.61 $\pm$ 0.47	73.79 $\pm$ 0.44	67.77 $\pm$ 0.47	1787.9	$<\!10^{-15}$
	Mistral 7B	56.96 $\pm$ 0.50	54.56 $\pm$ 0.50	55.07 $\pm$ 0.50	175.9	$<\!10^{-15}$
	gpt-oss-20b	64.63 $\pm$ 0.48	70.62 $\pm$ 0.46	64.80 $\pm$ 0.48	1642.1	$<\!10^{-15}$
	Qwen 3 32B	62.28 $\pm$ 0.48	63.56 $\pm$ 0.48	58.60 $\pm$ 0.49	853.0	$<\!10^{-15}$
RVL-CDIP- N	Llama 4 Maverick	87.81 $\pm$ 2.03	88.31 $\pm$ 1.99	86.41 $\pm$ 2.12	8.4	0.015
	Mistral 7B	78.22 $\pm$ 2.55	76.42 $\pm$ 2.63	68.33 $\pm$ 2.88	83.4	$<\!10^{-15}$

Appendix

Table N.1: Zero-shot accuracy (%) by prompt, with 95% Wilson intervals as half-widths, and the Cochran

Q

omnibus test across the three prompts. RVL-CDIP contributes 38,377 documents, RVL-CDIP- N 1,001. All eight omnibus tests reject.

Test set	Model	Contrast	$\Delta$ (pp)	$b/c$	$p_{\text{Holm}}$
RVL-CDIP	Llama 4 Maverick	P1 vs. P2	$+6.17$ $[+5.82,+6.53]$	1285/3654	$<\!10^{-15}$ ***
		P1 vs. P3	$+0.16$ $[-0.09,+0.40]$	1147/1207	0.448 ns
		P2 vs. P3	$-6.02$ $[-6.37,-5.66]$	3631/1322	$<\!10^{-15}$ ***
	Mistral 7B	P1 vs. P2	$-2.40$ $[-2.80,-2.00]$	3530/2608	$<\!10^{-15}$ ***
		P1 vs. P3	$-1.90$ $[-2.24,-1.56]$	2575/1847	$<\!10^{-15}$ ***
		P2 vs. P3	$+0.51$ $[+0.12,+0.89]$	2682/2876	0.029 *

Appendix

Table N.2: Pairwise comparisons by McNemar’s exact test.

\Delta

is the accuracy of the second prompt minus the first, in percentage points, with a 95% paired interval;

b/c

are the discordant counts.

p_{\text{Holm}}

is corrected within each test set.

Model	RVL-CDIP	- N	RVL-CDIP	- N	$\Delta\log\text{OR}$		$p$
	Rel. error reduction		Odds ratio (P2 vs. P1)		Interaction
Llama 4 Maverick	$+19.1\%$	$+4.1\%$	2.84	1.26	$-0.81$	$[-1.42,-0.21]$	0.009 **
Mistral 7B	$-5.6\%$	$-8.3\%$	0.74	0.70	$-0.05$	$[-0.44,+0.35]$	0.815 ns
gpt-oss-20b	$+16.9\%$	$-7.0\%$	2.92	0.58	$-1.62$	$[-2.26,-0.98]$	$6.4\times 10^{-7}$ ***
Qwen 3 32B	$+3.4\%$	$-26.0\%$	1.24	0.22	$-1.71$	$[-2.37,-1.05]$	$3.3\times 10^{-7}$ ***

Appendix

Table N.3: Transfer of the Prompt 2 benefit. Relative error reduction is

(\text{err}_{P1}-\text{err}_{P2})/\text{err}_{P1}

; the odds ratio is McNemar’s conditional odds ratio

c/b

, above 1 where Prompt 2 helps. The final columns test whether the log-odds benefit differs between the two test sets, a scale on which the ceiling difference does not have an effect.

Model	Dropped	Contrast	$\Delta$ scored as error	$\Delta$ complete cases
Llama 4 Maverick	278	P1 vs. P2	$+6.17$ ***	$+5.80$ ***
		P1 vs. P3	$+0.16$ ns	$+0.15$ ns
		P2 vs. P3	$-6.02$ ***	$-5.65$ ***
Mistral 7B	2771	P1 vs. P2	$-2.40$ ***	$-2.90$ ***
		P1 vs. P3	$-1.90$ ***	$-1.70$ ***
		P2 vs. P3	$+0.51$ **	$+1.20$ ***

Appendix

Table N.4: Parse-failure sensitivity on RVL-CDIP:

\Delta

for each contrast with unparseable responses scored as errors, and with the affected documents removed. No comparison changes sign or significance verdict under either treatment. Significance marks here use uncorrected

p

-values, and Table N.2 Holm-corrected ones.

Model	Prompt 1	Prompt 2	Prompt 3	$Q$	$p$
Llama 4 Maverick	3.09 $\pm$ 0.63	6.53 $\pm$ 0.90	8.04 $\pm$ 0.99	129.7	$<\!10^{-15}$
Mistral 7B	13.13 $\pm$ 1.23	8.25 $\pm$ 1.00	20.14 $\pm$ 1.46	285.8	$<\!10^{-15}$
gpt-oss-20b	37.19 $\pm$ 1.76	52.35 $\pm$ 1.81	59.02 $\pm$ 1.79	575.0	$<\!10^{-15}$
Qwen 3 32B	62.22 $\pm$ 1.76	50.74 $\pm$ 1.82	64.15 $\pm$ 1.74	282.0	$<\!10^{-15}$

Appendix

Table N.5: Out-of-scope detection on RVL-CDIP- O (%): the share of its 2,909 documents that a model rejects, explicitly or with a label outside the seventeen offered, with 95% Wilson intervals as half-widths, and the Cochran

Q

omnibus test across the three prompts. Unparseable replies count as not rejected.

Model	Contrast	$\Delta$ (pp)	$b/c$	$p_{\text{Holm}}$
Llama 4 Maverick	P1 vs. P2	$+3.44$ $[+2.59,+4.29]$	31/131	$3.9\times 10^{-15}$ ***
	P1 vs. P3	$+4.95$ $[+4.09,+5.81]$	13/157	$<\!10^{-15}$ ***
	P2 vs. P3	$+1.51$ $[+0.63,+2.39]$	64/108	0.002 **
Mistral 7B	P1 vs. P2	$-4.88$ $[-6.17,-3.59]$	257/115	$4.0\times 10^{-13}$ ***
	P1 vs. P3	$+7.01$ $[+5.63,+8.40]$	117/321	$<\!10^{-15}$ ***
	P2 vs. P3	$+11.89$ $[+10.52,+13.27]$	57/403	$<\!10^{-15}$ ***

Appendix

Table N.6: Pairwise comparisons of out-of-scope detection by McNemar’s exact test.

\Delta

is the detection rate of the second prompt minus the first, in percentage points, with a 95% paired interval;

b/c

are the discordant counts.

p_{\text{Holm}}

is corrected across the twelve contrasts.

Model	Prompt	Explicit	Own label	Unparseable
Llama 4 Maverick	1	3.09	0.00	0.03
	2	6.53	0.00	0.14
	3	8.04	0.00	0.07
Mistral 7B	1	4.78	8.35	3.64
	2	3.88	4.37	4.43
	3	8.42	11.72	7.46

Appendix

Table N.7: Routes to rejection on RVL-CDIP- O , as a percentage of documents: an explicit none_of_the_above , a label outside the seventeen offered, and unparseable replies, which count as not rejected. Detection in Table N.5 is the sum of the first two columns.

Model	Initial weights	Optimizer	Schedule	LR	Batch	Epochs
Six CNNs	ImageNet (torchvision)	SGD, momentum 0.9	polynomial, power 0.5	$10^{-3}$	64	30
DiT-base	microsoft/dit-base	AdamW	cosine, 10% warmup	$3\times 10^{-5}$	32	10
BERT-base	bert-base-uncased	AdamW	linear, 10% warmup	$5\times 10^{-5}$	16	5
RoBERTa-base	roberta-base	AdamW	linear, 10% warmup	$5\times 10^{-5}$	16	5
LayoutLM-base	microsoft/layoutlm-base-uncased	AdamW	linear, 10% warmup	$5\times 10^{-5}$	16	5
LayoutLMv3-base	microsoft/layoutlmv3-base	Adam	constant	$10^{-5}$	64	10

Appendix

Table O.1: Training setup of the eleven fine-tuned models of Section 10 . Every model is trained for the full number of epochs and the epoch with the highest validation accuracy is kept.

Document Image Understanding

Document image understanding turns page images into recognized text, layout, structured data, and answers grounded in visual evidence.

See all 148 papers

Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap
Jun 30, 2026Stefan Larson, Attila Nagy, Sam Desai +8Data Leakage Training Data Curation
Vanderbilt University · ML Collective · University of Michigan +1
What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination
Oct 15, 2026Esther XinData Contamination in Language Models Benchmark Contamination
Independent Researcher
Retrieval Augmented Classification for Confidential Documents
Apr 9, 2026Yeseul E. Chang, Rahul Kailasa, Simon Shim +2Retrieval-Augmented Classification Text Classification
Department of Security Convergence, Chung-Ang University, Seoul, South Korea · Department of Applied Data Science, San Jose State University, San Jose, CA, United States · Department of Industrial Security, Chung-Ang University, Seoul, South Korea

Jun 30, 2026J/K move · Enter open · S save

Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap

Stefan Larson, Attila Nagy, Sam Desai, Cyrus Desai+7

Vanderbilt University · ML Collective · University of Michigan+1

Data Leakage Training Data Curation Text Classification

Open paper AskPDF arXiv

RVL-CDIP is a popular dataset for benchmarking document classifiers. However, the dataset contains ample amounts of label errors as well as non-trivial amounts of test-train overlap, both of which may impact model performance metrics. In this paper, we address these two problems by (1) finding and fixing label errors, and (2) detecting and addressing test-train overlap. We produce several variations of RVL-CDIP with label error and test-train overlap fixes, and benchmark document classification performance on these new RVL-CDIP variations. Our rigorous analysis of RVL-CDIP finds that the corpus contains 12% label error and approximately 35% test-train duplication. Remediation sees improvements in classification accuracy when errors are removed, but sees decreases in accuracy when duplicates are removed. We additionally evaluate models on RVL-CDIP-N, an out-of-distribution benchmark, finding that training on error-corrected data substantially improves OOD generalization, with supervised models gaining an average of 8.1 percentage points in accuracy and improvements as large as 14 percentage points.

Abstract

Document Image Understanding

Explore similar work

Revising RVL-CDIP: Quantifying Errors and Test-Train Overlap

What Should We Remove? Evaluating the Reliability and Costs of RLVR Data Decontamination

Retrieval Augmented Classification for Confidential Documents