ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024

Eunseob Choi; Kyeonghun Kim; Hyuk-Jae Lee; Nam-Joon Kim

cs.AIOct 30, 2026

ReuScope: Auditing Data Reuse and Evaluation Sensitivity in Capsule Vision 2024

Authors: Eunseob Choi, Kyeonghun Kim, Hyuk-Jae Lee, Nam-Joon Kim

Organizations: Gwangju Institute of Science and Technology · OUTTA · Seoul National University

Abstract

We develop ReuScope, a protocol for auditing data reuse and evaluation sensitivity, using Capsule Vision 2024 as a case study. It combines measurements of data relationships with two comparisons: changing the evaluation population while holding predictions fixed, and changing training data while holding evaluation fixed. Byte-identical files cross the public training-validation boundary, and all 41 Kvasir-Capsule recordings that contribute frames occur in both partitions. Shared source recordings establish cross-split overlap; missing cross-recording patient identifiers prevent counting unique shared patients. Across ten paired seeds, removing all training rows attributed to Kvasir-Capsule, plus five rows from other sources flagged as near-matches to it, lowers public-validation balanced accuracy from 82.5% to 61.2%. Evaluating the same fitted models on the validation rows of the other sources reverses the comparison: removal improves balanced accuracy by 1.38 percentage points. The public-validation drop measures sensitivity to source removal, not an amount of leakage-induced inflation. Matched controls show that sample size and class counts alone do not explain the observed score differences. An exploratory experiment removes the training neighbors of validation frames that have close training matches, or equally many distant training rows, and yields a difference-in-differences estimate, relative to matched unflagged frames, of -0.355 percentage points in true-class probability; the paired-t interval for accuracy spans zero. The resulting procedure distinguishes observed reuse from conditional performance sensitivity and specifies the population, class support and uncertainty unit for each conclusion.

Figures & tables

Panel of Figure 1 — Figure 1: Overview of data relationships and training comparisons across evaluation views in ReuScope. The protocol audits file identity, shared videos and similarity candidates across data splits. Paired Full and Removal models are evaluated on three validation populations, separating training changes from evaluation changes. The resulting performance contrasts demonstrate evaluation-population sensitivity rather than establishing score inflation from data reuse.

Panel of Figure 2 — Figure 2: File identity and video sharing across the public training–validation boundary. (a) Byte-identity groups by source and released label relationship. The SEE-AI groups involve 215 validation files; the AIIMS groups involve six. (b) Training and validation row shares within each of the 41 contributing KVASIR video prefixes, sorted by identifier and normalized separately for each prefix. Every prefix occurs in both partitions despite the absence of byte-identical KVASIR crossings. The dashed line marks the announced 30% validation share.

Panel of Figure 3 — Figure 3: A training comparison reverses across evaluation populations (DINOv2-L/14). Full uses 37,607 rows; Removal excludes KVASIR and five other-source rows passing the Kvasir-Capsule hash screen, retaining 10,591. Markers show ten-seed mean BA; faint paths follow each fitted model’s scores across views. Full leads on public validation; Removal has higher mean BA on non-KVASIR. Below each view are paired Removal-minus-Full changes in percentage points and pointwise 95% $t$ intervals conditional on fixed lists. The internal-filtered interval includes zero. Table 1 reports all six training pools.

(a) Training pools (DINOv2, BA)
Pool	$n_{\rm train}$	Public	Non-KV	Filt.
Full	37,607	$.825$	$.745$	$.773$
Random	10,596	$.779$	$.695$	$.707$
Comp	10,596	$.734$	$.388$	$.401$
Class-match-R	10,591	$.738$	$.391$	$.406$
Removal	10,591	$.612$	$.759$	$.778$

Table 1: Training and evaluation sensitivity. Public/Non-KV/Filtered: 16,132/4,551/4,326 rows, ten classes each. Non-KV excludes KVASIR; Filtered additionally excludes internal-match candidates (definitions in Section 2 ). BA: mean class recall; AUC: macro one-versus-rest.

Endpoint	E local	E distant	U local	U distant
True-class probability	$90.654$	$90.975$	$89.670$	$89.636$
Accuracy	$99.949$	$99.977$	$99.609$	$99.558$
Fixed six-class BA	$99.667$	$99.852$	$94.947$	$94.938$

Table 2: Local and distant removal: absolute outcomes and true-class probability contrasts. E: exposed; U: matched-unflagged (588 rows each). For Distant, we average the three fixed control arms within each seed, then across the ten seeds. BA averages recall over the same six classes.

Question	Observation	Supported conclusion
Identical content?	SEE-AI: 210 groups, 215 validation files; AIIMS: 6 groups, 6 files.	Released bytes cross the public boundary. Label relationships are assessed separately.
Shared videos?	KVASIR: 0 byte crossings; all 41 contributing prefixes shared.	The split separates files while retaining source-video overlap.
Similar to which counterpart?	NCC $\geq 0.99$ : 540 chosen-counterpart vs. 799 any-match rows.	Different matching rules count different relationships.
Training sensitivity?	Removal-minus-Full BA: $-0.2132$ public, $+0.0138$ non-KVASIR.	The training contrast depends on the evaluation population.
Patient-level linkage?	Shared recordings observed; cross-recording patient IDs unavailable.	Unique-patient overlap cannot be quantified from public metadata.

Table 3: Selected conclusions from the CV2024 audit. The counterpart comparison counts KVASIR validation rows with exact-pHash matches in KVASIR training. Missing information differs from a measured zero; populations and uncertainty are defined with each analysis.

Appendix figures & tables30 assets

Supplementary material from the paper’s appendix.

Appendix

Source	Validation rows	Groups	Validation files	Rate	Labels intersect	Labels disjoint	Renamed groups
KVASIR	11,581	0	0	0.00%	0	0	0
SEE-AI	4,289	210	215	5.01%	33	177	39
KID	165	0	0	0.00%	0	0	0
AIIMS	97	6	6	6.19%	6	0	6

Appendix

Table 4: Byte-identical public train–validation crossings. Rates are proportions of affected validation files; a group can contain several files. The final columns count groups whose training and validation label sets intersect or are disjoint. Renamed groups have different training and validation basenames, so a filename-only check misses them. Label-disjoint groups are release/label anomalies; label intersection does not establish clinical label truth.

Source	$n$	Both minima zero	Each min $\leq 2$	Each min $\leq 6$	PDQ $\leq 50$	PDQ $\leq 90$
KVASIR	38,592	23,444	38,259	38,592	38,357	38,592
SEE-AI	14,285	0	0	3	0	451
KID	541	0	0	1	0	39
AIIMS	321	0	0	1	0	320

Appendix

Table 5: Query-level matching to Kvasir-Capsule. pHash and dHash are minimized independently over reference images. Both minima can be zero without a shared reference match or byte identity. PDQ is a separate perceptual hash; its cutoffs are study-specific. Distance 90 illustrates liberal candidate selection; it is not an official default.

$\max(d_{p},d_{d})$	$n$	Mean NCC	Median NCC	$\geq.99$	$\geq.95$	$\geq.90$
0	20,448	0.9994	0.9996	99.39%	99.99%	100.00%
1–2	15,473	0.9979	0.9995	93.23%	99.76%	99.96%
3–6	2,372	0.9822	0.9862	36.17%	96.21%	99.07%
>6	299	0.9690	0.9754	6.35%	86.96%	99.00%

Appendix

Table 6: Normalized cross-correlation (NCC) by selected-pair hash distance. Both hash distances refer to the pair scored by NCC. The final columns give the fraction of pairs reaching each NCC threshold. Query-level minima can correspond to different reference images and do not define these bands.

Pool	$n$	Mean cosine	% cosine $\geq.90$
KVASIR all	38,592	0.9658	98.32
pHash 0	32,854	0.9662	98.49
pHash 1–2	5,673	0.9635	97.43
pHash 3–4	65	0.9509	92.31
SEE-AI	3	0.7104	0.00
KID	1	0.3928	0.00

Appendix

Table 7: DINOv2 feature cosine similarity for CV2024 queries paired with Kvasir-Capsule references. All similarities use the same feature pipeline; the pHash rows partition the KVASIR pair inventory by hash distance. Threshold percentages describe feature-space similarity, not duplicate labels.

Transformation	$d_{p},d_{d}\leq 6$	$d_{p},d_{d}\leq 2$
Center crop, 90%	0/50 (0%)	0/50 (0%)
Center crop, 75%	0/50 (0%)	0/50 (0%)
Center crop, 60%	0/50 (0%)	0/50 (0%)
Rotation, $2^{\circ}$	40/50 (80%)	5/50 (10%)
Rotation, $5^{\circ}$	5/50 (10%)	0/50 (0%)
Rotation, $10^{\circ}$	0/50 (0%)	0/50 (0%)

Appendix

Table 8: Synthetic joint-rule hash sensitivity on 50 Kvasir-Capsule frames from 30 video prefixes. Each cell counts original–transformed pairs satisfying both 64-bit hash thresholds. Crop percentages indicate the retained fraction of each side; cropped and resized images return to the original dimensions. The same frames are used in every row. These pass rates describe the tested transformations and frames, not recall on naturally occurring duplicates.

Source	Validation rows	Basename matched rows	Exact-pHash rows	Joint-rule candidate rows
KVASIR	11,581	0	1,381	4,896
SEE-AI	4,289	174	218	264
KID	165	0	1	4
AIIMS	97	0	6	6

Appendix

Table 9: Within-source public train–validation candidates. Counts are validation rows with a training counterpart under each screen, including separate rows sharing a basename. The joint rule requires both hash distances to be at most six for the same pair. Candidate counts do not imply pixel verification; byte identities are reported separately.

Source	Rows	Pairs	Mean NCC	$\geq.99$	$\geq.95$	$\geq.90$
SEE-AI	4,289	264	0.9831	220	228	244
KID	165	4	0.9391	2	2	3
AIIMS	97	6	1.0000	6	6	6

Appendix

Table 10: NCC for selected within-source train–validation candidate pairs. The Rows column gives the validation size; the Pairs column counts one selected training counterpart per eligible row, using the lexicographically first full path satisfying

d_{p},d_{d}\leq 6

on that pair. NCC uses grayscale

96\times 96

bicubic resizing; threshold columns count selected pairs. These conditional counts do not estimate true-duplicate prevalence or detector precision.

Pair group	$n$	Median	5th–95th percentile	95% joint-prefix gap interval
Selected exact-pHash	1,381	0.880	$[0.738,0.963]$	—
Adjacent same-video	1,381	0.647	$[0.313,0.899]$	$[+0.186,+0.285]$
Distant same-video	1,381	0.432	$[0.244,0.653]$	$[+0.401,+0.487]$
Different-video, same-class	1,381	0.442	$[0.246,0.615]$	—
Different-video, any-class	1,381	0.441	$[0.253,0.612]$	—
Non-KVASIR, same-source/class	1,381	0.330	$[0.191,0.495]$	—

Appendix

Table 11: Structural similarity for selected pairs and comparison groups. Each group contains 1,381 pairs. The 5th–95th percentiles describe individual pair SSIM values. The 95% joint-prefix intervals concern selected-minus-control median SSIM for same-video controls; they resample shared video prefixes together, conditional on the selected pairs. Dashes mark the reference row or comparisons without a reported joint interval.

Sampled prefixes	95% range (%)	Width / 41-prefix width
41	$[6.899,17.996]$	1.000
30	$[6.353,19.046]$	1.144
20	$[5.636,20.807]$	1.367
10	$[4.389,24.824]$	1.841

Appendix

Table 12: Sensitivity to the number of resampled video prefixes. The exact-pHash rate, 11.9247%, is a census of KVASIR validation. These conditional percentile ranges describe hypothetical prefix sampling, not uncertainty in the enumerated count or a patient-level bound.

Evaluation view	Cohort	$n$	Mean $\Delta$	Nominal 95% BCa	Entries changing rank
Non-KVASIR	All (mixed baseline)	25	$-$ 0.073	$[-0.167,-0.043]$	18
Non-KVASIR	Primary	24	$-$ 0.050	$[-0.067,-0.032]$	19
Non-KVASIR	Subset	19	$-$ 0.062	$[-0.079,-0.045]$	14
Non-KVASIR	Subset excl. A	18	$-$ 0.066	$[-0.082,-0.049]$	14
Internal-filtered	All (mixed baseline)	25	$-$ 0.063	$[-0.165,-0.034]$	18
Internal-filtered	Primary	24	$-$ 0.039	$[-0.056,-0.023]$	16

Appendix

Table 13: Public-submission re-scoring. Score differences are relative to the organizer-reported public-validation score, the mean of macro AUC and balanced accuracy. Nominal 95% bias-corrected and accelerated (BCa) ranges resample entries individually with frames fixed. Entry independence is unestablished; we report descriptive resampling ranges and do not interpret them as calibrated population intervals. Rank changes are within each cohort. The all-25 rows use a mixed baseline: one reported public-validation score disagrees with full-list re-scoring. The primary cohort excludes that entry; the fixed subset additionally omits Teams U–Y, and its sensitivity further omits Team A.

Metric	Cohort ( $n$ )	Non-KVASIR	Internal-filtered
Pooled challenge score	24 (primary)	$-$ 0.0503	$-$ 0.0393
Equal-source BA	24 (primary)	$-$ 0.0308	$-$ 0.0255
Equal-source combined	24 (primary)	$-$ 0.0199	$-$ 0.0163
Pooled challenge score	19 (subset)	$-$ 0.0624	$-$ 0.0488
Equal-source BA	19 (subset)	$-$ 0.0370	$-$ 0.0305
Equal-source combined	19 (subset)	$-$ 0.0220	$-$ 0.0177

Appendix

Table 14: Mean paired score changes (view minus public validation) for the primary 24-entry score-consistent cohort (Team G alone excluded) and the fixed 19-entry subset sensitivity (Teams U–Y additionally omitted). The pooled challenge score averages BA and macro AUC. Equal-source scores weight sources equally: four in the public view and three in the non-KVASIR and filtered views. Within each source, BA averages recalls over present classes, and the combined score averages BA and present-class macro AUC. These descriptive means hold predictions fixed.

Cohort	$n$	Public	Non-KVASIR	Internal-filtered
Score-consistent (primary)	24	0.557	0.459	0.497
All scoreable	25	0.566	0.355	0.390
19-entry subset sensitivity	19	0.244	0.068	0.139
18-entry subset excl. A	18	0.463	0.257	0.340

Appendix

Table 15: Spearman correlation of public-view combined scores with organizer-reported official-test combined scores. The Public column uses the organizer-reported scores; the non-KVASIR and internal-filtered columns use recalculated scores. The primary 24-entry score-consistent cohort excludes only Team G, whose reported public-validation score disagrees with re-scoring. The fixed 19-entry subset sensitivity additionally omits Teams U–Y, without an independently justified exclusion rationale, and the 18-entry sensitivity also omits Team A. All four cohorts have lower observed agreement on both reduced views; possible dependence among entries limits population-level inference.

Training pool	Balanced accuracy	Mean AUC	Combined	Accuracy
Full	$0.2804\pm 0.0046$	$0.7177\pm 0.0015$	$0.4991\pm 0.0026$	$0.1758\pm 0.0098$
Random-10,596	$0.2786\pm 0.0038$	$0.7352\pm 0.0036$	$0.5069\pm 0.0028$	$0.3142\pm 0.0119$
Source-only-10,596	$0.2688\pm 0.0034$	$0.7533\pm 0.0013$	$0.5110\pm 0.0019$	$0.0988\pm 0.0063$

Appendix

Table 16: Official AIIMS test evaluation. Mean

\pm

standard error (SE) over ten seeds. Source-only training uses 10,596 rows, distinct from the 10,591-row removal input. The combined score averages balanced accuracy and mean AUC. Paired by seed, Source-only exceeds Full in mean AUC in all ten seeds, whereas the interval for its BA difference includes zero (see text); these results do not validate either filtered public view as an official-test proxy.

Class	SEE-AI	KID	AIIMS	SEE-AI	KID	AIIMS	training
	Source-only training			Non-KVASIR validation			Removal
Angioectasia	530	18	0	228	9	0	548
Bleeding	519	3	0	223	2	0	522
Erosion	2,340	0	0	1,003	0	0	2,338
Erythema	580	0	0	249	0	0	580
Foreign Body	249	0	0	107	0	0	249

Appendix

Table 17: Class and source composition after source removal. Source-only training contains 10,596 rows; non-KVASIR validation contains 4,551 rows. The last column gives class totals for the distinct 10,591-row Removal input.

Training pool	Rows	Balanced accuracy	$\Delta$ vs. Full
Full	37,607	$0.825\pm 0.008$	—
Random	10,596	$0.779\pm 0.005$	$-$ 0.046
Source-only	10,596	$0.611\pm 0.008$	$-$ 0.214
Removal	10,591	$0.612\pm 0.007$	$-$ 0.213
Class-match-R	10,591	$0.738\pm 0.005$	$-$ 0.087
Comp	10,596	$0.734\pm 0.007$	$-$ 0.092

Appendix

Table 18: Training-pool results on the public validation list (16,132 rows). Balanced accuracy is mean

\pm

sample SD over ten seeds at epoch ten. Removal and Class-match-R contain 10,591 rows; other reduced pools contain 10,596. Replacement controls A–D are defined in Appendix C.2 .

Contrast	Public $\Delta$ [95% CI]	Non-KVASIR $\Delta$ [95% CI]
Source-only $-$ Random	$-0.1682$ $[-0.1754,-0.1610]$	$+0.0624$ $[+0.0516,+0.0732]$
A $-$ Comp	$-0.0690$ $[-0.0752,-0.0628]$	$+0.0978$ $[+0.0898,+0.1057]$
B $-$ Comp	$+0.0092$ $[+0.0034,+0.0151]$	$+0.0927$ $[+0.0855,+0.0999]$
C $-$ A	$-0.0042$ $[-0.0085,+0.0002]$	$-0.0050$ $[-0.0111,+0.0011]$
D dup $-$ C	$+0.0765$ $[+0.0706,+0.0824]$	$-0.0882$ $[-0.0961,-0.0802]$
D dup $-$ Comp	$+0.0033$ $[-0.0022,+0.0088]$	$+0.0046$ $[-0.0026,+0.0118]$

Appendix

Table 19: Training-pool contrasts by evaluation population. Each cell gives the mean BA difference and a pointwise 95% paired-

t

interval over ten seeds. Each contrast includes any changes to class weights and sampling induced by the stated training-pool changes.

Training pool	Train rows	Public	Non-KVASIR	Internal-filtered
Full	37,607	0.825	0.745	0.773
Random	10,596	0.779	0.695	0.707
Comp	10,596	0.734	0.388	0.401
Class-match-R	10,591	0.738	0.391	0.406
Removal	10,591	0.612	0.759	0.778
Source-only	10,596	0.611	0.757	0.776

Appendix

Table 20: DINOv2-L/14 training pools crossed with fixed validation views. Entries are epoch-ten mean BA over ten matched seeds; predictions are held fixed across views. Public, non-KVASIR and internal-filtered views contain 16,132, 4,551 and 4,326 rows, respectively, each supporting all ten classes.

Source	$n$	Classes	Full	Removal	$\Delta$	95% paired- $t$
KVASIR	11,581	9	0.943	0.470	$-$ 0.4730	$[-0.4817,-0.4644]$
SEE-AI	4,289	8	0.697	0.710	+0.0129	$[+0.0013,+0.0245]$
KID	165	5	0.812	0.768	$-$ 0.0443	$[-0.0863,-0.0022]$
AIIMS	97	2	0.929	0.951	+0.0215	$[+0.0083,+0.0346]$

Appendix

Table 21: Validation performance by source. BA averages recall over the classes represented in each source. The contrast is Removal minus Full; intervals use ten paired seeds. Source-wise macro averages are not additive contributions to pooled BA.

Class	Rows	Full	Removal	$\Delta$ recall
Angioectasia	497	0.789	0.667	$-$ 0.122
Bleeding	359	0.855	0.842	$-$ 0.014
Erosion	1,155	0.704	0.657	$-$ 0.047
Erythema	297	0.558	0.403	$-$ 0.155
Foreign Body	340	0.896	0.650	$-$ 0.246
Lymphangiectasia	343	0.825	0.592	$-$ 0.233

Appendix

Table 22: Per-class mean recall on public validation over ten seeds. All classes are reported. Removal-minus-Full recall differences average to the pooled BA contrast.

Backbone	Validation	Full	Random	Source- only	$\Delta$
ResNet-50	Public	0.697	0.658	0.458	$-$ 0.2002
ResNet-50	Non-KVASIR	0.578	0.522	0.610	+0.0878
ConvNeXt-Tiny	Public	0.692	0.645	0.436	$-$ 0.2096
ConvNeXt-Tiny	Non-KVASIR	0.562	0.507	0.602	+0.0955

Appendix

Table 23: Source sensitivity with frozen CNN backbones and nonlinear heads. Full uses 37,607 training rows; Random and Source-only each use 10,596. Values are ten-seed mean BA;

\Delta

is Source-only minus Random.

Removal rule	Train rows	KVASIR kept	Public BA
Full	37,607	27,011	$0.825\pm 0.008$
$d_{p},d_{d}\leq 0$	21,241	10,645	$0.813\pm 0.007$
$d_{p},d_{d}\leq 2$	10,825	229	$0.712\pm 0.008$
NCC $\geq 0.99$	12,697	2,101	$0.744\pm 0.008$
NCC $\geq 0.95$	10,715	119	$0.699\pm 0.004$
NCC $\geq 0.90$	10,613	17	$0.654\pm 0.008$

Appendix

Table 24: Training-data removal thresholds for DINOv2-L/14, evaluated on all 16,132 public-validation rows. BA is the epoch-ten mean

\pm

sample SD over seeds 0–9.

d_{p}

and

d_{d}

are independently minimized pHash and dHash distances to the source reference; NCC uses the selected source counterpart. Full, Source-only and Removal are this family’s own reference runs. Appendix D.1 describes these auxiliary experiments and their hardware differences.

Label control	Histogram kept	Public BA	Non-KVASIR BA
Reassignment	No	$0.414\pm 0.006$	$0.716\pm 0.010$
Permutation	Yes	$0.504\pm 0.010$	$0.736\pm 0.011$

Appendix

Table 25: Sensitivity to KVASIR label assignment. Each model retains all 37,607 training paths; only the labels of the 27,011 KVASIR rows change. BA is mean

\pm

sample SD at epoch ten over ten seeds.

Omitted source	Train rows	Eval. rows	Classes	Full BA	Omitted BA
KVASIR	10,596	11,581	9	0.942	$0.466\pm 0.015$
SEE-AI	27,611	4,289	8	0.697	$0.239\pm 0.008$
KID	37,231	165	5	0.812	$0.495\pm 0.057$

Appendix

Table 26: Held-out-source BA after removing that source from training. Each row is a distinct training intervention; evaluation uses only the omitted source’s public-validation rows. Full is the corresponding full-training reference from each experiment’s own runs, so its KVASIR value (0.942) differs in the third decimal from the primary runs in Table 21 (0.943). Omitted BA reports epoch-ten mean

\pm

sample SD over ten seeds.

Endpoint	Mean contrast	95% paired- $t$ interval	Seed signs
True-class probability	$-$ 0.00355	$[-0.00486,-0.00224]$	10/0/0
NLL	+0.00495	$[+0.00329,+0.00660]$	0/0/10
Probability margin	$-$ 0.00472	$[-0.00647,-0.00297]$	10/0/0
Balanced accuracy	$-$ 0.00194	$[-0.00485,+0.00097]$	4/6/0
Accuracy	$-$ 0.00079	$[-0.00163,+0.00004]$	4/6/0

Appendix

Table 27: Local-versus-distant removal: exposed-minus-unflagged contrasts in Eq. 3 . Probability, probability-margin, balanced-accuracy and accuracy contrasts are in absolute score units, not percentage points; NLL uses natural logarithms. Seed signs count negative/zero/positive contrasts over ten paired seeds. Intervals are pointwise and do not adjust for design selection.

Endpoint	95% seed percentile	95% video-prefix percentile range
True-class probability	$[-0.00462,-0.00244]$	$[-0.00693,-0.00067]$
NLL	$[+0.00357,+0.00632]$	$[+0.00092,+0.00931]$
Probability margin	$[-0.00615,-0.00325]$	$[-0.00920,-0.00093]$
Balanced accuracy	$[-0.00460,-0.00004]$	$[-0.00399,+0.00000]$
Accuracy	$[-0.00153,-0.00017]$	$[-0.00161,+0.00000]$

Appendix

Table 28: Bootstrap ranges for local-versus-distant removal: exposed-minus-unflagged, local-minus-distant contrasts in Eq. 3 . Probability, margin, balanced-accuracy and accuracy contrasts are in absolute score units, not percentage points; NLL uses natural logarithms. Ranges are pointwise and exploratory, without adjustment for design selection. Seed resampling retains the pairing of all four arms. Video-prefix resampling is joint across arms and evaluation strata, with training seeds averaged within each draw. For balanced accuracy, the seed range uses the fixed six classes, whereas the prefix range averages recall over the classes present in each draw.

Fold	Test rows	Test classes	Missing train class	Terminal BA	Selected best BA
0	9,418	8	None	$0.3959\pm 0.0002$	$0.4127\pm 0.0000$
1	9,421	9	None	$0.2668\pm 0.0003$	$0.2869\pm 0.0003$
2	9,425	6	None	$0.4105\pm 0.0013$	$0.4805\pm 0.0001$
3	9,475	9	None	$0.3922\pm 0.0025$	$0.4392\pm 0.0001$
4	9,422	8	Blood	$0.2726\pm 0.0000$	$0.2801\pm 0.0005$

Appendix

Table 29: Video-grouped evaluation on Kvasir-Capsule (zero-based folds). BA is mean

\pm

descriptive across-record SD over two recorded runs. Terminal denotes the last recorded epoch; best is selected using evaluation BA. Evaluation BA also determines early stopping. Last recorded epochs: 10–17. BA averages recall over the classes present in each test fold; training sets overlap across folds.

Fraction	Train rows	Terminal BA	Selected best BA	Last recorded epoch range
0.1	2,302	$0.1339\pm 0.0175$	$0.1666\pm 0.0057$	7–11
0.25	5,761	$0.1846\pm 0.0206$	$0.2123\pm 0.0057$	8–11
0.5	11,528	$0.2364\pm 0.0084$	$0.2501\pm 0.0125$	7–9
1.0	23,061	$0.2339\pm 0.0001$	$0.2574\pm 0.0000$	11

Appendix

Table 30: Class-stratified training-frame sampling on Kvasir-Capsule. BA is mean

\pm

descriptive across-record SD over two recorded runs. Terminal denotes the last recorded epoch; best is selected using evaluation BA. Evaluation BA also determines early stopping.

Videos	Selected pool rows (mean)	Terminal BA	Selected best BA	Last recorded epoch range
5	4,255.5	$0.1835\pm 0.0417$	$0.1987\pm 0.0398$	5–7
10	8,744.0	$0.1939\pm 0.0251$	$0.2313\pm 0.0009$	9
15	14,009.0	$0.2094\pm 0.0162$	$0.2244\pm 0.0120$	9–12
20	17,289.0	$0.2117\pm 0.0076$	$0.2401\pm 0.0017$	5–8
25	23,061.0	$0.2400\pm 0.0079$	$0.2640\pm 0.0009$	10

Appendix

Table 31: Kvasir-Capsule video selection at a fixed budget of 23,000 effective training rows. BA is mean

\pm

descriptive across-record SD over two recorded runs. Terminal denotes the last recorded epoch; best is selected using evaluation BA. Evaluation BA also determines early stopping. Pool size precedes repetition or truncation to the effective training list.

Comparison	$d_{p}+d_{d}\leq 6$	$d_{p},d_{d}\leq 6$	NCC $\geq.99$ , 96	NCC $\geq.99$ , 256
BCN / HAM	0	1	0	0
HAM / unknown lesion ID	1	64	0	0
Unknown lesion ID / MSK	2	2	2	2
Other ISIC combinations	0	0	0	0
ISIC total	3	67	2	2
HyperKvasir / CV2024 KVASIR	0	1	0	0

Appendix

Table 32: External hash-candidate pairs and pixel similarity. The sum-rule and joint-rule columns score single pairs under different rules. Every joint-rule candidate was evaluated from original images. ISIC strata follow lesion-ID metadata, including an unknown-ID group; they are not verified institutions. NCC columns count joint-rule candidates passing the threshold using

96\times 96

bicubic and

256\times 256

bilinear grayscale images.

Experiment family	Runs	GPU (runs)	Minutes per run	Hours
Primary DINOv2	110	A100 (110)	10.7–32.2	22.23
Local removal	40	A6000 (24), A100 (8), L40S (8)	28.8–64.7	24.34
Frozen-backbone CNN	60	A100 (60)	3.0–8.6	5.32
Official test	30	L40S (28), A100 (1), A6000 (1)	7.8–20.9	6.06
Auxiliary CV2024 controls	154	A100, A6000, L40S	21.8 (mean)	55.96
DINOv2 feature extraction	1	RTX 4070 Ti SUPER (1)	9.5	0.16

Appendix

Table 33: Compute of the reported experiments. Minutes per run give the range of elapsed time for one run (the mean for the auxiliary controls); Hours give each family’s total elapsed execution time. GPU counts are as described in the text. Auxiliary CV2024 controls comprise the threshold, label, source-omission and add-back runs and their Full, Removal and Source-only references. The Kvasir-Capsule total and the overall total are estimates (see text).

ML Reproducibility

Machine learning reproducibility studies whether model results and scientific claims can be independently checked and reliably repeated.

See all 145 papers

Jul 22, 2026

Benchmarking the Domain Gap: Model Selection Instability Under Domain Shift in Video Capsule Endoscopy

Video capsule endoscopy (VCE) classification is typically evaluated within a single dataset, yet clinical deployment demands robustness across acquisition sources, labeling policies, and patient populations. We examine this gap using Kvasir-Capsule, Capsule Vision 2024 (CV2024), and a shared-label subset of Galar. We fine-tune a suite of general-domain pretrained backbones on the official Kvasir-Capsule folds under a standardized protocol and evaluate the same checkpoints on two non-source targets within a documented shared-label decision space. We find that the predictive value of in-domain ranking is target-dependent: Kvasir-Capsule ranking aligns more closely with Galar than with CV2024, while the two non-source targets agree only weakly. Consequently, the strongest in-domain backbone leads on one target yet falls to mid-pack on the other, and no single evaluation target reliably predicts the others. A second CV2024-trained configuration set reproduces this target-dependent instability. We conclude that capsule endoscopy model selection should report cross-target ranking stability rather than peak single-dataset performance.

Dan Hanson, Debesh Jha

Department of Computer Science · Biomedical Perception & Intelligence Lab · University of South Dakota +1

Endoscopy Medical Imaging Medical Image Classification Medical Image Benchmarks Cross-Dataset Generalization

Ask PDF

Sep 21, 2026

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

Retrieval-augmented generation (RAG) is hard to monitor in production: exhaustive relevance labels do not exist for non-stationary multi-million-passage corpora that re-index in real time. As a result, retrieval quality is generally understudied and often deprioritised in favour of generation-oriented metrics. In this work, we propose auditing retrieval coverage by probing for evidence of missing documents rather than enumerating every relevant one. Our method Re:CAP (REtrieval Coverage Audit by iterative Probing) is a reference-free audit loop applied to a deployed RAG pipeline's initial answer and retrieved context: it identifies the topics already covered, generates probing questions for plausibly missing topics, retrieves candidate documents, and applies an LLM-as-judge to retain only those that introduce previously-unretrieved information. On four public benchmarks, Re:CAP recovers 9-29% of gold labels that flat BM25 top-500 cannot reach, rising to 48% on TREC-COVID. On MuSiQue Re:CAP beats flat hybrid top-500 by +12.9 pp on recall at less than half the document budget. An ensemble BM25, dense, and hybrid baseline (top-500 each) still leaves out 21.2% of gold docs on TREC-COVID that Re:CAP recovers; human annotators judge that 78.9% of those structurally distinct documents add new information to the baseline answer (Fleiss

κ

= 0.79, n = 123), and 73.9% on live production traffic (n = 180). End-to-end recall is reproducible to within

\pm

1% across three independent runs, making Re:CAP a stable instrument for periodic retrieval audits.

Aviral Joshi, Hanoz Bhathena, Max Nelson +1

Machine Learning Center of Excellence, JPMorgan Chase & Co.

RAG Evaluation IR Evaluation

Ask PDF

Sep 28, 2026

What Paired Evaluations Reveal under Visual Perturbations

Robustness evaluation must examine diverse visual perturbations, while benchmarks cover only some real-world conditions and physical testing is costly. Paired evaluations link clean and perturbed predictions for the same image, capturing changes in correctness, confidence, and acceptance beyond aggregate accuracy. We investigate how this image correspondence supports two needs in robustness evaluation: interpreting paired evaluation results and prioritizing samples for physical testing. To interpret paired evaluation results, we fix both sets of prediction records and vary their correspondence within each class. We prove that classwise correct-correct counts give the same sharp bounds on lost acceptance and mean true-class probability decrease among retained-correct inputs as any feasible five-state refinement. Distinguishing persistent from changed wrong answers can further constrain accepted-error transitions, while shared correspondence can establish policy orderings left unresolved by separate cost intervals. To prioritize samples for physical testing, we retain each image's synthetic responses and rank clean-correct images by their mean true-class probability under corruption. Across 44 classifiers, testing the highest-risk 20% finds 67% and 45% of failures under mild screen and print recaptures, versus 58% and 36% for clean confidence and 60% and 37% for an equal-size natural-transformation average. With both probability averaging and an A3Rank scoring adaptation, the tested corruption set yields higher mean failure recall than the natural-transform set; differences between scores depend on the source and budget. Together, these findings show that the value of correspondence depends on the evaluation objective: classwise counts suffice for specified reliability bounds, while image-specific synthetic responses improve the allocation of physical tests within the evaluated pool.

Yongda Wei, Chen Zhang, Yifei Wang +4

Image Corruption Robustness Adversarial Robustness

Ask PDF

Abstract

ML Reproducibility

Explore similar work

Benchmarking the Domain Gap: Model Selection Instability Under Domain Shift in Video Capsule Endoscopy

Re:CAP - Auditing Retrieval Coverage in Production RAG Pipelines

What Paired Evaluations Reveal under Visual Perturbations