Multimodal AI Detection for Math-Heavy Scientific Text

Yixuan Huang; Danish Pruthi; Bhuwan Dhingra

cs.CLOct 15, 2026

Multimodal AI Detection for Math-Heavy Scientific Text

Authors: Yixuan Huang, Danish Pruthi, Bhuwan Dhingra

Organizations: Duke University · Indian Institute of Science

Abstract

As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We release our data, models and code at https://nlplab.cs.duke.edu/snapjudge.

Figures & tables

Figure 1: The same mathematical expression shown as LaTeX source, PDF-extracted text, and a rendered image. We study AI detection across three representations and find that rendered images generally yield higher detection rates than extracted text and approach the performance obtained with the original LaTeX source.

	Train	Val	Test	Calib.
LaTeX	34.1k/30.2k	1.3k/2.2k	11.5k/12.2k	357.4k
Text	32.3k/30.1k	1.1k/1.9k	9.9k/11.9k	354.0k
Image	31.6k/30.0k	1.1k/1.7k	9.9k/11.8k	351.6k

Table 1: Dataset size: AI–human pairs (proofs / prose) per split, and human-written passages in the calibration pools.

Shallow baselines (features + logistic regression)
	LaTeX				Text
	Open avg.		Frontier avg.		Open avg.		Frontier avg.
	Prose	Proofs	Prose	Proofs	Prose	Proofs	Prose	Proofs
stylometric	0.030	0.003	0.014	0.000	0.014	0.002	0.006	0.000
char TF-IDF	0.558	0.394	0.288	0.100	0.480	0.317	0.238	0.145
word TF-IDF	0.664	0.584	0.359	0.308	0.559	0.485	0.247	0.238

Table 2: Detection rate at

0.1\%

FPR on the LaTeX source (left) and on inputs derived from the PDF (right), i.e, extracted text, OCR text ( OCR ) or the rendered page ( SnapJudge ). Open avg. and Frontier avg. average over the ten open-source and the five frontier models. When the LaTeX source is not available, SnapJudge is the most accurate detector, especially on frontier proofs. Owing to API cost, we evaluate Pangram on frontier models and its v3.3.2 only on LaTeX source.

Figure 2: Detection rate of our 27B detectors on frontier passages. The left panel tightens the FPR from

0.1\%

0.001\%

for proofs (solid) and prose (dashed), and Table 8 lists the values. Pangram 4 is shown after recalibration at

0.1\%

and with its own labels at

0.004\%

, the FPR reported in its technical report. The right two panels split the passages by math fraction at

0.1\%

FPR, where the dashed and dotted green lines show the LaTeX detector on the same passages with every math segment replaced by a placeholder or with the math removed. Pangram 4 is scored on the text as written.

Figure 3: Detection rate at

0.1\%

FPR on subsets defined by GPT-5.6 (left three panels) and mean score of human-written proofs (right). Numbers below bars count the passages.

	all 15 models in training		leave-one- model-out
9B detector	Proofs	Prose	Proofs	Prose
LaTeX	0.855	0.889	0.745	0.820
text	0.754	0.849	0.669	0.808
SnapJudge	0.880	0.847	0.812	0.806

Table 3: Generalization of the 9B detectors at

0.1\%

FPR.

TPR at 0.1% FPR	MIRAGE	Peer review	RAID	EditLens
Binoculars	0.242	0.540	0.771	0.194
Fast-DetectGPT	0.033	0.459	0.479	0.332
EditLens-Llama	0.995	0.964	0.849	1.000
EditLens-RoBERTa	0.851	0.684	0.537	0.995
Benchmark paper	0.777 ‡	0.765	0.927 ‡	1.000
Ours (text)	0.997	1.000	0.823	0.998

Table 4: Cross-domain evaluation on four public benchmarks. ‡ : at

5\%

FPR.

Detector	Spearman $\rho$
Binoculars	0.420
Fast-DetectGPT	0.345
EditLens-Llama	0.651
EditLens-RoBERTa	0.561
Ours (text)	0.624
SnapJudge	0.649

Table 5: Spearman

\rho

with the degree of AI editing.

Figure 4: Share of arXiv passages flagged as AI by the LaTeX detector (tuned to have a

0.01\%

FPR).

Appendix figures & tables20 assets

Supplementary material from the paper’s appendix.

Appendix

27B models	LaTeX	Text	SnapJudge	OCR	AI-fraction estimator
backbone	Qwen3.5-27B	Qwen3.5-27B	Qwen3.5-27B	Qwen3.5-27B	Qwen3.5-27B
input	LaTeX source, 512-token windows, stride 256	PyMuPDF text, 512-token windows, stride 256	rendered image, visual tokens of $32\times 32$ pixels	OCR text of the rendered image, 512-token windows, stride 256	rendered image
instruction	none	none	“Determine whether this mathematical proof was written by a human or generated by AI.”	none	“Estimate how much of this scientific page excerpt was written by AI.”
head output	2 classes	2 classes	2 classes	2 classes	5 buckets
LoRA rank $r$	8	8	16	8	16
learning rate	$10^{-4}$	$10^{-4}$	$2\times 10^{-4}$	$10^{-4}$	$2\times 10^{-4}$

Appendix

Table 6: Training settings of the 27B models behind our main results (top) and of the 9B detectors of Section 4.3 (bottom).

LaTeX
		Proofs					Prose
	rep.	GPT-4.1	GPT-5.1	o4-mini	Claude	Gemini	GPT-4.1	GPT-5.1	o4-mini	Claude	Gemini
Shallow baselines (features + logistic regression)
stylometric	LaTeX	0.000	0.000	0.000	0.000	0.000	0.026	0.002	0.001	0.008	0.030
char TF-IDF	LaTeX	0.227	0.030	0.093	0.065	0.088	0.661	0.119	0.145	0.098	0.418
word TF-IDF	LaTeX	0.404	0.268	0.333	0.292	0.243	0.729	0.229	0.211	0.164	0.460

Appendix

Table 7: Detection rate for each frontier family at

0.1\%

FPR on proofs and narrative prose; rows marked

0.01\%

and

0.001\%

FPR repeat our detectors at those FPRs.

		Proofs		Prose
Detector	FPR	Frontier avg.	Open avg.	Frontier avg.	Open avg.
LaTeX	0.1%	0.938	0.920	0.957	0.943
	0.05%	0.916	0.900	0.945	0.938
	0.01%	0.846	0.849	0.919	0.924
	0.005%	0.824	0.832	0.907	0.918
	0.001%	0.678	0.745	0.858	0.893

Appendix

Table 8: Values behind Figure 2 : detection rate of the 27B detectors at five false-positive rates, as the frontier models average plotted in the figure and as the average over the ten open-source models, together with the Pangram 4 values on the text layer plotted in the figure and the corresponding values on the LaTeX source;

0.004\%

is the false-positive rate that Pangram 4 reports for its own labels.

Figure 5: Detection rate at

0.1\%

FPR by passage length in words, frontier models average; Pangram 4 on the text input.

EditLens test set
	EditLens Llama	EditLens RoBERTa	Fast- DetectGPT	Binoculars	Ours, text	SnapJudge
AI-edited	0.593	0.382	0.025	0.009	0.204	0.249
MIRAGE
polish	0.706	0.254	0.002	0.015	0.439	0.393
rewrite	0.757	0.248	0.002	0.013	0.398	0.354
APT-Eval

Appendix

Table 9: Detection rate on AI-edited passages. Off-the-shelf detectors are held at

0.1\%

FPR; our detectors are at

0.01\%

FPR.

	Easy				Hard
	AI-BP	AI-EP	AI-HI	H-AI	AI-BP	AI-EP	AI-HI	H-AI	H
Pangram 3.0 †	100.0	100.0	100.0	3.0	97.0	99.3	92.6	3.1	0.0
GPTZero †	96.7	93.3	91.0	3.0	96.0	95.8	89.4	3.4	1.0
Ours, LaTeX	100.0	100.0	99.9	44.3	99.9	100.0	99.95	35.6	0.0
Ours, text	100.0	100.0	100.0	45.9	100.0	100.0	99.95	40.4	0.0
SnapJudge	100.0	99.6	99.9	21.7	98.3	98.3	96.0	15.0	0.0

Appendix

Table 10: Percentage of reviews classified as AI on the test split of the peer-review benchmark of Saha et al. (2026) . AI-BP, AI-EP and AI-HI are reviews written by an LLM, H-AI are human-written reviews polished by an LLM, and H are human-written reviews. Our detectors use the thresholds at a

0.005\%

false-positive rate on our calibration pools. † Reported by Saha et al. (2026) ,where a review counts as positive only when the entire review is labelled as AI-generated.

	Easy			Hard
H-AI flagged	LaTeX	Text	SnapJudge	LaTeX	Text	SnapJudge
$\leq 20\%$	98.6	99.0	100.0	97.1	98.7	98.0
$\leq 10\%$	92.9	90.2	98.8	67.2	65.5	87.6
$\leq 5\%$	74.5	50.3	92.0	29.9	16.9	66.1
$\leq 1\%$	8.1	0.8	56.6	5.1	1.9	12.0
$0\%$	0.1	0.0	12.3	0.4	0.1	0.5

Appendix

Table 11: Detection rate of reviews written by an LLM (AI-BP, AI-EP and AI-HI pooled) on the test split of the benchmark of Saha et al. (2026) , when the threshold of each detector is raised until at most the given share of the

2{,}802

polished reviews (H-AI, easy and hard pooled) is flagged.

Input	Passage	Model	Before	After	Pangram 4
LaTeX	Proofs	GPT-6 astra	0.395	0.789	0.084 (0.011)
		Claude Opus 5.5	0.429	0.684	0.122 (0.000)
	Prose	GPT-6 astra	0.465	0.530	0.230 (0.120)
		Claude Opus 5.5	0.415	0.420	0.250 (0.120)
Text	Proofs	GPT-6 astra	0.274	0.645	0.108 (0.022)
		Claude Opus 5.5	0.391	0.615	0.146 (0.021)

Appendix

Table 12: Detection rate on the two newest models at a

0.005\%

false-positive rate on our calibration pools. Before is the detector of the main paper, which has seen no data from these models (mean of two seeds), and After is the detector retrained with data from the new models added (one seed). For Pangram 4 we count a passage as detected when its own label is AI or mixed and give the share labelled AI in parentheses. Pangram 4 was run only on the AI-generated passages.

Detector	Proof	Prose
CNNDetection ( Wang et al., 2020 )	0.55	0.54
UnivFD ( Ojha et al., 2023 )	0.49	0.54
NPR ( Tan et al., 2024 )	0.42	0.47
SAFE ( Li et al., 2025 )	0.53	0.48
Community Forensics ( Park and Owens, 2025 )	0.51	0.49
SnapJudge (ours)	0.99	0.99

Appendix

Table 13: AUROC of AI-generated image detectors and of our detector on the rendered images.

month	$n$	prose	proofs	$n$	prose	proofs	$n$	prose	proofs
	Mathematics			Computer science			Other fields
2023-01	626	0.0	0.2	600	0.5	0.0	322	0.3	0.0
2023-02	583	0.0	0.2	600	0.5	0.0	304	0.0	0.0
2023-03	638	0.0	0.0	600	0.8	0.0	381	0.5	0.0
2023-04	645	0.2	0.2	600	0.7	0.2	299	0.7	0.3
2023-05	566	0.2	0.2	600	0.8	0.0	367	0.3	0.0

Appendix

Table 14: Monthly data behind Figure 4 . For each field,

n

is the number of papers, and the prose and proof columns give the share (%) of passages detected as AI by the LaTeX detector at

0.01\%

FPR.

	Prose			Proofs
year	math	CS	other	math	CS	other
2018	0.0	0.0	0.0	0.0	0.0	0.0
2019	0.0	0.0	0.0	0.0	0.0	0.0
2021	0.0	0.0	0.0	0.0	0.0	0.0
2022	0.0	0.1	0.1	0.0	0.0	0.0
2023	0.3	1.0	0.6	0.1	0.1	0.0

Appendix

Table 15: Share (%) of passages detected as AI by the LaTeX detector at

0.01\%

FPR, by year and field.

Figure 6: Detection rate of SnapJudge on frontier-model test passages rendered at decreasing resolution.

Year	submissions	with proofs	without	$\Delta$ ( $z$ )	with proofs	without	$\Delta$ ( $z$ )
		mean review score			acceptance rate (%)
2020	2,213	4.64	4.34	$+0.30$ (3.8)	36.5	29.1	$+7.4$ (3.3)
2021	2,594	5.65	5.46	$+0.19$ (4.4)	35.9	32.1	$+3.8$ (1.8)
2022	2,617	5.77	5.41	$+0.36$ (7.0)	47.2	39.3	$+7.9$ (3.8)
2023	3,792	5.68	5.42	$+0.26$ (5.4)	46.7	40.1	$+6.6$ (3.4)
2024	7,404	5.44	5.05	$+0.39$ (11.8)	46.0	36.2	$+9.8$ (6.7)

Appendix

Table 16: Review scores and acceptance rates of proof-carrying versus other ICLR submissions, 2020–2026.

\Delta

is the difference between proof-carrying and other submissions, with the two-sample

z

statistic in parentheses;

|z|>1.96

corresponds to significance at the

5\%

level (two-sided).

The nine high-proof areas (share $\geq 25\%$ in 2026)
	share of submissions with proofs
Primary area	2024	2025	2026
learning theory	69.0%	71.5%	78.4%
causal reasoning	49.5%	56.3%	58.5%
optimization	50.6%	57.8%	57.2%
probabilistic methods (Bayesian methods, sampling, UQ)	40.6%	42.4%	51.6%

Appendix

Table 17: Proof-carrying submissions by research area, 2024–2026.

Generator	Proof passages	Prose passages	Cost (USD)
GPT-4.1	1,477	1,244	$\approx$ 90
GPT-5.1	1,487	1,600	$\approx$ 85
o4-mini	801	1,600	$\approx$ 60
Claude Sonnet 5	1,053	1,147	$\approx$ 70
Gemini 3.1 Pro	1,040	1,045	$\approx$ 95
Total	5,858	6,636	$\approx$ 390

Appendix

Table 18: API cost of the frontier-model passages used in the dataset, and of the LLM judge.

	Flawed proofs (17)	Control proofs (13)
Mean severity	7.08	4.23
Standard deviation	2.46	2.20
Median	8.0	4.0
Severity $\geq 5$	12/17 (71%)	2/13 (15%)

Appendix

Table 19: Severity scores from 0 to 10 that GPT-5.6 Sol assigns to proofs from withdrawn ICLR 2026 submissions whose reviewers pointed out a flaw, and to control proofs that reviewers described as correct, averaged over the judge’s samples for each proof. For this check the judge uses the scoring prompt given below. A flawed proof receives a higher score than a control proof with probability

0.81

(one-sided Mann-Whitney test,

p=0.0016

Figure 7: Reliability of the 27B detectors.

Our test sets
	Text		SnapJudge		LaTeX
Score range	score	AI share	score	AI share	score	AI share
0–0.05	0.001	0.032	0.012	0.017	0.001	0.026
0.05–0.2	0.113	0.255	0.096	0.102	0.115	0.189
0.2–0.35	0.282	0.351	0.267	0.203	0.282	0.234
0.35–0.5	0.443	0.478	0.426	0.295	0.450	0.489

Appendix

Table 20: Values behind Figure 7 .

true $\backslash$ pred.	1	2	3	4	5	MAE
1 $(0,.25)$	91.4	6.5	1.1	0.2	0.8	0.077
2 $[.25,.5)$	2.8	91.6	4.8	0.2	0.6	0.058
3 $[.5,.75)$	0.5	5.1	89.0	3.8	1.6	0.062
4 $[.75,1)$	0.5	0.5	4.8	72.4	21.8	0.055
5 (AI)	0.1	0.2	1.0	0.6	98.1	0.023
all						0.055

Appendix

Table 21: The AI-fraction estimator on 5,000 framed test windows.

Figure 8: A recompiled page of a 2021 test paper in which an AI-generated proof (dashed red box, $p=1$ ) is surrounded by the human-written text of the paper. The dashed blue boxes are two windows opened by the sampler on this page, each fully covering the AI passage together with part of the human-written context; their AI fraction $p$ is the height of the AI text divided by the height of all text inside the window.

AI-Generated Text Detection

AI-generated text detection develops methods to distinguish human-written, machine-generated, and AI-edited text.

See all 170 papers

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
Jun 13, 2026Jierui Zhang, Siyuan Tan, Xinhang Li +8Mathematical Reasoning Benchmarks LLM Evaluation
School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · Graduate College for Engineers Beijing University of Posts and Telecommunications Beijing, China · School of Mathematical Sciences Fudan University Shanghai, China +6
Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery
Jun 7, 2026Syed Rifat Raiyan, Mohsinul Kabir, Hasan Mahmud +2Mathematical Reasoning Benchmarks Automated Theorem Proving
Systems and Software Lab (SSL), Department of Computer Science and Engineering, Islamic University of Technology, Gazipur, Bangladesh · Department of Computer Science, University of Manchester, Manchester, United Kingdom
SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity
Jun 14, 2026Yifan Mo, Xiao Fu, Yue Su +4AI for Science LLM Evaluation
Vrije Universiteit Amsterdam, Amsterdam, Netherlands · Wageningen University & Research, Netherlands

Jun 13, 2026J/K move · Enter open · S save

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

Jierui Zhang, Siyuan Tan, Xinhang Li, Longzhuangzhi Lin+7

School of Computer Science Beijing University of Posts and Telecommunications Beijing, China · Graduate College for Engineers Beijing University of Posts and Telecommunications Beijing, China · School of Mathematical Sciences Fudan University Shanghai, China+6

Mathematical Reasoning Benchmarks LLM Evaluation Automated Theorem Proving Training Data Curation LLM Mathematical Reasoning

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources. This evaluation gap limits trustworthy AI assistance in proof-certified scientific progress. Existing evaluations often emphasize final answers or rely on costly expert grading, while end-to-end proof generation remains open-ended and hard to verify automatically. We introduce Mask-Proof, a pipeline that turns real proofs into automatically checkable masked-step tasks. It masks key formula steps, provides the necessary surrounding context, and evaluates model reconstructions with an LLM-based equivalence judge using repeated votes for stability. The resulting Mask-ProofBench contains 292 curated problems across diverse research areas. Experiments with 17 models show that reasoning-enhanced models outperform standard models by 12% to 27%. Our evaluator achieves 96.8% agreement with expert annotators, enabling faithful, reproducible, and comparable measurement of step-level mathematical reasoning. Benchmark, annotations, and code are available at https://github.com/weating/Mask-Proof.

Abstract

AI-Generated Text Detection

Explore similar work

Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs

Artificial Intelligence for Mathematical Reasoning: An Integrated Survey of Language Models, Neuro-symbolic Systems, and Verified Discovery

SciText2Eq: Assessing LLMs for Explainable Equation Generation for Scientific Creativity