Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem

Divya Godara; Sachin Gupta

cs.CLSep 30, 2026

Evidence First, Arithmetic Second: A System Report and Failure Analysis for DocSem

Authors: Divya Godara, Sachin Gupta

Organizations: Independent Researcher San Jose, USA

Abstract

EVICALC, our system for the DocSem shared task, achieved 8.61% joint accuracy on 1,730 tasks in the official final test evaluation. It reads a PDF, selects a passage, asks a language model to write an arithmetic expression, and evaluates that expression in local code. Saved intermediate results support inspection of failures. A separate public-validation run achieved 92.17% answer accuracy and 1.00 evidence F1. The configurations and metrics differ, so these scores are not a controlled comparison. Our manual, post-hoc analysis is descriptive: in one inspected case, optical character recognition (OCR) and block grouping merged the relevant passage into another block, and the system answered from unrelated text. An exploratory study of reading page images on 100 documents returned evidence identifiers for only 22 documents. These descriptive findings motivate further evaluation; they do not establish the causes of the overall score.

Figures & tables

Figure 1: Submitted adaptive v6 pipeline. Blue marks inputs, purple model calls, teal selection/calculation and output, amber conditional retries, and gray records and diagnostics. Arithmetic expressions are evaluated in local code. Crops come from the selected passage in the supplied PDF. The v5/v6 passes change only answers: unflagged rows and failed retries retain the prior answer (Section 3.3 ). Dashed arrows lead to saved records and optional checks, not a required verification step. Sonnet denotes claude-sonnet-5 .

Run	Answer	Evidence F1
Two model samples	90.32	1.00
Combined provider outputs	90.78	1.00
Prompt with problem-type checks	92.17	1.00
Reading rule on saved outputs	98.62	1.00

Table 1: Recorded public-validation feedback (percent except evidence F1). The first three rows came from PDF-based runs; the last recomputed saved outputs.

Measure	Result
Stories selected	55 / 100
Correct story among selections	about 49 / 55
Evidence emitted	22 / 100
Exact badge among emissions	about 20–21 / 22

Table 2: Exploratory page-image study on 100 documents. Output counts are exact; correctness estimates are approximate local judgments, not organizer labels.

Document QA

Document question answering uses AI to answer questions from text, tables, images, and structured content in documents.

See all 93 papers

When Harness Beats Scale, and When Reading Beats Both
Sep 28, 2026Ivan Bondarenko, Nikolay O. NikitinNumerical Reasoning in Language Models Language Model Scaling
Novosibirsk State University · ITMO University
Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets
Jul 27, 2026Zongyou Yang, Yinghan HouLLM Evaluation LLM Answer Verification
Dyson School of Design Engineering Imperial College London London, United Kingdom · Department of Electrical and Electronic Engineering Imperial College London London, United Kingdom
CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
May 13, 2026Dongsheng Ma, Jiayu Li, Zhengren Wang +9VLM Evaluation Citation Verification
Peking University · Shanghai Artificial Intelligence Laboratory

Sep 28, 2026J/K move · Enter open · S save

When Harness Beats Scale, and When Reading Beats Both

Ivan Bondarenko, Nikolay O. Nikitin

Novosibirsk State University · ITMO University

Numerical Reasoning in Language Models Language Model Scaling Document QA

Open paper AskPDF arXiv

We describe our system for DocSem, the document-grounded quantitative reasoning shared task at DocInsights 2026, and analyze why it succeeded on labeled data and failed on the test set. The pipeline pairs hybrid block retrieval with Program-of-Thoughts (PoT) generation executed in a sandboxed interpreter, self-consistency sampling, and entity enrichment from chunk-level knowledge graphs. On our held-out split, application architecture moved the metrics far more than model scale did: PoT added 0.282 joint accuracy to a compact 7B model but at most 0.005 to a 72B model, and a 27B model with the full harness matched the 72B (0.884 vs.\ 0.873) at roughly 2.7

\times

fewer parameters and a quarter of the CO

_2

. We read this through a distinction between world knowledge, which scales steeply with parameters, and language knowledge, which scales gently, and show that structured-output training makes a compact model harness-ready rather than merely small. On the raster, watermarked test PDFs the same system collapsed to 13.58% joint (rank 149 of 163); a controlled re-rendering of the validation set reproduces the OCR half of the collapse while bounding what the simulation misses. Auditing the physical nature of evaluation inputs precedes architecture, and the leaderboard's bimodality is consistent with reading quality, not reasoning, having separated the field.

Abstract

Document QA

Explore similar work

When Harness Beats Scale, and When Reading Beats Both

Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets

CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence