A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion

Palvika Bansal; Rittika Jindal; Ruichong Wang

cs.CLOct 14, 2026

A Strong Harness Narrows the Model Gap: A Comparative Evaluation for Legislative Document-to-XML Conversion

Authors: Palvika Bansal, Rittika Jindal, Ruichong Wang

Organizations: Thomson Reuters Technology

Abstract

Legal-content reproduction requires matching the source exactly in section hierarchy, verbatim text, citations, and amendment spans; a dropped citation or mislabeled amendment carries into the products lawyers rely on. We report an industrial pipeline converting U.S. state legislation into strict, schema-constrained XML, and hypothesize that its accuracy comes from harness engineering, the deterministic tooling built around a model, more than the model itself. Our controlled ablation crosses three harnesses (zero-shot, a single tool-using agent, and a decomposed multi-agent pipeline) with three Claude models on 153 gold bills from two jurisdictions. The harness narrows the model gap: the weakest model in the full pipeline beats the strongest model alone (0.948 versus 0.806, paired bootstrap p less than 0.001). Because the two agentic harnesses share identical tools, their gap comes from orchestration: decomposition plus the persisted state and per-agent context it brings. Across four scored dimensions, the model spread at zero-shot (0.68 to 0.81) narrows to 0.95 at the full pipeline; the residual sits almost entirely in citation normalization, where the canonical identifier is retrieved from an external authority rather than generated from the weights, so a deterministic lookup tool supplies the exactness that a larger model does not.

Figures & tables

Figure 1: The three harnesses, all starting from the same bill PDF and ending at the same publication XML. Because tool access is held constant between T2 and T3, the difference between them isolates orchestration: decomposition, the persisted state it enables, and the per-agent context it creates.

	T1	T2	T3
Bill PDF (native)	yes	yes	yes
XML templates	yes	yes	yes
Jurisdiction rules	yes	yes	yes
Step-ordered playbook	none	9 steps	21 steps
Tools	none	47	17
Sub-agents	none	none	7

Table 1: What each harness receives. Rows above the middle rule are knowledge and are held constant; rows below it are harness . T2 and T3 reach the same underlying tools: T2 calls the 47 leaf tools directly, while T3 reaches those same 47 leaves through 17 composite orchestrator tools—the counts differ in exposure, not in access. The step counts differ because T2’s playbook stops at the scored artifact, while T3’s continues into the publication stages that follow it.

			Structure & text			Citations
Model	Harness	Overall $\uparrow$	NED $\uparrow$	TEDS $\uparrow$	Amend. $F_{1}$ $\uparrow$	Local. $\uparrow$	Norm. $\uparrow$
Haiku 4.5	T1	0.677 [.654,.700]	0.846	0.647	0.288	0.206	0.024
	T2	0.872 [.816,.921]	0.882	0.869	0.808	0.367	0.103
	T3	0.948 [.932,.963]	0.982	0.946	0.876	0.813	0.801
Sonnet 4.6	T1	0.807 [.785,.829]	0.872	0.794	0.598	0.185	0.282
	T2	0.922 [.899,.943]	0.948	0.915	0.871	0.869	0.881

Table 2: Main results: per-dimension and composite exactness for all nine model

\times

harness cells, pooled across both jurisdictions (micro

F_{1}

for citations and amendments, higher is better throughout). Every model climbs monotonically from the zero-shot baseline (T1) to the full pipeline (T3), and the full-pipeline rows are shaded. Overall is the unweighted mean of NED, TEDS, and amendment

F_{1}

. The two citation columns are reported separately and excluded from it (§ 5 ). Because it averages per document over a slightly different population than its displayed columns, Overall need not equal their row-wise mean (see footnote). Bracketed ranges beneath Overall are 95% bootstrap confidence intervals (§ 5 ). 7 7 7 Overall need not equal the mean of this table’s own NED/TEDS/Amend.

F_{1}

columns: it is computed per document then macro-averaged, and each column pools over a different document subset (TEDS drops the two capped documents; the displayed Amend.

F_{1}

is the applicable-only variant, whereas Overall uses the all-documents variant). The gap is largest at zero-shot; Appendix D works the arithmetic through.

Figure 2: Overall exactness rises with the harness for every model. Unaided, the models spread widely, but inside the full system they converge.

Figure 3: Citation normalization is near-zero for every model unaided and is delivered by the harness. The stronger models resolve citations as soon as they have the tool (T2). The weakest model does not, and reaches parity only under full decomposition (T3).

Harness	Pooled std $\downarrow$	Low-score ( $<0.70$ ) $\downarrow$
T1	0.167	31%
T2	0.188	8%
T3	0.099	4%

Table 3: Pooled variance and rate of low-scoring documents (

<0.70

), by harness. The full pipeline (T3) both tightens the distribution and shrinks the tail.

	Idaho			New Jersey
Harness	Haiku	Sonnet	Opus	Haiku	Sonnet	Opus
T1	0.695	0.833	0.860	0.636	0.754	0.697
T2	0.917	0.972	0.964	0.783	0.816	0.798
T3	0.974	0.977	0.976	0.879	0.871	0.809

Table 4: Overall score by jurisdiction and harness; New Jersey is harder under every harness, yet the harness’s advantage over the unaided model is wider there, not narrower. Full-pipeline (T3) cells shaded. Each cell averages over its own scored documents, so the jurisdiction columns combine into Table 7 ’s Overall weighted by those counts, not by the cohort totals.

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

		Body nodes			Length (chars)
Cohort	$n$	med	p90	max	med	p90	max
All	153	130	652	40,944	5.8K	42K	1.16M
Idaho	103	97	366	1,955	4.0K	23K	42K
New Jersey	50	187	1,744	40,944	9.4K	74K	1.16M

Appendix

Table 5: Corpus size distribution over the 153-document gold set (103 Idaho, 50 New Jersey). Documents vary in breadth, not depth. XML tree depth is

8

for nearly all documents (max

9

), fixed by the shallow schema hierarchy, so we report body-node and character-length distributions. New Jersey is larger than Idaho at every percentile. “Length” is the character count of the bill body (K/M denote thousands/millions).

Figure 4: The T3 harness, modeled on the production pipeline (the most-scaffolded harness in Figure 1 ). The ablation runs it with a single uniform model across every agent, unlike the deployed system’s mixed configuration (see the Limitations section). An initializer routes each input to the workflow agent, which reads an editable playbook and runs a six-stage task-agent spine (document extraction, structure and auxiliary-artifact construction, statute-impact extraction, XML conversion, citation resolution and footnoting, validation). The initializer and these six stage agents are the seven sub-agents counted in Table 1 . Stages call domain services for storage, taxonomy, ID-mapping, NER, codes lookup, and distribution, and editorial review gates publication after validation. T2 exposes this same tool set to a single agent, and T1 uses none of it.

Figure 5: State handling in the two agentic harnesses. In T2 (left) a single agent holds and re-threads all intermediate state inside one long context with no checkpoint, so a failure loses the run’s progress. In T3 (right) state is passed explicitly between agents through a persisted execution store, so a run can pause and resume at asynchronous checkpoints and a stalled step recovers.

Figure 6: The five dimensions across the nine cells (exact values in Table 7 ). Text and structure (NED, TEDS) are already high at T1 and flatten quickly; amendments improve mostly on the T1 $\rightarrow$ T2 jump; citations are the one dimension that stays low until the full pipeline, and the one where Haiku visibly lags Sonnet and Opus.

Figure 7: Per-document Overall score distribution across the nine configurations, the variability behind the pooled statistics in Table 3 . Variance is non-monotonic: the single-agent harness (T2) is the widest, and only the full pipeline (T3) collapses both the spread and the low-scoring left tail.

Bill	Harness	Gold	Pred	NED	Amend. $F_{1}$
SB_2025	T1	40,944	1,087	0.10	0.00
SB_2025	T3	40,944	22,239	0.87	0.92
AB_4700	T1	8,229	334	0.12	0.00
AB_4700	T3	8,229	22,146	0.95	0.97

Appendix

Table 6: Stress-bill collapse and recovery, T1 vs. T3, both scored on Sonnet, the only model with a completed run on these two documents (so the pattern is not verified to generalize to Haiku or Opus). TEDS is unscored under the node cap, but every scorable dimension moves from abandoned to solved. Predicted node counts exceed gold at T3 because the pipeline’s serialization further subdivides interior structure, such as clause, citation, and amendment spans, into more atomic XML nodes than the gold annotation uses. NED and amendment

F_{1}

score character- and span-level content rather than node count, so this inflation does not by itself depress those scores.

Failure mode	Docs	Example
Dropped / partial subset	23	NJ SB 3587
Granularity / neighbor id	3	ID SB 1023
Lookup miss (left unresolved)	2	ID SB 1142
Range over-expansion	1	ID SB 1008
Localization text-capture artifact	3	NJ AB 5729

Appendix

Table 7: Qualitative taxonomy of the T3 citation-normalization residual: the 32 of 167 outputs containing at least one cite.query node (pooled across the three T3 models) with any imperfect resolution, grouped by failure mode with a representative bill and assigned from a diff of predicted against gold cite.query resolutions (§ 6.3 ). The dropped-subset and lookup-miss modes are coverage-limited, meaning the codes service returns no identifier. The range over-expansion is a single Idaho outlier that on its own inflates that jurisdiction’s raw residual. The final row is not a resolution error: the identifier resolves correctly, but the captured visible span differs from gold.

Legal NLP

Legal natural language processing (NLP) develops methods for analyzing and generating statutes, cases, contracts, and legal advice.

See all 109 papers

Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority
Oct 21, 2026M. Mikail DemirLegal Citation Verification LLM Abstention
Department of Information Science, University at Albany, SUNY, USA
LegalCiteBench: Evaluating Citation Reliability in Legal Language Models
May 11, 2026Sijia Chen, Hang Yin, Shunfan ZhouLLM Evaluation Legal IR
Northeastern University · Phala
The GELATO Dataset for Legislative NER
Mar 14, 2026Matthew Flynn, Timothy Obiso, Sam NewmanNamed Entity Recognition Legal NLP
Brandeis University

Oct 21, 2026J/K move · Enter open · S save

Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority

M. Mikail Demir

Department of Information Science, University at Albany, SUNY, USA

Legal Citation Verification LLM Abstention

Open paper AskPDF arXiv

Given the enormous number of claims to be checked, verifiable legal AI requires scalable, cost-effective models that answer the practical needs of law. System One decision models are built on large language models but, instead of generating text, return a calibrated probability for each option. This suits the verification of legal authority, where the question is closed: a given holding, or none of them. We test whether such models abstain when the correct authority is missing. We extend CaseHOLD with an abstention condition in which the correct holding is replaced in 1,670 of 5,314 items, and compare four decision models with six generative LLMs. When no option is correct, generative LLMs still pick one in 38-86% of items; the most cautious decision model does so in 30%. On standard CaseHOLD, a decision model matches the best generative LLM (88.1% vs. 88.2%) at 1/55 of the cost. Accepting that LLM's answer only when a decision model agrees halves its wrong citations, and distilling a decision model's probabilities into Qwen3-8B, without labels, yields a well-calibrated local model. Legal AI should be judged by its wrong citations, not by accuracy alone.

Abstract

Legal NLP

Explore similar work

Commit or Abstain? System One Decision Models, LLMs, and Missing Legal Authority

LegalCiteBench: Evaluating Citation Reliability in Legal Language Models

The GELATO Dataset for Legislative NER