Estimating Uncoded Crash Factors with Tabular Foundation and System One Models: Kumo Tabular and Jev

Amir Rafe; Subasish Das

stat.APOct 7, 2026

Estimating Uncoded Crash Factors with Tabular Foundation and System One Models: Kumo Tabular and Jev

Authors: Amir Rafe, Subasish Das

Organizations: Texas State University, San Marcos, USA

Abstract

Road safety programs count the coded fields of police crash records, while the officer's narrative, which often records factors the fields omit, is rarely read. A safety office thus cannot tell how much its counts miss or where to review. This study develops and evaluates a system that joins both views of the 5,601,890 Texas crashes from 2017 to 2025 into population estimates with stated validity. An in-context tabular foundation model, Kumo Tabular, reads the coded record of every crash, a calibrated System One model, Jev, reads the narratives of two probability samples, and human judgments recalibrate its probabilities. A multiwave predict-then-debias estimator joins the three tiers, and a second human tier drawn with recorded probabilities checks the estimates by design. For hydroplaning, medical episodes, fatigue, animals, and phone use, the narrative documents more injury crashes than the coded field, 15,074 against 7,340 for phone use, and the human check agrees with all fifteen estimates within its margin. A re-read list ranked by Kumo Tabular finds confirmed discordance 7 to 58 times as often as random reading. At the planning cost of human coding, one further round of human judgments would cut the root mean square relative half-width from 22.0 to 16.2 percent, against 21.2 for reading every narrative. Two calibrated readers of different views, joined by a sampling design, give a safety office counts, a discordance map, a validated re-read list, and a reading budget, with Kumo Tabular reading the table at 15 times the speed of TabPFN 3.5.

Figures & tables

Panel of Table 1 — Table 1. What prior studies did and what this study does.

Figure 1. The population and its coverage. Panel (a) gives the size of each phase on a log scale. Panel (b) splits each year’s crashes into the narrative population

U

, crashes with a narrative but no coordinates, and crashes whose narrative is empty or short, with column width proportional to the crashes of that year. Panel (c) gives the same split by severity on the KABCO scale, from fatal (K) to no injury (O). Both share axes start at 75 percent.

Phase	Units	Size	Reader	Measurement	Design information
Phase 0	Every crash	5,601,890	Kumo Tabular	$\hat{f}_{i}$ for every factor	Complete enumeration
Phase 1, wave 1	Random stratum of $U$	150,000	Jev	$p_{i}$ , recalibrated to $\tilde{q}_{i}$	$\pi^{(1)}=n_{1}/N$ , simple random sample
Phase 1, wave 2	$U$ outside wave 1	79,510	Jev	$p_{i}$ , recalibrated to $\tilde{q}_{i}$	Recorded $\pi^{(2)}_{i}$ , Poisson draw on the discordance score
Phase 2	Human-judged narratives	400 (2,309 judgments)	Two or three coders	$y_{i}$	Calibration weights, model-dependent
Check	Wave-1 narratives outside phase 2	600 (427 read by a specialist)	Three coders	$y_{i}$	Recorded inclusion probabilities, Poisson draw, design-based

Table 2. The phases of the design and what is known at each.

Figure 2. The two-view design. Each column of cells is one crash, sorted by the table reader’s discordance score, and the cells show which crashes each reader and the human coders touch. The cells are schematic, and the counts are design facts.

Figure 3. Predictability of the narrative flag from the coded record against the share of narrative positives that the coded field misses. Panel (a) gives the factors without a coded field and panel (b) those with one. Each comet runs from the AUC of a booster on sixteen coded factor indicators (tail) to the AUC of Kumo Tabular on the full coded record (head), and the head grows with the number of flagged wave-1 narratives, as the key in panel (b) shows. The lines at an AUC of 0.85 and at half of the narrative positives divide the plane into the labeled regions.

		AUC for the narrative flag			Narrative surplus
Factor	Flags	Sixteen	Full	Kumo	Field misses	Confirmed misses	AUC, field false
Factors with a coded field
Unbelted †	933	0.64	0.73	0.77	80%	37%	0.69
Wrong-way	2,199	0.69	0.80	0.81	75%	44%	0.72
Phone use †	1,562	0.72	0.88	0.89	66%	65%	0.81
Medical episode	1,608	0.80	0.89	0.89	61%	47%	0.78

Table 3. Predictability of the narrative reading from the coded record, and the narrative surplus.

	Narrative states the factor				Second human tier
Factor	Estimate	95 percent interval	Sensitivity	Coded field	Human check	95 percent interval
Transported	224,996	208,813 to 234,969	-1,062	none	241,450	216,770 to 266,130
Hit-and-run	143,701	135,642 to 164,848	-896	none	102,984	4,368 to 201,600
Witness cited	135,504	115,809 to 152,568	-252	none	157,977	132,589 to 183,364
Alcohol	65,971	60,078 to 73,566	-699	73,809	69,802	65,188 to 74,416
Vehicle defect	59,396	35,923 to 60,699	-2,087	none	61,595	45,927 to 77,262

Table 4. Injury-or-fatal crashes in

U

whose narrative states each factor, 2017 to 2025, with the design-based human check.

Figure 4. Model estimates, their design-based human check, and the sources of uncertainty for injury-or-fatal crashes, 2017 to 2025. Panel (a) gives, on a log scale, the model estimate with its 95 percent interval (amber), the human check of the second human tier with its interval (open circle), and the coded-field count (diamond) where one exists. Panel (b) splits the half-width of each model estimate by variance share into the design tier and the recalibration tier, with the recalibration share printed. † Pooled recalibration map.

Figure 5. Injury-or-fatal crashes per year whose narrative states each factor. Panel (a) gives the narrative estimate over the coded-field count, in bold where the whole 95 percent interval lies on one side of parity, and the trend column draws the two annual series. Panel (b) gives the estimates in thousands for the factors without a coded field. The white seam marks the 2020 change of the crash report. † Pooled recalibration map.

	Statewide			Setting		Road system		Year
Factor	Crashes	Interval	Rate	Rural	Urban	State	Local	2019	2021
Alcohol	37,061	26,713 to 51,177	7.4	8.4	7.0	6.9	8.0	7.7	8.5
Hydroplane	34,198	30,463 to 38,845	6.8	8.6	6.2	8.6	4.7	7.0	7.7
Wrong-way	33,418	16,391 to 51,926	6.7	6.3	6.8	5.9	7.5	6.8	6.7
Phone use †	26,754	22,611 to 34,425	5.3	5.7	5.2	5.3	5.4	4.8	5.5
Medical episode	25,979	16,476 to 31,405	5.2	5.5	5.1	5.1	5.3	4.4	5.2

Table 5. Confirmed discordance between narrative and coded field, Texas 2017 to 2025.

Figure 6. Where confirmed discordance concentrates. Panel (a) gives, for each pair of domains, the ratio of their discordance rates per 1,000 crashes, in bold where the paired 95 percent interval of the ratio excludes one. The pairs are rural against urban, the state system against local roads, interstates against city streets, fatal crashes against crashes without injury, and 2021 against 2019. Panel (b) gives, for phone use, the rate of each county with at least 10,000 crashes that passes the suppression rule, with its 95 percent interval, in vermilion where the interval excludes the statewide rate. † Pooled recalibration map.

Figure 7. The re-read list, measured on the second-wave sample. Panel (a) gives, for each factor, the lift over random reading at the top one, five, and twenty percent of the unread narratives, with the 95 percent interval at one percent. Panel (b) gives the records confirmed discordant per hundred read from the top one percent of the list, with its interval, and in a random reading of the same length. † Pooled recalibration map.

Figure 8. Iso-precision maps of the reading budget. Each panel spans spending on narrative reads against spending on human-coded narratives, shaded by the half-width of the 95 percent interval relative to the estimate. The green curve is the cheapest path to each precision, and the markers are the current design (dot), 400 more human-coded narratives (triangle), and reading every narrative (square), and the grey band lies beyond full coding. The cost of a human-coded narrative is a planning value of 0.60 dollars, six judgments at ten cents each from the coder throughput of the phase-2 human tier.

		AUC		Calibration			Cost
Reader	Context	Flags	Discordance	Error	Slope	$\rho^{2}$	Rows per s	GPU GB
Kumo Tabular, small	60,000	0.865	0.759	0.002	1.01	0.279	8,393	12.1
Kumo Tabular, medium	30,000	0.861	0.741	0.002	1.01	0.280	2,872	66.6
Kumo Tabular, large	10,000	0.853	0.718	0.003	1.03	0.269	3,632	45.4
TabPFN 3.5	60,000	0.872	0.772	0.002	1.02	0.286	556	3.0
TabPFN 3	60,000	0.859	0.746	0.004	0.95	0.278	1,004	1.7

Table 6. The table readers on the same test rows.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Symbol	Meaning
$U$ , $N$	Geolocated crashes with a narrative of more than forty characters, and their number
$x_{i}$	The coded record of crash $i$ (97 features)
$v$	A factor, such as alcohol or unbelted occupants
$c_{vi}$	The coded-field indicator for $v$ , defined for the nine factors with a coded counterpart
$p_{vi}$	The narrative reader’s probability that the narrative states $v$
$\tilde{q}_{vi}$	The recalibrated probability $g_{v}(p_{vi},c_{vi})$ , the phase-1 measurement

Appendix

Table B.1. Notation used in the estimator and the evaluation.

			Held-out Brier		Flags confirmed
Factor	Map	Positives	Raw	Chosen	Field true	Field false
Aggression	Platt	31	3.69	1.75	none	none
Alcohol	Platt, coded shift	25	4.56	2.71	100% (17)	71% (13)
Animal	Platt	23	0.57	0.35	100% (22)	none
Drugs †	Isotonic	8	1.20	0.00	100% (7)	52% (2)
Fatigue	Platt	27	0.84	0.42	96% (18)	89% (12)

Appendix

Table B.2. Recalibration maps and the confirmation of Jev flags by coded-field status.

Target	Kumo small	TabPFN 3.5	TabPFN 3	TabICLv2	CatBoost	CatBoost pool
Narrative flags
Alcohol	0.963	0.964	0.960 ∗	0.959 ∗	0.957 ∗	0.958 ∗
Drugs	0.900	0.910	0.905	0.903	0.898	0.905
Hit-and-run	0.932	0.935 ∗	0.930 ∗	0.928 ∗	0.927 ∗	0.931
Medical episode	0.869	0.879 ∗	0.871	0.869	0.858	0.864
Phone use	0.891	0.892	0.888	0.888	0.884	0.883

Appendix

Table D.1. AUC of each table reader on each target.

Explore similar work

Sep 21, 2026cs.CL

Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)

Crash datasets that carry an investigator narrative hold information the coded fields omit. Coding those narratives at scale has been blocked by three obstacles. Frontier large language models are costly at that scale, their generated text cannot be verified, and no rule says how much output a human must check. This paper formulates narrative coding as gated, typed decisions answered by Jev, a System One model that returns probabilities over analyst-defined options and generates no text. A screen covered 499,500 Texas narratives and 195,857 were coded with a 27-question schema. Cost is governed by schema size rather than narrative length. The probabilities are audited against coded fields and against 2,416 blinded human judgments drawn under a stated sampling design. Two frontier large language models are benchmarked on the same records. Against human labels the typed model attains an F1 of 0.908. One frontier model gains 0.059 and the other is indistinguishable from it. Calibration varies by model rather than by paradigm, so each model must be audited. Recalibration on the same labels reduces calibration error by a factor of 3.3. Agreement with coded fields understates fidelity to the narrative by a median of 0.26 in kappa. A resolution-floor bound covers any model that reports probabilities on a discrete grid. A review budget over flagged records gives the records a human must read per variable and per year. Adding the calibrated variables to the coded fields raises the injury and fatal crashes attributed to nine factors by 10,747 per year.

Amir Rafe, Subasish Das

Ingram School of Engineering, Texas State University, San Marcos, TX, USA

LLM Auditing Language Model Calibration

Ask PDF

Jul 31, 2026cs.LG

Benchmarking Frontier Large Language Models Against Official Crash Database Coding Using Police Crash Narratives

Police crash narratives contain information that may supplement structured crash databases, but manual review is labor-intensive and it remains unclear how well large language models (LLMs) reproduce official crash coding. This study benchmarked six frontier LLMs by comparing narrative-derived crash attribute codes with corresponding fields in the Arkansas fatal-crash database. The analysis linked 5,587 fatal-crash narratives with 5,889 structured crash records from Arkansas (2015-2025), yielding 4,194 matched crashes. Six LLMs were evaluated using an identical zero-shot prompt to code crash manner, non-motorist relation, intersection type, work-zone relation, roadway surface condition, and light condition. Performance was evaluated using agreement, macro-averaged F1 score, Cohen's kappa, coverage, selective agreement, and comparisons with always-majority, always-Unknown, and keyword-rule baselines. Repeated-measures analyses and a generalized estimating equations model assessed differences among models and attributes. GPT-5.5 High achieved the highest agreement among the evaluated LLMs, but the always-majority baseline produced higher raw agreement and the keyword-rule baseline achieved macro-averaged F1 score and Cohen's kappa comparable to the best-performing LLM. Agreement was highest for non-motorist relation and crash manner and lowest for light condition, roadway surface condition, and work-zone relation. Differences across crash attributes exceeded differences across models. These results provide a benchmark for evaluating LLM-based crash coding and show that deployment should be evaluated on an attribute-specific basis using transparent baselines and human review.

Sudhir Bharati, Rajendra K C Khatri, Sudip Bharati

Planning and Research Division, Arkansas Department of Transportation, Little Rock, Arkansas, USA · Department of Mathematics, Philander Smith University, Little Rock, Arkansas, USA · Itahari International College, Itahari, Nepal

LLM Evaluation

Ask PDF

Sep 10, 2026stat.ML

CHOIR: heterogeneity-aware conformal prediction for crash injury severity across driver safety strata

Transportation agencies increasingly predict crash-injury severity with statistical and machine-learning models, but these models do not state how often their output contains the recorded injury level or for which groups of drivers it fails, a gap that matters most for motorcyclists and unrestrained drivers. This study develops and evaluates a certification layer that gives any fitted severity model a finite-sample, distribution-free coverage guarantee within prespecified safety strata. The layer, CHOIR (Conformal Heterogeneity-aware Ordinal Inference with Risk control), combines groupwise and weighted conformal prediction with conformal risk control to return contiguous KABCO intervals, and adds a declared sensitivity analysis for medically assessed injury and bounds on fatal omission. It is evaluated on 4.04 million Texas crashes from 2017-2023, one sampled driver per crash, with seven base models from the ordered logit to a tabular foundation model, and on held-out counties and later years. Under one pooled threshold every model reaches 0.90 coverage overall but covers motorcyclists or unrestrained drivers at 0.868 or lower, and class-balanced gradient boosting covers unrestrained drivers at only 0.374. Calibration within four safety strata places all 28 model-by-stratum estimates between 0.898 and 0.907, at the cost of sets spanning 3.6 to 4.6 of five categories for these groups, and the certified ordered logit is within 0.03 categories of the narrowest model. Injury-model coverage should therefore be certified within safety groups rather than on average, calibration rather than model complexity determines validity, and a statewide threshold should not be applied to small rural counties without local calibration data.

Amir Rafe, Subasish Das

Civil Engineering, Texas State University, 601 University Drive, San Marcos, 78666 TX, USA

Distribution Shift Robustness Group-Conditional Conformal Prediction Conformal Prediction

Ask PDF