cs.LGAug 20, 2026

Auditing Recorded Predictive Lead Service-Line Classifications Against Physical Verification: A Statewide Study of New York

Authors: Muhammad Sarmad Sohail

Organizations: Independent researcher

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Under the US Lead and Copper Rule Revisions, a utility may determine a service line's material with a predictive model instead of inspecting it. New York State publishes, per address, which method was used. Almost no address carries both a model classification and a physical verification, so the check is between populations within a utility rather than paired addresses. We screen all 153 New York localities that classified at least 100 addresses this way. Seventy-five (49%), covering 125,990 addresses or 57% of those screened, record one value. Zero variance alone is not misconduct: 68 of the 75 match their own verification or have too little to test. Seven are contradicted by their own crews, six beyond any sampling explanation. Five are boroughs of New York City, which file as one system; one is East Rochester, 550 km away. New York City is the largest case: a predictive model is the recorded basis for 43,215 addresses, and on all of them the recorded material is "Known Other". The city records "Unknown" on 121,779 addresses, 1,880 already excavated, and lead on 120,692. In the model bucket both counts are zero, and the 95% upper bound on the rate is 0.0085%. Across the rest of New York the same method records lead or the hedge "Unknown but could be lead" on 12.21% of 176,888 addresses, a comparison whose weaknesses we report. The model-cleared population is newer, median year built 1984 against 1930, and construction era accounts for about a third of the gap and not the rest: holding era fixed, records-based classification finds lead at 4.3-31.9%, physical verification at 1.5-14.5%, the model in no era. Six era-aware estimators place the expected lead lines among them at 1,150-1,450. Two findings need no comparison: 7,782 of these addresses are in pre-1940 buildings, and the archived 2025 snapshot shows the public-side determination was copied from a customer-side model output.

Figures & tables

Figure 1: Every New York water system with at least 100 model-classified addresses (153 systems, 221,090 addresses). Vertical axis is the share of those classifications recorded as lead or as Unknown but could be lead . Hollow markers on the floor are systems whose model output takes a single value; red diamonds are the subset contradicted by the same system’s own physical verification. The field’s full range is in live use: Poughkeepsie records lead on 81.9% of 7,189 addresses, DeWitt hedges on 97.0% of 2,033.
LocalityModel-classifiedOnly recorded valueVerifiedLead foundExpectedP(0)P(0)
Staten Island16,434Known Other20,5002.31%38010−16710^{-167}
Brooklyn10,308Known Other52,0247.54%77710−35110^{-351}
Queens8,513Known Other66,46014.19%1,20810−56610^{-566}
Bronx4,867Known Other17,7695.46%26610−11910^{-119}
Manhattan3,315Known Other11,7513.40%11310−5010^{-50}
East Rochester472Known Other5579.69%4610−2110^{-21}
Table 1: Localities whose model output takes one value and whose own crews contradict it. The last two columns ask how surprising the zero is: how many lead findings the model bucket would contain if it shared the locality’s own physically verified lead rate, and the probability of observing none. Troy is the one case a referee should discount.
Basis of classificationAddressesShareLeadUnknownMedian built
Records532,90665.5%19.80%9.26%1931
Field inspection95,08711.7%8.45%0.01%1931
Excavation72,1748.9%9.86%2.60%1930
Not verified70,5298.7%0.00%100.00%1930
Predictive model43,2155.3%0.00%0.00%1984
Table 2: Public-side classifications in New York City by recorded basis, 813,911 addresses. “Unknown” counts materials recorded as some form of unknown; median year built from the MapPLUTO join.
Figure 2: Lead rate by construction era and basis of classification, New York City. Records-based classification is a desk method and finds lead in every era; physical verification finds lead in every era; the predictive model finds it in none, including 4,681 addresses in buildings built 1920–1940 where every other method in the same city finds lead at 14–32%.
Figure 3: Eleven estimators in three families: direct stratification, non-parametric spatial nearest-neighbour matching at two calipers, and gradient-boosted trees. Estimators controlling for construction era are red; the shaded band spans them. Where a stratum has fewer than 20 verified neighbours its addresses are counted unscored and the total extrapolated over the covered share rather than dropped; coverage is annotated below 100%.
Cross-validation groupingGroupsAUC, no ZIP featureAUC, with ZIP feature
ZIP code2000.7200.638
1 km block8220.7360.730
2 km block2490.7270.705
5 km block610.6390.660
10 km block210.6450.637
Table 3: Cross-validated discrimination as a function of how far apart training and test data are held. A whole group falls in one fold.
More in this topic

Classification

Classification maps observations to discrete labels, including familiar, unknown, hierarchical, or multiple categories.

See all 410 papers

Explore similar work

CardsList
Oct 7, 2026

Conformal Prediction under Partial Verification

Conformal prediction provides prediction sets with finite-sample guarantees, but the label verification required for calibration can be expensive. We develop a partial verification method that returns exactly the same prediction sets as complete verification. We characterize calibration certificates, the verified information sufficient to determine the conformal threshold, and design a procedure that coordinates verification across calibration examples. For finite thresholds at high coverage, its verification cost is less than twice the minimum certificate cost when candidates are checked in order. Across retrieval, mathematical solutions, and configuration evaluation, it reduces verification cost by 15-82% compared with verifying calibration examples one at a time, while producing identical prediction sets.

Zijun Yu, Yu Gu, Vahid Partovi Nia +1

McGill University · Polytechnique de Montréal

Conformal Prediction
AskPDF
Jul 13, 2026

From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth

Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (Delta AUC = -0.144, 95% CI [-0.471,0.176]) and from a deterministic heuristic (Delta AUC = -0.089, 95% CI [-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity.

Aleh Manchuliantsau

Independent Researcher

LLM EvaluationProbabilistic ForecastingAutomated Evaluation
AskPDF
Sep 2, 2026

Seeing the City or Recognizing the Place? What Street-View Imagery Adds Beyond Existing Urban Data in VLM-Based Urban Sensing

Street-view imagery is increasingly analysed with vision-language models (VLMs) to infer urban attributes, but predictive accuracy alone does not show how much a photograph contributes beyond data already available for the same place. Using three VLMs, we compare image-based predictions with existing urban data for seven attributes drawn from five public resources. Each urban unit is evaluated with imagery, with location or text context, and against non-image predictions from nearby observations or public records. We also replace images and add conflicting records to test which source the predictions follow. Nearby observations or public records matched or exceeded image-only predictions for road damage, curb ramps, population, and house price. Images were more informative for building type, building function, and the floor count of low-rise buildings. For floor count, the advantage of images over nearby OpenStreetMap labels increased by 5.7 percentage points per doubling of distance to the nearest labelled building and declined for tall buildings whose rooflines often fell outside the frame. When images and records disagreed, predictions usually moved towards the supplied record. Released OpenFACADES floor annotations, generated with OpenStreetMap floor values as input, showed the same dependence: their agreement with the reference increased with building height, whereas that of image-only reruns decreased. The value of street-view imagery therefore depends on whether an attribute is visible and how well the place is already covered by existing data. Because machine-derived labels are often reused as references, these comparisons also bear on how urban datasets are documented and evaluated.

Kaizhen Tan

New York University

VLM Evaluation
AskPDF