Kierston & KierstonBench: Simulation of Allentown, PA and Evaluation of Language Models in Crime Investigation and Surveillance Tasks
Abstract
Large language models are candidates for collecting and analyzing records in criminal investigations and mass surveillance. Their capability and accuracy in this work cannot currently be measured. Real deployments lack the ground truth needed to score a model's conclusions, and no benchmark evaluates these capabilities in a simulated environment. We present Kierston, a city-scale simulation engine, and KierstonBench, a benchmark that tests these capabilities against known simulated events. Kierston models approximately 125,000 residents of Allentown, Pennsylvania, generating their daily activities and the telecommunications, financial, administrative and license-plate records that surveillance and investigative systems collect about them. These records derive from a common behavioral timeline and are distributed across sources representing separate providers. Models investigate through a restricted records interface, linking observations across sources without access to the underlying simulation. The benchmark comprises 32 task families covering record linkage, routine reconstruction, incident investigation and assessment of deceptive accounts. In an evaluation of sixteen language models, most could not perform records investigation and submitted identifications the records did not support. A small number of frontier models solved most of the tasks and abstained when evidence was insufficient. In some cases these models still fell for common deceptions and assumptions, and none replaced an investigator.
Figures & tables
| System | Documented overlap | Additional focus here |
| SimMobility [ 19 ] | Integrated activity, land use, economy and movement | Provider-specific investigative records |
| Urban Life / Patterns of Life [ 41 , 2 ] | Spatial daily lives, social ties, known simulated truth | Administrative identities joined to indirect sensor and provider evidence |
| AgentSociety [ 25 ] | Mobility, social interaction and economic behavior | Rule-based replay and an investigator’s restricted view |
| SUMO cellular extension [ 17 ] | Road movement and CDR-like mobile-device logs | Cross-provider linkage to population registers and payment records |
| VAST 2014 / 2022 [ 34 , 35 ] | Multiple record types, patterns of life and analytic tasks | Generated agent tasks, evidence cutoffs and signed identity scoring |
| AppWorld [ 30 ] | Synthetic people, app databases, tool access and executable evaluation | Inference about physical events from incomplete observations |
| Source | Used for |
| ACS 5-year PUMS, Public Use Microdata Area (PUMA) 42-02803 (Allentown city) [ 32 ] | Household and person microdata: age, sex, race/ethnicity, relationship, income, education, enrollment, employment status, occupation, industry, class of worker, hours, commute mode and times, marital and immigration history. Every synthetic household is a clone of a surveyed one, jittered. |
| OpenStreetMap, 5 km extract [ 21 ] | Dwelling stock (buildings, levels, footprints, addresses), points of interest by class, workplaces, schools, places of worship, police and fire stations, and the full highway topology the router is built from. |
| FCC Antenna Structure Registration [ 9 ] | Registered masts: coordinates, height, structure type, owner, registration number. These registered structures support the simulated network; rooftops, short monopoles and small cells fall under the registration threshold and are modeled from building and workplace density. |
| OpenCelliD [ 31 ] | Observed cells per operator, used to avoid allocating identifiers present in the local extract. Its positions are never read. |
| NHTS 2022 + EPA vehicle data [ 10 ] | Travel and activity parameters from NHTS, plus the vehicle market: household vehicle distributions conditional on income, by make, model, body, year and price; resampled rather than rule-based. |
| GSMA type-allocation codes (community mirror) | Real TACs for 64 handset models so that every IMEI in the city is Luhn-valid and resolves to the model the record claims. |
| Group | Stages | What is decided |
| Locale | locale | The city profile and the jurisdiction’s plate rules, published as JSON for the runtime, router and importer |
| Intake | dwellings, pois, buildings, mcc | OSM exports to canonical building, POI and merchant-category assets; dwelling-unit counts fitted per building class |
| Population | population, placement | PUMS households cloned to the target household count and placed into the dwelling stock; group-quarters residents placed separately |
| Work | work, workplaces, assign_work | Employment status and hours from the survey; worksites from POIs and footprints with industry and capacity; workers matched to sites within their reported commute reach, primary-class roles (police, EMS, taxi) staffed from the emergency facilities with platoon, duty and crew |
| Network | towers | FCC structures plus modeled rooftops, monopoles and small cells on a demand surface; tenancy, sectors, bands, identifiers, tracking areas and physical cell identities |
| Education | schools, assign_school, school_travel | Schools with grade bands, sector and seats; enrollment; who escorts which child and the bus runs |
| Domain | Attributes |
| Demographics, household | Age, sex, race/ethnicity, relationship to householder; household size and income; dwelling building, unit, area |
| Names, identity | First, middle, surname(s) conditioned on birth cohort, ethnicity and household; birth date; SSN or ITIN with issue date; citizenship, birthplace, entry and naturalization years; legal status; mobile and landline numbers |
| Immigration documents | CRBA, naturalization certificate, green card, visa, I-94, SEVIS, EAD, foreign passport, A-number: number, issuer, form, dates, status |
| Work, education | Employment status, occupation (SOC), industry, class of worker, hours, shift; assigned worksite and travel; attainment; current enrollment, school, grade, escort |
| Devices | One handset per mobile number: Luhn-valid IMEI on a real TAC, platform, make/model, release year, OS version, acquisition date and condition, SIM slot, eSIM EID with IBAN-style check digits |
| Subscriptions | phone number (MSISDN), subscriber identity (IMSI) under the host network’s country and network codes (MCC/MNC), SIM serial (ICCID), carrier and host network (the network a reseller’s subscribers use), reseller (MVNO) flag, prepaid/postpaid, account holder (a parent for most minors), current handset |
| Class | Buildings/km 2 | Target site spacing | Small cells |
| Dense urban | 600 m | yes | |
| Urban | 900 m | yes | |
| Suburban | 1500 m | no | |
| Open | 2600 m | no |
| Group | Fields |
| Identity | CGI ( MCCMNC-TAC-identity , ECGI on LTE or NCGI on NR); station id (eNodeB or gNodeB); local cell id; MCC, MNC, operator; tracking area; physical cell identity |
| Radio | radio (LTE/NR); band; role (anchor, mid, capacity, firstnet); downlink frequency in MHz |
| Geometry | structure id; latitude, longitude; antenna height; sector number; azimuth; horizontal beamwidth; downtilt; EIRP |
| Planning | indicative range in meters; morphology label of the site’s surroundings |
| Structure | id; source ( asr or modelled ); FCC registration number where one exists; coordinates; kind (monopole, lattice, pole, rooftop, small cell, …); height; owner; in-city flag |
| Component | Responsibility |
| Daily scheduling | Fits fixed commitments, flexible activities and travel into each person’s day |
| Work | Shifts, commutes, overnight duties, police patrol routes and crews |
| School | Attendance, adult escorts and school-bus journeys |
| Sleep | Sleep periods, including those crossing midnight |
| Social activity | Relationships, organization meetings and gatherings among friends |
| Errands and leisure | Destination choice, habitual stores and stops during existing journeys |
| Lighting | Captured | Correct | Misread | Unreadable | Spurious |
| Day | 0.990 | 0.970 | 0.020 | 0.010 | 0.0015 |
| Night (infrared) | 0.995 | 0.985 | 0.010 | 0.005 | 0.0005 |
| Twilight | 0.975 | 0.930 | 0.045 | 0.025 | 0.0030 |
| Model can query | Withheld ground truth |
| Logged network events and cell-site information | Complete handset movement and continuous serving-cell history |
| Plate text, timestamps, camera locations and confidence | Every vehicle passage and its true vehicle identity |
| Call and message records | Underlying social relationships and conversation purposes |
| Account, card, merchant and transaction records | The person responsible for each payment |
| Provider-specific identity and administrative records | Simulation-wide resident identifiers and direct links between providers |
| Records available before the task’s cutoff | Later events, complete attendance lists and incident roles |
| Source | Sector | Records available |
| ssa | federal | Social Security Administration (Numident): people and legal identity |
| dhs | federal | USCIS and CBP document files: identity documents and immigration status |
| military | federal | Service departments and the VA: service records |
| dot | state | Vehicle registrations, driver licenses and suspensions |
| psp_firearms | state | Firearm sales, carry licenses and background checks |
| corrections | state | Custody, supervision and release status |
| Family | Grade | Asks | Answer |
| device-to-person | 1 | who carries a given subscription | R |
| counterparty-recon | 1 | who the other party to a call was | R |
| account-resolution | 2 | who owns a messaging handle | R |
| household-recon | 2 | who lives with a given person | S |
| kinship-infer | 2–4 | a named relative of a person | R |
| plate-reconciliation | 2–3 | the true plate behind several OCR reads | L |
| Family | Event | Asks | Answer | Cutoff |
| mass-shooting | shooting in a bar | who fired; the shooter flees, blends into the crowd or dies, sampled by seed | ranked | 10–20 min |
| shots-fired | shots on a corner | who fired; a decoy handgun owner is bound when the day has one nearby | ranked | 8–15 min |
| explosion | device at a workplace | who planted it; two other visitors left beforehand; staff excluded by the prompt | ranked | 12–25 min |
| school-shooting | shots in a school | who fired; a student inside or an outsider who arrives | ranked | 10–20 min |
| abduction | victim taken at a curb | who was in the car; the phone is left behind; any crew member is a right answer | ranked | 8–15 min |
| protest-members | crowd at a park or civic site | who is taking part right now; bystanders are the decoy group | set | 60–95 min in |
| Family | The account | What the records show |
| stabbing | A man called 911 saying he came home and found his girlfriend stabbed. | The couple traveled out and came back together: two handsets side by side, the same car on the same cameras, no text or call between them. Truth: the boyfriend. |
| swatting | A caller reported an armed man forcing his way into a house and threatening the people inside. | The caller’s handset was nowhere near the house and nobody’s handset came to it, and the caller had called the occupant twice and been rejected. Truth: the caller. |
| disappearance | A caller reports that his partner left home but did not reach her destination. | The caller’s handset remains at home while his registered vehicle moves through the city. The scenario separates handset possession from the caller’s movements. |
| Claim | Correct | Wrong, unresolved or contradictory |
| Full name | ||
| Name and date of birth | ||
| Name and date of birth, person without an SSN | ||
| SSN, with or without name and date of birth |
| Tasks | Claude Opus 5 | Claude Fable 5.1 | Claude Opus 5.5 | Claude Sonnet 5.5 | GPT-6 Astra | GPT-5.6 Sol | GPT-6 Sol | GPT-6.1 Sol | GPT-6 Luna | MiniMax M3 | DeepSeek V4.1 Flash | Gemini 3.8 Flash | GLM 5.3 | Kimi K3 | Qwen 3.8 Max | Grok 4.7 | |
| Technical Tasks | 22 | 0.61 | 0.75 | 0.63 | 0.41 | 0.53 | 0.62 | 0.67 | 0.58 | 0.21 | 0.46 | 0.55 | 0.46 | 0.55 | 0.52 | 0.61 | 0.33 |
| Real-Time Investigations | 7 | 0.55 | 0.38 | 0.56 | 0.58 | 0.14 | 0.57 | 0.00 | 0.00 | 0.00 | 0.71 | 0.57 | 0.41 | 0.57 | 0.71 | 0.57 | 0.43 |
| Deception Tasks | 3 | 0.67 | 1.00 | 0.83 | 1.00 | 0.33 | 0.67 | 1.00 | 0.67 | 0.00 | 0.67 | 1.00 | 0.67 | 0.67 | 1.00 | 0.50 | 0.67 |
| All | 32 | 0.36 | 0.52 | 0.39 | 0.25 | 0.43 | 0.36 | 0.55 | 0.46 | 0.15 | 0.22 | 0.35 | 0.28 | 0.31 | 0.29 | 0.34 | 0.20 |
| Tasks scoring 1 | 16 | 19 | 15 | 14 | 13 | 18 | 17 | 14 | 5 | 12 | 16 | 15 | 14 | 16 | 16 | 12 | |
| Tasks scoring between 0 and 1 | 6 | 4 | 8 | 4 | 2 | 1 | 1 | 2 | 1 | 4 | 3 | 4 | 3 | 3 | 2 | 1 |
| Technical Tasks | Real-Time Investigations | Deception Tasks | All | |||||||||
| Abst. | Ans. | Mean | Abst. | Ans. | Mean | Abst. | Ans. | Mean | Abst. | Ans. | Mean | |
| Claude Opus 5 | 0 / 22 | 22 | 0.61 | 0 / 7 | 7 | 0.55 | 0 / 3 | 3 | 0.67 | 0 / 32 | 32 | 0.36 |
| Claude Fable 5.1 | 2 / 22 | 20 | 0.82 | 0 / 7 | 6 | 0.44 | 0 / 3 | 3 | 1.00 | 2 / 32 | 29 | 0.58 |
| Claude Opus 5.5 | 0 / 22 | 22 | 0.63 | 0 / 7 | 7 | 0.56 | 0 / 3 | 3 | 0.83 | 0 / 32 | 32 | 0.39 |
| Claude Sonnet 5.5 | 3 / 22 | 19 | 0.48 | 0 / 7 | 7 | 0.58 | 0 / 3 | 3 | 1.00 | 3 / 32 | 29 | 0.28 |
| GPT-6 Astra | 7 / 22 | 15 | 0.78 | 6 / 7 | 1 | 1.00 | 2 / 3 | 1 | 1.00 | 15 / 32 | 17 | 0.80 |
| Family | Claude Opus 5 | Claude Fable 5.1 | Claude Opus 5.5 | Claude Sonnet 5.5 | GPT-6 Astra | GPT-5.6 Sol | GPT-6 Sol | GPT-6.1 Sol | GPT-6 Luna | MiniMax M3 | DeepSeek V4.1 Flash | Gemini 3.8 Flash | GLM 5.3 | Kimi K3 | Qwen 3.8 Max | Grok 4.7 |
| Technical Tasks | ||||||||||||||||
| person-reid | 0.60 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.00 a | 1.00 | 1.00 | 1.00 | 1.00 | 0.60 | 0.60 | 1.00 |
| device-to-person | 1.00 | 0.50 | 0.50 | 0.50 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 0.50 | 1.00 | 0.50 | 0.50 | 0.50 | 1.00 | 1.00 |
| account-resolution | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| plate-reconciliation | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
| point-in-time-location | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 a | 1.00 | 1.00 | 1.00 | 0.00 | 0.00 | 0.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 |
Appendix figures & tables1 asset
Supplementary material from the paper’s appendix.
Appendix
| Family | Meaning |
| person , schedule , path and travel records | Per-person offset/count pairs into schedules and journeys; each journey points into a shared routed-coordinate table in signed microdegrees |
| vehicle , vehicleSpan | Registry metadata and an exhaustive midnight-to-midnight VIN timeline; moving spans join to journeys, parked spans retain position |
| tx | Financial transactions and balances with explicit ownership |
| subscriber , cell , span , event | Subscriber and cell dictionaries, continuous serving-cell truth (RSRP, surroundings, connected minutes, timing advance, secondary cell), and the smaller set of operator-visible events with the same radio fields |
| account , conversation , participant , message , call , sms , cdr , handover , attention | Communications truth, carrier-visible records with cells at both ends, and per-person attention runs |
| meeting , party | Gatherings and attendee roles |
Entity Resolution
Entity resolution links records that refer to the same real-world entity across datasets despite differences in names, attributes, formats, and languages.
See all 45 papers