eess.SYMar 26, 2026

On Integrating Resilience and Human Oversight into LLM-Assisted Modeling Workflows for Digital Twins

Authors: Lekshmi P, Neha Karanjkar

Organizations: Indian Institute of Technology Goa Goa, India

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

LLM-assisted modeling holds the potential to rapidly build executable Digital Twins of complex systems from only coarse descriptions and sensor data. However, resilience to LLM hallucination, human oversight, and real-time model adaptability remain challenging and often mutually conflicting requirements. We present three critical design principles for integrating resilience and oversight into such workflows, derived from insights gained through our work on FactoryFlow - an open-source LLM-assisted framework for building simulation-based Digital Twins of manufacturing systems. First, orthogonalize structural modeling and parameter fitting. Structural descriptions (components, interconnections) are LLM-translated from coarse natural language to an intermediate representation (IR) with human visualization and validation, which is algorithmically converted to the final model. Parameter inference, in contrast, operates continuously on sensor data streams with expert-tunable controls. Second, restrict the model IR to interconnections of parameterized, pre-validated library components rather than monolithic simulation code, enabling interpretability and error-resilience. Third, and most important, is to use a density-preserving IR. When IR descriptions expand dramatically from compact inputs hallucination errors accumulate proportionally. We present the case for Python as a density-preserving IR : loops express regularity compactly, classes capture hierarchy and composition, and the result remains highly readable while exploiting LLMs strong code generation capabilities. A key contribution is detailed characterization of LLM-induced errors across model descriptions of varying detail and complexity, revealing how IR choice critically impacts error rates. These insights provide actionable guidance for building resilient and transparent LLM-assisted simulation automation workflows.

Figures & tables

Figure 1: LangGraph-based architecture for LLM-assisted structural model generation in FactoryFlow
IDCategoryDescription & Example
T1Naming errorsInconsistent identifiers, off-by-one indexing, or violated naming conventions. Example: Grid systems use 0-based indexing when description specifies 1-based (S21, S24); buffers renamed to generic identifiers from few-shot examples ( edge[0] , src_edge[0] ); naming errors increase with description detail rather than decrease.
T2Parameter errorsIncorrect parameter values, misapplied defaults, or ambiguous conflict resolution. Example: S13 prioritized stage-specific values over generic statements; routing policies defaulted to ROUND_ROBIN when unspecified (S30, S31-S35).
T3Node hallucinationAddition or omission of nodes (machines, buffers, sources, sinks). Example: S3 hallucinated an extra machine to satisfy perceived edge constraints; S35 hallucinated 10 nodes not in description.
T4Edge hallucinationAddition or omission of edges (connections between components). Example: S18 inferred two feedback connections instead of one based on ambiguous phrasing; S35 hallucinated 7 edges.
T5Parameter hallucinationGeneration of parameter specifications not present in description or incorrect parameter structures. Example: S23, S25 hallucinated routing behavior as explicit lists instead of supported policy types; S35 hallucinated many parameter values not specified.
T6Hierarchy mismatchIncorrect hierarchical structure, flattening of nested subsystems, or misplaced component scope. Example: S13, S15, S18, S24, S31 collapsed explicitly described subsystems into flat topologies; S26 reinterpreted hierarchical system as single serial flow; S24 assumed single global source/sink instead of per-row hierarchy.
Table 1: Error taxonomy with definitions and examples.
Figure 2: Error counts across models ordered by complexity (IR size).
Figure 3: Aggregate error type frequency across all models.
Figure 4: Error composition across individual models for both coarse and detailed descriptions.
Appendix figures & tables6 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Examples of various types of errors observed
Figure 6: GUI of DataFITR ( DataFITR tool ) - Selecting data and fitting distributions.
Figure 7: GUI of DataFITR - Results of input modeling.
Figure 8: Documentation page of FactorySimPy ( documentation page ).
Figure 9: GUI of FactoryFlow ( GitHub repository (PoC) ), illustrating model generation of a simple system.
Figure 10: GUI of FactoryFlow with code generated for the description ”A system with two machines in parallel”
More in this topic

Digital Twins

Digital twins are virtual models synchronized with real systems to support monitoring, prediction, simulation, and decision-making.

See all 253 papers

Explore similar work

CardsList
Sep 7, 2026

When Can LLM Digital Twins Reduce Human Measurement? From Behavioral Fidelity to Statistical Substitutability

LLM-based digital twins promise to reduce repeated human data collection by generating person- specific responses, yet existing evaluations provide little evidence about whether they can reduce human measurement while preserving valid inference. To address this, we introduce statistical substitutability, an inferential criterion that evaluates the extent to which twin predictions can reduce human measurement for a particular estimand while preserving valid inference. We develop a framework, grounded in mixed-subject and prediction-powered inference, that evaluates statistical substitutability along four dimensions: aggregate fidelity, paired respondent-level signal, finite-sample human-label recovery, and stability across populations. Across two empirical evaluations spanning behavioral experiments, multiple models, and alternative respondent representations, we find that digital twins can reproduce average human effects while providing little information about which individuals differ from those averages. Newer models and richer respondent information improve some dimensions of performance but do not reliably translate into human-data savings. Human calibration can reduce aggregate prediction error, yet limited labeled samples often fail to produce stable precision gains. Importantly, these findings demonstrate that behavioral fidelity is neither necessary nor sufficient for statistical substitutability. More broadly, they suggest that AI-generated evidence should be evaluated based on its ability to support valid scientific inference rather than its ability to reproduce human outcomes alone. Digital twins should therefore be judged for confirmatory use by whether they reduce uncertainty about human quantities, not merely by whether they reproduce human means, distributions, or effects.

Steven Wang, Kyle Hunt, Shaojie Tang +1

University at Buffalo

Human Behavior SimulationDigital Twins
AskPDF
Oct 13, 2026

DTScribe: Generating Models for Digital Twin Reporting using Local Large Language Models

The development of a digital twin brings together many stakeholders and technical components. We have previously investigated the reporting of the conceptual architecture for these digital twins as an artifact for discussion. However, we find that the activity of modeling the digital twin for visualization and consensus-building can be quite tedious and time-consuming. In this article, we explore the automation of model generation using large language models, targeting the model required for digital twin reporting. Our approach is designed to ingest digital twin documents and output both the conceptual architecture visualization and a reporting page for that digital twin. We first extract from the input the key details aligning with our twenty-one-characteristic Digital Twin Description Framework described in previous work. Then, we generate a description model in the Ontological Modeling Language, representing the digital twin as a constellation of models, data, enablers, and services. We utilize refinement loops with large language models acting as judges and ontological reasoning to increase artifact quality as measured by our metrics. We investigate how architectural and parameter choices affect artifact quality, such as how refinement loops enable local large language models to produce sufficiently high-quality results comparable to larger models. We also detail insights and lessons learned useful for practitioners, such that they can integrate our techniques to accelerate their digital twin reporting or for other model generation tasks. These insights include the use of medium-sized local models (around 20 billion parameters), the avoidance of large language models as judges and clustering for characteristic extraction, and the recommendation of at least two reasoner loops for Ontological Modeling Language refinement.

Kérian Fiter, Angelica Portocarrero, Bentley Oakes

Département de Génie Informatique et Génie Logiciel, Polytechnique Montréal, 2500, chemin de Polytechnique, Montréal, H3T 1J4, Québec, Canada · Facultad de Ingeniería y Ciencias Básicas, Universidad Autónoma de Occidente, Carrera 25 #115-85, Santiago de Cali, 760030, Colombia

Digital Twins
AskPDF
Apr 8, 2026

SYN-DIGITS: A Synthetic Control Framework for Calibrated Digital Twin Simulation

AI-based persona simulation -- often referred to as digital twin simulation -- is increasingly used for market research, recommender systems, and social sciences. Despite their flexibility, large language models (LLMs) often exhibit systematic bias and miscalibration relative to real human behavior, limiting their reliability. Inspired by synthetic control methods from causal inference, we propose SYN-DIGITS (SYNthetic Control Framework for Calibrated DIGItal Twin Simulation), a principled and lightweight calibration framework that learns latent structure from digital-twin responses and transfers it to align predictions with human ground truth. SYN-DIGITS operates as a post-processing layer on top of any LLM-based simulator and thus is model-agnostic. We develop a latent factor model that formalizes when and why calibration succeeds through latent space alignment conditions, and we systematically evaluate ten calibration methods across thirteen persona constructions, three LLMs, and two datasets. SYN-DIGITS supports both individual-level and distributional simulation for previously unseen questions and unobserved populations, with provable error guarantees. Experiments show that SYN-DIGITS achieves up to 50% relative improvements in individual-level correlation and 50--90% relative reductions in distributional discrepancy compared to uncalibrated baselines.

Grace Jiarui Fan, Chengpiao Huang, Tianyi Peng +2

Finance Division, Columbia Business School. · Department of IEOR, Columbia University · Decision, Risk, and Operations Division, Columbia Business School. +1

Synthetic ControlLanguage Model CalibrationLarge Language Model-Based Social SimulationHuman Behavior SimulationDigital Twins
AskPDF