Counterfactual Evaluation

Follow this topic Ask about this topic

Counterfactual evaluation asks whether a system would change its prediction, explanation, or action appropriately when relevant evidence or conditions change, rather than merely measuring performance on observed cases. Common approaches use paired interventions and benchmarks, simulator or world-model rollouts, controlled replay, and causal or off-policy estimators to compare factual and alternative outcomes. These methods are applied to language and vision models, clinical decisions, recommender systems, and interactive agents, where ordinary accuracy may hide failures in evidence use, safety, or responsiveness. A central challenge is that alternative outcomes are usually unobserved, so conclusions depend on assumptions about causal structure, simulator fidelity, intervention realism, and which metrics reflect useful behavior.

Latest papers

97 papers13 papers in the last four weeks, down 41% on the four weeks before. 0.1% of all new papers.

The Digital Twin Counterfactual Framework: A Validation Architecture for Simulated Potential Outcomes
Apr 1, 2026Olav LaudyCounterfactual Evaluation Causal Effect Estimation
What-If Explanations Over Time: Counterfactuals for Time Series Classification
Mar 29, 2026Udo Schlegel, Thomas SeidlGraph Counterfactual Explanations Counterfactual Explanations
LMU Munich, Germany · Munich Center for Machine Learning (MCML), Germany
Evaluating Counterfactual Strategic Reasoning in Large Language Models
Mar 19, 2026Dimitrios Georgousis, Maria Lymperaiou, Angeliki Dimitriou +2Strategic Reasoning Risks in Language Models LLM Evaluation
National Technical University of Athens
Do Metrics for Counterfactual Explanations Align with User Perception?
Mar 16, 2026Felix Liedeker, Basil Ell, Philipp Cimiano +1Counterfactual Evaluation Explainability Evaluation
Bielefeld University, Bielefeld, Germany
ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders
Feb 18, 2026Ofer Meshi, Krisztian Balog, Sally Goldman +5User Simulation Recommender Systems
Google
Winner's Curse Drives False Promises in Data-Driven Decisions
Feb 9, 2026Hamsa Bastani, Osbert Bastani, Bryce McLaughlinPolicy Evaluation Counterfactual Evaluation
Wharton School · University of Pennsylvania
Counterfactual Operator Relevance for PDE Discovery: Screening, Pruning, and Identifiability
Jun 25, 2025Ronald KatendeCounterfactual Evaluation PDE Discovery
Department of Mathematics Kabale University Kikungiri Hill, Katuna Road, 317, Kabale, Uganda

Apr 1, 2026J/K move · Enter open · S save

The Digital Twin Counterfactual Framework: A Validation Architecture for Simulated Potential Outcomes

Olav Laudy

Counterfactual Evaluation Causal Effect Estimation Causal Reasoning Causal Identifiability Digital Twins

Open paper AskPDF arXiv

The fundamental problem of causal inference - that the counterfactual outcome for any individual is never observed - has shaped the entire methodology of the field. Every existing approach substitutes assumptions for missing data: ignorability, parallel trends, exclusion restrictions. None produces the counterfactual itself. This paper proposes the Digital Twin Counterfactual Framework (DTCF): rather than estimating the counterfactual statistically, we simulate it using a digital twin and subject the simulation to a hierarchical validation regime. We formalize the digital twin simulator as a stochastic mapping within the potential outcomes framework and introduce a hierarchy of twin fidelity assumptions - from marginal fidelity through joint fidelity to structural fidelity - each unlocking a progressively richer class of estimands. The central contribution is threefold. First, a five-level validation architecture converts the unfalsifiable claim that the simulator produces correct counterfactuals into falsifiable tests against observable data. Second, a formal decomposition separates causal quantities into those that are marginally validated (ATE, CATE, QTE - testable through observable-arm comparison) and those that are copula-dependent (the ITE distribution, probability of benefit/harm, variance of treatment effects - permanently reliant on the unobservable within-individual dependence structure). Third, bounding, sensitivity, and uncertainty quantification tools make the copula dependence explicit. The DTCF does not resolve the fundamental problem of causal inference. What it provides is a framework in which marginal causal claims become increasingly testable, joint causal claims become explicitly assumption-indexed, and the gap between the two is formally characterized.