cs.LGMar 3, 2026
SaveCausal Discovery Should Embrace the Wisdom of the Crowd
Organizations: Department of Industrial and Systems Engineering, University of Washington, Seattle, WA 98195, USA · Department of Applied Mathematics, Illinois Institute of Technology · Department of Electrical & Computer Engineering, Texas A&M University
Abstract
This paper argues for recognizing an emerging paradigm of causal learning by wisdom of the crowd. Recent developments in government, industry, and research point to the rise of decentralized and crowd-based approaches within causal modeling, where causal knowledge distributed across many contributors can be systematically elicited and integrated with causal learning workflows. In this paradigm, causal learning becomes a distributed decision-making problem: each participant contributes partial and potentially noisy knowledge, while collective contributions help construct a global causal structure. This direction is enabled by advances in crowdsourcing platforms, expert knowledge elicitation, aggregation techniques, and large language model (LLM)-augmented information acquisition. Its promise is increasingly visible in early research and emerging real-world practices. Building on this momentum, we outline a framework for crowd-based causal learning spanning elicitation, modeling, aggregation, and optimization. We further discuss the opportunities and challenges introduced by this paradigm and call for interdisciplinary collaboration across causal learning, collective intelligence, human-AI interaction, and decision science.
Figures & tables
Figure 1: An overview of the crowd-based causal learning.
| Phase | Methodology | Role of Expert |
|---|---|---|
| Early Manual Construction | Direct Elicitation | Sole source of structure |
| Pure Data-Driven Learning | Independence Tests / Scoring | Validation only |
| Hybrid Learning | Priors / Hard Constraints | Guiding the search space |
| Distributed Wisdom of the Crowd | Decentralized Solicitation | Distributed “Puzzle Solving” |
Table 1: Evolution of expert involvement in causal discovery.
Figure 2: Bayesian network Asia with eight nodes and eight causal links representing a simplified respiratory diagnosis scenario.
Figure 3: Expert- and query-level knowledge complexity sorted by median absolute score.
Figure 4: Illustration of expert types in four dimensions.
Figure 5: A toy example illustrating the different expert types. Solid edges represent the causal links asserted by the expert, including correct (black), reverse (yellow) and spurious (red) edges, while dashed edges indicate low-confidence causal links regardless of validity.
Appendix figures & tables8 assets
Supplementary material from the paper’s appendix.
Appendix
| Aspect | Edge-based elicitation | Ordering-based elicitation |
|---|---|---|
| Question format | When the question is presented as – , please use to denote a direct causal influence from , use to denote , and use to denote no direct influence between and . | How strongly do you believe that is an upstream causal variable of ? |
| Response space | Integer score in | Integer score in |
| Direction encoding | Encoded by the sign of the score | Encoded by the sign of the score |
| Causal interpretation | Direct causal influence only | Upstream causal influence, including both direct and indirect effects |
| Example in the Asia network | For , participants indicate whether they believe there is a direct causal edge. | For , participants may give a positive score if they believe Smoking is upstream of Dyspnea through direct or indirect causal paths. |
Table 2: Comparison between edge-based and ordering-based elicitation.
Figure 6: Uncertainty of the causal ordering under average aggregation of expert comparisons from five experts (ordered by accuracy from high to low: one good expert, three moderate-quality experts, and one poor expert) across two networks of different sizes. The top row shows results for the Alarm network ( ), and the bottom row shows results for the Random-100 network ( ). From left to right, the panels correspond to 20, 50, and 100 expert comparisons, respectively. Darker shades indicate higher posterior rank probability. The rows correspond to variables, and the x-axis indicates the ordering positions of the variables.
Figure 7: Uncertainty of the causal ordering under two aggregation strategies for expert comparisons from five experts (ordered by accuracy from high to low: one good expert, three moderate-quality experts, and one poor expert) in the Alarm network ( ). The top row shows results under simple average aggregation, whereas the bottom row shows results under inverse-variance-weighted aggregation, which assigns larger weight to higher-accuracy experts and smaller weight to lower-accuracy experts. From left to right, the panels correspond to 20, 50, and 100 expert comparisons, respectively. Darker shades indicate higher posterior rank probability. The rows correspond to variables, and the x-axis indicates the ordering positions of the variables.
| Characteristic | Edge-wise | Ordering-wise | Useful Metrics |
|---|---|---|---|
| Completeness | Incompleteness leads to missing | Allow sparse local coverage | Edge coverage, recall |
| (More edge information per query ) | (Higher pruning per query ) | Path coverage, path recall | |
| Belief validity | Local error, harder to detect | Error propagate structurally, easier to detect | Edge precision, FDR |
| (Cumulate more errors v.s more edge info, ) | (More info for pattern detection ) | Pairwise order accuracy, rank-based correlation | |
| Confidence level | Not enough info, need repeated queries | Soft constraints, reveal belief geometry | Abstention rate, re-query variance for one edge |
| (- if no repeated, query familiar , o.w., ) | (Query familiar , o.w., ) | Score dispersion, posterior over orders |
Table 3: Comparison of edge-wise and ordering-wise causal knowledge. ( ) / ( ) denote performance improvement / degradation.
Table 4: Prompt template for eliciting causal judgments from LLM experts.
Table 5: Prompt template for self-verification of the elicited judgments.
| Category | Research Challenge | Key Questions |
|---|---|---|
| Characterizing the knowledge profile of an expert | • Can we build a unified statistical model, i.e., by probabilistic graphical models, that can universally characterize human causal knowledge adequately? • Can topic or embedding methods capture an expert’s domain familiarity? • How can we detect and represent structured expertise, such as an expert’s strength in local subnetworks but not globally? • Can we build interpretable reliability maps across the graph for each expert? | |
| Expert Knowledge Modeling | Modeling structured uncertainty or abstention | • How can we model expert uncertainty in a structured way (e.g., confidence scores, conditional statements)? • Can abstentions be modeled as informative signals rather than nulls? • What are formal representations of “partial knowledge” beyond edge judgments (e.g., subgraph templates, logical rules)? |
| Fusing heterogeneous experts | • How to integrate mixed forms of input: edges, orderings, counterfactuals? • Can we define a unifying representation that respects all input formats? • How to resolve conflicting inputs across representation types and reliability levels? • Can we detect and screen out bad actors to avoid malicious behaviors? | |
| Adapting expert modeling over time | • Can we update beliefs about expert accuracy online as data come in? • What signals should we track to detect changes in knowledge quality? • Can interactive systems personalize which graph regions to route to each expert? | |
| Optimal elicitation strategies under a fixed budget | • What strategies maximize expected information gain per question? • Can we prioritize resolving parts of the graph with the highest causal ambiguity? • Can budget-aware strategies be adapted dynamically as expert responses arrive? | |
| Selecting between query types | • When should we ask about edges vs. causal orderings vs. counterfactuals? • Can we learn which query types are more accurate for different experts? • How to model and exploit the costs and uncertainties of different query types? |
Table 6: Future research directions for crowd-powered causal learning, categorized by modeling, elicitation, human–AI collaboration, and broader systems challenges. Each challenge includes specific questions to advance this emerging field.
| Category | Research Challenge | Key Questions |
|---|---|---|
| Optimizing division of labor | • When should a query be routed to an LLM vs. a human? • How to dynamically reassign tasks based on model confidence or experts’ quality? • How to combine LLM reasoning and crowd input into unified learning pipelines? | |
| Human–AI Collaboration | LLMs as elicitation intermediaries | • Can LLMs help translate between formal causal concepts and human-friendly prompts, or refine user input into structured formats? • If so, how to model this process, and do quality control and optimization? |
| Fundamental limits of crowd-based causal learning | • Identifiability limits when human knowledge is partial, inconsistent, or noisy? • Can we derive information-theoretic lower bounds on the number or type of queries required to recover causal structure? • Do graph properties (e.g., sparsity, treewidth) affect learnability from crowd input? • What theoretical guarantees can we provide when combining observational data with distributed human knowledge? | |
| Broader Theoretical and Systems Questions | Benchmark development | • What benchmark tasks and datasets are needed to evaluate systems that use human input for causal discovery? • How can we simulate crowd-like behaviors (e.g., imperfect expertise, varying reliability) to support controlled comparisons? • What metrics capture the quality of learned causal structures when ground truth is incomplete or unavailable? • How should we evaluate human effort efficiency and knowledge acquisition cost in interactive systems? |
| Ethical and equitable participation | • How to ensure inclusivity and reduce bias in knowledge collection? • What incentives encourage high-quality participation from diverse contributors? • How to ensure transparency, attribution, and accountability in aggregating crowd knowledge? • What risks arise when aggregating causal beliefs across cultural, disciplinary, or epistemic divides? |
More in this topic
Causal Discovery
Causal discovery infers cause-and-effect structures from observational and interventional data, distinguishing causal relationships from correlation.
See all 221 papers