cs.AIMar 8, 2026

HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery

Authors: Chen Zhu, Xiaolu Wang

Organizations: China Agricultural University

Read PDF
Ask AI
Cite

Copy citation

DOINot available for this paper

Abstract

Large language models (LLMs) have enabled agent-based systems that aim to automate scientific research workflows. Most existing approaches focus on fully autonomous discovery, where AI systems generate research ideas, conduct analyses, and produce manuscripts with minimal human involvement. However, empirical research in economics and the social sciences poses additional constraints: research questions must be grounded in available datasets, identification strategies require careful design, and human judgment remains essential for evaluating economic significance. We introduce HLER (Human-in-the-Loop Economic Research), a multi-agent architecture that supports empirical research automation while preserving critical human oversight. The system orchestrates specialized agents for data auditing, data profiling, hypothesis generation, econometric analysis, manuscript drafting, and automated review. A key design principle is dataset-aware hypothesis generation, where candidate research questions are constrained by dataset structure, variable availability, and distributional diagnostics, reducing infeasible or hallucinated hypotheses. HLER further implements a two-loop architecture: a question quality loop that screens and selects feasible hypotheses, and a research revision loop where automated review triggers re-analysis and manuscript revision. Human decision gates are embedded at key stages, allowing researchers to guide the automated pipeline. Experiments on three empirical datasets show that dataset-aware hypothesis generation produces feasible research questions in 87% of cases (versus 41% under unconstrained generation), while complete empirical manuscripts can be produced at an average API cost of 0.8−0.8-1.5 per run. These results suggest that Human-AI collaborative pipelines may provide a practical path toward scalable empirical research.

Figures & tables

FeatureHLERAI ScientistAPELLM AgentsWriting Tools
DomainEmpirical EconMLPolicyGeneralWriting
Hypothesis generation✓\checkmark✓\checkmark✓\checkmark∘\circ-
Empirical / experimental analysis✓\checkmark✓\checkmark✓\checkmark--
Automated paper drafting✓\checkmark✓\checkmark✓\checkmark-✓\checkmark
Automated review / critique✓\checkmark✓\checkmark∘\circ--
Human-in-the-loop✓\checkmark-∘\circ∘\circ✓\checkmark
Table 1: Comparison of HLER with existing AI research automation systems. ✓\checkmark = supported, ∘\circ = partially supported, −- = not supported. “LLM Agents” refers to general-purpose multi-agent frameworks such as AutoGen [ 9 ] and similar systems [ 11 ] . “Writing Tools” refers to LLM-based writing assistants without empirical analysis capabilities.
Figure 1: Architecture of the HLER system. The orchestrator coordinates a multi-agent empirical research pipeline. The core workflow includes data auditing, data profiling, question generation and screening, econometric analysis, manuscript drafting, and automated critique. Two feedback loops are shown: the question quality loop (left), in which candidates are generated and screened before human selection; and the research revision loop (right), in which the reviewer agent triggers re-analysis and manuscript revision through the Econometrics and Paper agents. Human decision gates allow researchers to select research questions and approve final outputs.
ConditionQuestionsFeasible (%)Var. missingDesign incompat.
Dataset-aware (HLER)7969 (87%)46
Unconstrained (ablation)8234 (41%)2017
Table 2: Feasibility of generated research questions under dataset-aware versus unconstrained generation across 14 pipeline runs on three datasets.
DimensionDraft v1Draft v2Final draft
Novelty5.2 (1.1)5.3 (1.0)5.5 (0.9)
Identification4.1 (1.2)5.2 (0.9)5.5 (0.8)
Data quality5.4 (0.8)5.8 (0.7)6.1 (0.6)
Clarity4.3 (1.0)5.8 (0.8)6.4 (0.7)
Policy relevance5.1 (0.9)5.5 (0.8)5.8 (0.7)
Overall4.8 (0.9)5.9 (0.7)6.3 (0.6)
Table 3: Mean reviewer scores (1-10 scale) across revision iterations for 12 completed pipeline runs. SD in parentheses.
More in this topic

Multi-Agent LLM Systems

Multi-agent large language model systems coordinate multiple AI agents to solve tasks through communication, specialization, shared tools, and memory.

See all 1,039 papers

Explore similar work

CardsList
Apr 30, 2026

AgentEconomist: An End-to-end Agentic System Translating Economic Intuitions into Executable Computational Experiments

A long-standing challenge in economics lies not in the lack of intuition, but in the difficulty of translating intuitive insights into verifiable research. To address this challenge, we introduce AgentEconomist, an end-to-end interactive system designed to translate abstract intuitions into executable computational experiments. Grounded in a domain-specific knowledge base covering over 13,000 high-quality academic papers, the system employs a modular multi-stage architecture. Specifically, the Idea Development Stage generates literature-grounded hypotheses, the Experimental Design Stage configures simulator-aligned experimental parameters and protocols, and the Experimental Execution Stage runs experiments and returns structured analyses. Together, these stages form a human-in-the-loop, iterative workflow that translates economic intuitions into executable computational experiments. Through extensive experiments involving human expert evaluation and large language models (LLMs) as judges, we show that the system generates research ideas with stronger literature grounding and higher novelty and insight than state-of-the-art generic LLMs. Overall, AgentEconomist adopts a human-AI collaboration paradigm that enables researchers to focus on high-level intuitions, while delegating the labor-intensive processes of translation and computational execution to agents.

Jiaju Chen, Jinghua Piao, Xia Xu +4

Zhongguancun Academy Beijing, China · University of Science and Technology of China Hefei, China · Department of Electronic Engineering, Tsinghua University Beijing, China +1

AI-Assisted Scientific ResearchLarge Language Model-Guided Scientific DiscoveryAutonomous Experimentation
AskPDF
Jun 11, 2026

(Human) Attention Is (Still) All You Need: Human oversight makes AI-assisted social science reliable

Large language models (LLMs) are increasingly used for tasks once reserved for trained researchers, including hypothesis generation, specification choice, and drafting conclusions. We argue that the reliability of AI-assisted research depends not only on model capability, but also on how cognitive labour is structured between humans and machines. We study this problem through Human-in-the-Loop Economic Research (HLER), a decision architecture based on pre-commitment, decision sequencing, accountability, and attention allocation. In a pre-specified 2*4 factorial experiment with 280 complete research runs across four datasets, an unconstrained multi-agent baseline produced critical failures in 72% of runs. Using the same underlying model, the same agent decomposition, and identical prompts for the shared reasoning agents, HLER reduced the failure rate to 16% by imposing three architectural commitments: LLMs reason but do not execute data work, data and estimation are handled deterministically, and three human decision gates bind the workflow. Fisher's exact test rejects equality of failure rates at p<0.001. Reliability gains were largest on the least publicly represented dataset, a Qing-dynasty population register, consistent with a task-based production model with Frechet-distributed output quality. An 80-run ablation suggests that deterministic computation and human gates contribute independently, with exploratory evidence of complementarity. We interpret HLER as a research harness rather than an autonomous AI scientist: it sharply reduces failures, makes residual weaknesses more visible, and prevents unreliable claims from being advanced as publication-ready outputs.

Chen Zhu, Xiaolu Wang, Weilong Zhang

China Agricultural University · University of Cambridge

AI Agent ReliabilityAI-Assisted Scientific ResearchHuman-in-the-Loop AILLM ReliabilityHuman Oversight of AIMulti-Agent LLMs+1
AskPDF
Jul 23, 2026

pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development

In many social-science research tasks, such as economics, LLM-based agents must produce outputs for which no cheap, task-complete, machine-readable correctness signal exists. This creates a distinctive reliability problem for multi-agent systems: how should generation, critique, coordination, and human judgment be organized when no component can certify the final result? We address this problem through pAI-Econ-claude, a gated, human-in-the-loop multi-agent architecture for AI-assisted economic theory development. Agents coordinate through a shared workspace of inspectable intermediate records; specialized gates diagnose targeted failure modes and recommend loopbacks without certifying correctness; and human checkpoints retain authority over decisions that are costly to reverse. We evaluate the architecture on five matched economic-theory tasks against an ungated baseline. Two evaluators blinded to configuration agreed on all five pairwise rankings, preferring the gated architecture in four tasks and the baseline in one. Mean failure severity fell from 1.58 to 1.16, while overall usefulness rose from 2.60 to 3.10. The largest observed gain occurred when a reality check rejected a false market-structure premise and a proof review prompted revision of a false welfare claim. The negative case shows that scaffolding can also compress an economically important mechanism too aggressively. The results support a bounded claim: gated oversight improves the auditability of AI-assisted economic theory without substituting for formal verification, and the allocation of irreversible human judgment is a more informative design variable than pure agent autonomy. The workflow is publicly available at https://github.com/maxwell2732/pAI-Econ-claude.

Chen Zhu, Xiaolu Wang, Weilong Zhang

China Agricultural University · University of Cambridge

AI-Assisted Scientific ResearchAI Agent EvaluationMulti-Agent CollaborationHuman-in-the-Loop AILLM Agent Reliability
AskPDF