Designing Informative Actions: Action Construction for Long-Horizon Research Agents
Abstract
Large language model agents support automated scientific research by connecting literature analysis and planning with experimentation. Evaluating which action an agent selects does not establish which experiment its instruction specifies, because omitted interventions or replication counts leave consequential choices to the executor. We introduce a reading-set framework for evaluating action construction under a common cost rule. Across 30 synthetic states, complete-design requests increase affordable efficiency by 465.98% and 327.91% relative to the action-name baseline in the two model families, respectively. An author-constructed research-action evaluation dataset provides 60 scenarios for complementary judgment evaluation. On this dataset, the scenario-level information-per-cost measure has a mean within-scenario correlation of 0.733 with judgments from 18 human participants. The framework makes design specification and execution ambiguity measurable, providing a reproducible basis for assessing how research agents translate scientific intent into informative, budget-feasible experiments.
Figures & tables
| Setting | L1 Action-name interface | L2 Target interface | L3 Complete-design interface | L4 Design-with-prediction interface |
|---|---|---|---|---|
| DeepSeek (A)/low | 13.4 | 25.4 | 76.9 | 77.0 |
| DeepSeek (A)/max | 13.4 | 25.3 | 74.2 | 72.6 |
| Qwen3.8-27B (B)/na | 13.4 | 25.2 | 61.0 | 57.7 |
| Qwen3.8-27B (B)/low | 13.4 | 25.0 | 53.3 | 49.3 |
| DeepSeek (reasoning effort = max) | |
|---|---|
| Method | Efficiency (%) |
| Think then act (ReAct-inspired) [ 36 ] | 70.97 |
| Plan-and-Solve adaptation [ 37 ] | 70.98 |
| One-round Self-Refine [ 38 ] | 71.88 |
| Action-name interface (L1) | 13.36 |
| Target interface (L2) | 25.29 |
| Model | Pooled | Mean within-scenario |
|---|---|---|
| Yi-1.5-6B-Chat | 0.309 | 0.258 |
| InternLM2.5-7B-chat | 0.588 | 0.594 |
| GLM-4-9B-chat | 0.570 | 0.568 |
| Granite-3.3-8B-Instruct | 0.502 | 0.467 |
| Phi-4-mini-instruct | 0.244 | 0.217 |
| Mistral-7B-Instruct-v0.3 | 0.262 | 0.252 |
Appendix figures & tables10 assets
Supplementary material from the paper’s appendix.
Appendix
| Level | Interface | Requested fields | Executor interpretation |
|---|---|---|---|
| L1 | Action name | ACTION, REASON | A fixed set of generic designs; design details in REASON are not extracted. |
| L2 | Target | TARGET, WHY | Target-conditioned interventions; seed count completed by the executor. |
| L3 | Complete design | TARGET, ARMS, SEEDS, SUCCESS | One design when the scored fields parse and satisfy the domain rules. |
| L4 | Design + prediction | L3 fields plus PREDICTION | The same design scoring rule; the prediction is stored for inspection. |
| Setting | Level | Parsed /90 | All feasible /90 | Feasible readings (%) |
|---|---|---|---|---|
| A/low | L1 | 90 | 57 | 90.8 |
| A/low | L2 | 90 | 57 | 81.7 |
| A/low | L3 | 90 | 90 | 100.0 |
| A/low | L4 | 90 | 90 | 100.0 |
| A/max | L1 | 90 | 57 | 90.8 |
| A/max | L2 | 90 | 57 | 81.7 |
| Family | Contrast | |||
|---|---|---|---|---|
| A | L1 L2 | 0.1201 | 0.76 | 0.00197 |
| A | L1 L3 | 0.6224 | 3.47 | 7.42e-17 |
| A | L1 L4 | 0.6142 | 3.49 | 6.44e-17 |
| A | L3 L4 | -0.0082 | -0.13 | 1 |
| B | L1 L2 | 0.1176 | 0.74 | 0.00245 |
| B | L1 L3 | 0.4380 | 2.89 | 7.66e-15 |
| Method | Efficiency (%) |
|---|---|
| Think then act (ReAct-inspired) [ 36 ] | 55.92 |
| Plan-and-Solve adaptation [ 37 ] | 68.20 |
| One-round Self-Refine [ 38 ] | 70.05 |
| Action-name interface (L1) | 13.36 |
| Target interface (L2) | 25.23 |
| Complete-design interface (L3) | 60.98 |
| Level | Arms | Seeds | Cost | Score (%) |
|---|---|---|---|---|
| L1 | Reading set | – | 3.08 | 8.40 |
| L2 | Reading set | – | 3.95 | 27.66 |
| L3 | 1,3 | 8 | 5.00 | 55.51 |
| L4 | 0,1,2,3,4,5 | 9 | 7.75 | 94.34 |
| Comparison | Aggregation | Spearman |
|---|---|---|
| Model panel / metric | Pooled across 240 actions | 0.566 |
| Humans / metric | Mean within 60 scenarios | 0.733 |
| Model panel / humans | Mean within 60 scenarios | 0.824 |
| A effort | L1 | L1 padded | L3 short | L3 full | Gain retained (%) |
|---|---|---|---|---|---|
| low | 13.36 | 13.36 | 77.87 | 76.94 | 101.5 |
| max | 13.36 | 13.21 | 75.14 | 74.25 | 101.5 |
| Setting | Level | |||||
|---|---|---|---|---|---|---|
| A/low | L1 | 90 | 0.0000 | 0.5249 | 0.5249 | 0 |
| A/low | L2 | 90 | 0.0000 | 0.3877 | 0.3877 | 14 |
| A/low | L3 | 90 | 0.0000 | 0.1257 | 0.1257 | 0 |
| A/low | L4 | 90 | 0.0000 | 0.1344 | 0.1344 | 0 |
| A/max | L1 | 90 | 0.0000 | 0.5249 | 0.5249 | 0 |
| A/max | L2 | 90 | 0.0000 | 0.3887 | 0.3887 | 13 |
| Setting | Level | Conservative | Default | Permissive |
|---|---|---|---|---|
| A/low | L1 | 10.81 | 13.36 | 15.90 |
| A/low | L2 | 14.37 | 25.45 | 36.52 |
| A/low | L3 | 76.94 | 76.94 | 76.94 |
| A/low | L4 | 76.99 | 76.99 | 76.99 |
| A/max | L1 | 10.81 | 13.36 | 15.90 |
| A/max | L2 | 14.28 | 25.29 | 36.30 |
| Calibration policy | Score (%) |
|---|---|
| Full-domain affordable oracle | 100.0 |
| Top-prior target + exhaustive design search | 100.0 |
| Uniform over affordable designs | 44.3 |
| Top-prior single arm, two seeds | 10.7 |
| Fixed six arms, five seeds | 15.2 |
AI Agents for Scientific Discovery
AI agents for scientific discovery use language models, tools, and experiments to generate, test, and refine scientific knowledge.
See all 323 papers