WAR-CRAFT: Refining Web API Recommendations through Requirement Auditing
Organizations: University of Illinois Urbana-Champaign Urbana, United States · Huazhong University of Science and Technology Wuhan, China · Mohamed bin Zayed University of Artificial Intelligence Abu Dhabi, United Arab Emirates
Abstract
Web API recommenders return a set of APIs for a mashup request but do not check it against provider documentation, the main public evidence of an API's capabilities. The set can therefore contain unsupported or redundant APIs and miss requested capabilities. Filling every apparent gap does not fix this, because a gap may already be served, may not need its own API, or may have no documented provider. We study post-recommendation refinement, which revises a frozen recommender's output using only the request, the catalog, and provider documentation. We propose WAR-CRAFT, which treats refinement as three decisions. An LLM removes unsupported or subsumed APIs by judging candidates jointly, then audits the remaining set against a requirement checklist derived from the request. For each uncovered requirement, a fine-tuned catalog-constrained Resolver proposes a catalog API or abstains, and an independent documentation check admits only supported proposals. On ProgrammableWeb with 7 heterogeneous upstream recommenders, WAR-CRAFT raises mean SetF1 from 0.558 to 0.644, above the strongest of 5 refinement baselines (62.5%), and roughly halves functional redundancy (0.127 to 0.065) without changing mean functional coverage. Gains range from +2.2 points on generative recommenders to +18.2 on fixed-cutoff rankers. At matched set sizes, WAR-CRAFT still outperforms truncating the upstream output, so the gain is not explained by set-size calibration alone. Our code is available at https://github.com/ZsXu-enrico/WAR-CRAFT.
Figures & tables
| Base | System | SetP | SetR | SetF1 | NDCG | MAP | Cov. | Eff. | CovF1 | Red. | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|---|
| WAR-R1 | Initial | 0.7282 | 0.6771 | 0.6800 | 0.7032 | 0.6643 | 0.4414 | 0.5874 | 0.4574 | 0.0657 | 1.70 |
| + WAR-CRAFT | 0.7739 | 0.6937 | 0.7038 | 0.7145 | 0.6737 | 0.4439 | 0.6138 | 0.4718 | 0.0444 | 1.50 | |
| LLMAR | Initial | 0.7215 | 0.6734 | 0.6689 | 0.6909 | 0.6514 | 0.4363 | 0.5542 | 0.4370 | 0.0700 | 1.95 |
| + WAR-CRAFT | 0.7476 | 0.6927 | 0.6911 | 0.7076 | 0.6661 | 0.4509 | 0.5830 | 0.4617 | 0.0690 | 1.75 | |
| GPT | Initial | 0.6544 | 0.5804 | 0.5912 | 0.5987 | 0.5641 | 0.4085 | 0.5532 | 0.4309 | 0.0417 | 1.35 |
| + WAR-CRAFT | 0.6727 | 0.6095 | 0.6139 | 0.6254 | 0.5892 | 0.4320 | 0.5708 | 0.4497 | 0.0483 | 1.45 |
| SetF1 | ||||||
|---|---|---|---|---|---|---|
| Method | All | Cov. | CovF1 | |||
| Initial | 0.5580 | 0.5960 | 0.5137 | 0.4699 | 0.4574 | 2.09 |
| Trunc. to | 0.6009 | 0.6767 | 0.5126 | 0.4423 | 0.4473 | 1.87 |
| WAR-CRAFT | 0.6444 | 0.7206 | 0.5557 | 0.4835 | 0.4790 | 1.93 |
| Trunc. to (oracle) | 0.6240 | 0.7105 | 0.5233 | 0.5031 | 0.4559 | 2.71 |
| Method | SetF1 | CovF1 | Red. | Avg. | Calls |
| Initial | 0.5580 | 0.4479 | 0.1265 | 1.86 | 0 |
| Independent refinement | |||||
| Tuned- | 0.5982 | 0.4228 | 0.0227 | 1.23 | 0 |
| MMR-Cut | 0.6243 | 0.4349 | 0.0254 | 1.27 | 0 |
| DPP-Cut | 0.6252 | 0.4365 | 0.0238 | 1.26 | 0 |
| R&S | 0.5334 | 0.4631 | 0.0924 | 1.63 | 1.00 |
| Stage | SetF1 | CovF1 | Red. | Avg. |
|---|---|---|---|---|
| Initial | 0.5580 | 0.4479 | 0.1265 | 1.86 |
| + Role Writer | 0.5911 | 0.4396 | 0.0485 | 1.35 |
| + Audit | 0.6443 | 0.4715 | 0.0658 | 1.60 |
| + Verifier | 0.6444 | 0.4715 | 0.0654 | 1.59 |
| Source | Empty | Alignment | Grounding | Clarity | FRS |
|---|---|---|---|---|---|
| WAR-R1 | 83 | 0.9543 | 0.9349 | 0.9672 | 0.9521 |
| LLMAR | 39 | 0.9468 | 0.9210 | 0.9608 | 0.9429 |
| GPT | 115 | 0.9516 | 0.9223 | 0.9590 | 0.9443 |
| CVH-REC | 31 | 0.9320 | 0.9185 | 0.9539 | 0.9348 |
| MARS | 25 | 0.9334 | 0.9123 | 0.9549 | 0.9336 |
| WARBERT | 47 | 0.9279 | 0.9111 | 0.9498 | 0.9296 |
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
| Source | Tuned- | MMR-Cut | DPP-Cut |
|---|---|---|---|
| LLMAR | 4 | ||
| CVH-REC | 1 | ||
| GPT | 4 | ||
| MARS | 1 | ||
| WARBERT | 1 | ||
| Tool-Embed-4B FT | 1 |
| Source | SetF1 | CovF1 | Cov. | Red. |
|---|---|---|---|---|
| WAR-R1 | ||||
| LLMAR | ||||
| GPT | ||||
| CVH-REC | ||||
| MARS | ||||
| WARBERT |
| SetF1 | CovF1 | |||||||||||||
| Method | WAR-R1 | LLMAR | GPT | CVH-REC | MARS | WARBERT | Tool | WAR-R1 | LLMAR | GPT | CVH-REC | MARS | WARBERT | Tool |
| Initial | 0.6800 | 0.6689 | 0.5912 | 0.4698 | 0.5506 | 0.4274 | 0.5184 | 0.4574 | 0.4370 | 0.4309 | 0.4564 | 0.4812 | 0.4271 | 0.4452 |
| Independent refinement | ||||||||||||||
| Tuned- | 0.6802 | 0.6689 | 0.5907 | 0.5225 | 0.6189 | 0.4737 | 0.6330 | 0.4570 | 0.4388 | 0.4314 | 0.4055 | 0.4277 | 0.3780 | 0.4214 |
| MMR-Cut | 0.6809 | 0.6702 | 0.5912 | 0.6077 | 0.6182 | 0.5618 | 0.6404 | 0.4507 | 0.4400 | 0.4302 | 0.4196 | 0.4492 | 0.4053 | 0.4493 |
| DPP-Cut | 0.6810 | 0.6659 | 0.5912 | 0.6109 | 0.6192 | 0.5646 | 0.6437 | 0.4580 | 0.4366 | 0.4309 | 0.4301 | 0.4438 | 0.4134 | 0.4429 |
Recommender Systems
Recommender systems predict and rank items for people using interaction histories, content, context, and feedback.
See all 573 papers