Orcheo: A Modular Full-Stack Platform for Conversational Search

Shaojie Jiang; Svitlana Vakulenko; Maarten de Rijke

cs.IRFeb 16, 2026

Orcheo: A Modular Full-Stack Platform for Conversational Search

Authors: Shaojie Jiang, Svitlana Vakulenko, Maarten de Rijke

Organizations: University of Amsterdam · AI Colleagues · WU Vienna University of Economics and Business

Abstract

Conversational search (CS) requires a complex software engineering pipeline that integrates query reformulation, ranking, and response generation. CS researchers currently face two barriers: the lack of a unified framework for efficiently sharing contributions with the community, and the difficulty of deploying end-to-end prototypes needed for user evaluation. We introduce Orcheo, an open-source platform designed to bridge this gap. Orcheo offers three key advantages: (i) A modular architecture promotes component reuse through single-file node modules, facilitating sharing and reproducibility in CS research; (ii) Production-ready infrastructure bridges the prototype-to-system gap via dual execution modes, secure credential management, and execution telemetry, with built-in AI coding support that lowers the learning curve; (iii) Starter-kit assets include 45+ off-the-shelf components for query understanding, ranking, and response generation, enabling the rapid bootstrapping of complete CS pipelines. We describe the framework architecture and validate Orcheo's utility through case studies that highlight modularity and ease of use. Orcheo is released as open source under the MIT License at https://github.com/AI-Colleagues/orcheo.

Figures & tables

Framework	Primary focus	CS pipeline support	Modular sharing	Deployment tools	Evaluation support
Orcheo	Workflow orchestration	Native	Node registry	Full-stack	Built-in nodes
DECAF ( Alessio et al., 2023 )	CS experimentation	Native	Components	N/A	CAsT baselines
Chatty Goose ( Zhang et al., 2021 )	CS passage retrieval	Native	Components	Via ParlAI	CAsT ’19 baselines
ConvLab-2 ( Zhu et al., 2020 )	Task-oriented dialogue	Partial	Plugin system	Limited	Dialogue metrics
ParlAI ( Miller et al., 2017 )	Conversational AI	Limited	Agent zoo	Limited	Dialogue metrics
Pyserini ( Lin et al., 2021 )	Sparse/dense retrieval	Component only	N/A	N/A	External

Table 1. Comparison of Orcheo with related frameworks and platforms.

Figure 1. Orcheo’s node hierarchy. BaseNode provides shared infrastructure; AINode and TaskNode specialize for LLM-based and utility operations, respectively. Class hierarchy diagram showing BaseNode at the top, AINode and TaskNode as intermediate abstractions, and example concrete nodes (AgentNode, LLMNode, RetrievalEvaluationNode, DenseSearchNode, ReRankerNode) at the bottom.

Figure 2. Orcheo platform architecture showing the interaction between the presentation layer, backend, and execution components. Layered architecture diagram with four stacked layers: Presentation (ChatKit UI, SDK/CLI, Tracing Viewer), API (ChatKit Server, REST API, WebSocket, Authentication), Execution (Trigger Layer, LangGraph Runtime, Celery Workers, Celery Beat scheduler), and Persistence (Workflow DB, Checkpoint Store, Run History Store, Credential Vault, Redis Broker), connected by arrows from top to bottom.

Figure 3. Grounded generation pipeline expressed as an Orcheo graph. This pipeline is used as the system under test in the MultiDoc2Dial evaluation (Section 5.3 ). Workflow diagram showing a linear pipeline of four nodes: QueryRewrite, DenseSearch, ContextCompressor, and GroundedGenerator.

Benchmark	Metric	Orcheo	Published
QReCC	ROUGE-1 Recall	75.25	89.50
QReCC	Semantic Similarity	79.00	95.20 †
MultiDoc2Dial	Token F1	0 8.34	48.04
	SacreBLEU	13.32	34.56
	ROUGE-L	0 6.82	45.93

Table 2. Evaluation results on QReCC and MultiDoc2Dial using Orcheo workflows with GPT-4o-mini, alongside published reference results.

Figure 4. MultiDoc2Dial evaluation workflow. Metric nodes execute in parallel branches and results are merged by the analytics node. Evaluation workflow diagram where the MultiDoc2Dial dataset feeds into a conversational batch evaluation node, which then branches to three metric nodes (TokenF1, BleuMetrics, RougeMetrics) whose outputs are merged by an AnalyticsExport node.

Conversational Search

Conversational search uses dialogue to interpret evolving information needs, retrieve relevant material, and help people refine or verify results.

See all 23 papers

SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks
May 9, 2026Jinchao Hu, Meizhi Zhong, Kehai Chen +1Tool-Augmented Language Model Agents Agentic Search
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Independent Researcher Shenzhen / Beijing, China
OpenSaC: A Python Library for Search as Code in Deep Research Agents
Oct 19, 2026Qi Liu, Jiaxin Mao, Fengbin Zhu +2Agentic Search Deep Research Agents
Renmin University of China Beijing, China · National University of Singapore Singapore, Singapore
Two Views, One Voice: Evidence-Grounded Conversational Music Recommendation
Jul 24, 2026Sungwook Yoo, Sewook YooRecommender Systems Music Recommendation
Samsung South Korea

May 9, 2026J/K move · Enter open · S save

SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks

Jinchao Hu, Meizhi Zhong, Kehai Chen, Min Zhang

School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen · Independent Researcher Shenzhen / Beijing, China

Tool-Augmented Language Model Agents Agentic Search LLM Agent Skill Learning LLM Tool Use Knowledge-Intensive QA

Open paper AskPDF arXiv

Teaching language models to use search tools is not only a question of whether they search, but also of whether they issue good queries. This is especially important in open-domain question answering, where broad or copied queries often waste retrieval budget and derail later reasoning. We propose \Ours, a framework that makes query planning explicit through reusable search skills. At each step, the model first selects a skill, then generates a search or answer action conditioned on the selected skill card. The skill inventory itself is not fixed: SearchSkill maintains an evolving SkillBank, expands or refines it from recurrent failure patterns, and reconstructs affected trajectories before supervised training. The resulting two-stage SFT recipe aligns training with the inference-time protocol of skill selection followed by skill-grounded execution. Across open-source and closed-source models, SearchSkill improves exact match on knowledge-intensive QA benchmarks and yields better retrieval behavior, including fewer copied first queries, more atomic hop-focused queries, and more correct answers within a small search budget. These results suggest that explicit skill-conditioned query planning is a lightweight alternative to treating search as an undifferentiated action.

Abstract

Conversational Search

Explore similar work

SearchSkill: Teaching LLMs to Use Search Tools with Evolving Skill Banks

OpenSaC: A Python Library for Search as Code in Deep Research Agents

Two Views, One Voice: Evidence-Grounded Conversational Music Recommendation