hoBIT: A Profile-Aware Retrieval-Augmented Chatbot for University Academic Advising

Yoonseo Kim; Seongmin Lee; Joongheon Kim; SeongKu Kang

cs.IRAug 27, 2026

hoBIT: A Profile-Aware Retrieval-Augmented Chatbot for University Academic Advising

Authors: Yoonseo Kim, Seongmin Lee, Joongheon Kim, SeongKu Kang

Organizations: Korea University Seoul, Korea

Abstract

In university academic advising, identical questions can require different answers depending on a student's department, admission cohort, and degree program, causing profile-blind retrievers to surface plausible but inapplicable evidence. We present proFILL, a method for transforming hoBIT, our college's current rule-based advising chatbot, into a profile-aware retrieval-augmented generation (RAG) system. Rather than requiring a complete user profile upfront, proFILL progressively acquires only the profile attributes needed for each query, guided by both the query intent and the initially retrieved evidence, and uses them to condition retrieval over a profile-aware index. Extensive experiments and a human preference study show that proFILL outperforms diverse RAG baselines, is preferred by target users, and remains effective with open-weight models for cost-effective on-premise deployment.

Figures & tables

Figure 1: A conceptual comparison of (top) conventional RAG, which struggles with profile-dependent queries over semantically similar documents, and (bottom) the proposed hoBIT, which adaptively acquires the required profile information and retrieves the applicable document from a profile-indexed corpus.

Figure 2: Overview of hoBIT. Offline (left): each chunk is annotated with its applicable profile values and required fields before indexing. Online (right): query-driven profiling acquires missing attributes for profile-aware retrieval, while evidence-driven profiling requests additional information and re-retrieves when needed.

		Retrieval					Generation
	Retrieval System	MRR	R@1	R@5	R@10	R@50	ROUGE-L	Token-F1	Keyword Match	Source Match	Grounded Correctness	s/q
Deployment	BM25	0.089	0.017	0.118	0.267	0.728	0.139	0.167	0.564	0.412	0.412	4.0
	Dense	0.031	0.011	0.022	0.049	0.400	0.149	0.176	0.548	0.217	0.292	4.3
	Hybrid	0.061	0.009	0.069	0.152	0.698	0.146	0.173	0.574	0.380	0.395	4.3
	$+$ reranker	0.111	0.027	0.165	0.322	0.698	0.152	0.177	0.621	0.391	0.425	4.6
	HyDE	0.061	0.013	0.057	0.141	0.689	0.147	0.175	0.586	0.367	0.403	6.9

Table 1: Retrieval and generation performance. For all methods, gpt-4o-mini is used as the answer generator. s/q denotes end-to-end latency in seconds per query; absolute latency may vary with the hardware configuration and external LLM API response time.

Panel of Figure 3 — Figure 3: Human preference comparison. Results are reported as three-way vote splits. Question IDs are prefixed by the corresponding department.

Embedder	Setting	MRR	R@1	R@5	R@10	R@50
openai_small	proFILL	0.596	0.495	0.721	0.784	0.936
	w/o Hard filtering	0.311	0.157	0.502	0.671	0.923
	w/o Soft aug.	0.247	0.137	0.337	0.501	0.893
	w/o Re-retrieval	0.580	0.481	0.702	0.767	0.934
BGE-M3	proFILL	0.665	0.579	0.775	0.796	0.936
	w/o Hard filtering	0.475	0.349	0.624	0.704	0.934

Table 2: Ablation study. Results are reported with three different dense embedding models for retrieval: text-embedding-3-small ( OpenAI, 2024 ) , BGE-M3 ( Chen et al., 2024 ) , and Qwen3-Embedding ( Qwen Team, 2025 ) .

		Lexical		Matching
	Generator	ROUGE-L	Token-F1	Kw. Match	Src. Match	GC
Prop.	gpt-4o-mini	0.179	0.207	0.733	0.847	0.676
Open- weight	Qwen3-8B	0.169	0.197	0.721	0.879	0.670
	Ministral-8B	0.155	0.177	0.723	0.734	0.639
	LLaMA3.1-8B	0.204	0.230	0.665	0.817	0.649
	EXAONE3.5-7.8B	0.135	0.158	0.704	0.898	0.712

Table 3: Results across various LLMs, including proprietary and open-weight models. GC denotes Grounded Correctness.

Figure 4: The web interface with examples for two queries. proFILL requests only the information needed for each query on demand and uses it to answer the user’s question.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Intent class	Precision	Recall	F1	#Queries
greeting	1.000	0.630	0.773	100
ability	0.842	0.960	0.897	100
faq	0.971	1.000	0.985	100
smalltalk	0.704	1.000	0.826	100
retrieval	1.000	0.981	0.990	1,200

Appendix

Table 4: Intent classification results.

Category	Top-3 Precision	Completeness	#Queries
Academic status	0.713	0.973	100
Facilities/dining	0.683	0.931	100
Academic operations	0.670	0.935	100
Scholarships	0.643	0.933	100
Space/facility	0.633	0.944	100
Enrollment/tuition	0.593	0.921	100

Appendix

Table 5: Open-ended advising results by category.

ID	Query Topic	Win	Tie	Lose	Win%
DS-01	Graduation requirements	40	6	2	95
DS-02	Available major courses	25	4	18	58
DS-03	Required major courses	42	3	3	93
AI-01	Graduation requirements	43	3	2	96
AI-02	GE-required courses	27	13	7	79
AI-03	Foundational courses	21	15	11	66

Appendix

Table 6: Results of the per-question human preference evaluation.

Adaptive RAG

Adaptive retrieval-augmented generation (RAG) adjusts when, what, and how much evidence a language model retrieves for each query.

See all 59 papers

KohakuRAG: A simple RAG framework with hierarchical document indexing
Mar 8, 2026Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang +1Retrieval-Augmented Generation Structured RAG
National Tsing Hua University · Comfy Org Research · Kohaku-Lab
Know Before You Fetch: Calibrated Retrieval-Budget Allocation for Retrieval-Augmented Generation
Jun 29, 2026Zhe Dong, Fang Qin, Manish Shah +1Retrieval-Augmented Generation Selective Prediction
University of Maine at Presque Isle · Stanford University · Independent Researcher
Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems
May 5, 2026Yilun Zhao, Jinbiao Wei, Tingyu Song +3Agentic Search Synthetic Data Generation
Yale NLP Lab · National University of Singapore · NYU Shanghai

Mar 8, 2026J/K move · Enter open · S save

KohakuRAG: A simple RAG framework with hierarchical document indexing

Shih-Ying Yeh, Yueh-Feng Ku, Ko-Wei Huang, Buu-Khang Tu

National Tsing Hua University · Comfy Org Research · Kohaku-Lab

Retrieval-Augmented Generation Structured RAG Multi-Document QA Knowledge-Intensive QA

Open paper AskPDF arXiv

Retrieval-augmented generation (RAG) systems that answer questions from document collections face compounding difficulties when high-precision citations are required: flat chunking strategies sacrifice document structure, single-query formulations miss relevant passages through vocabulary mismatch, and single-pass inference produces stochastic answers that vary in both content and citation selection. We present KohakuRAG, a hierarchical RAG framework that preserves document structure through a four-level tree representation (document

\rightarrow

section

\rightarrow

paragraph

\rightarrow

sentence) with bottom-up embedding aggregation, improves retrieval coverage through an LLM-powered query planner with cross-query reranking, and stabilizes answers through ensemble inference with abstention-aware voting. We evaluate on the WattBot 2025 Challenge, a benchmark requiring systems to answer technical questions from 32 documents with

\pm

0.1% numeric tolerance and exact source attribution. KohakuRAG achieves first place on both public and private leaderboards (final score 0.861), as the only team to maintain the top position across both evaluation partitions. Ablation studies reveal that prompt ordering (+80% relative), retry mechanisms (+69%), and ensemble voting with blank filtering (+1.2pp) each contribute substantially, while hierarchical dense retrieval alone matches hybrid sparse-dense approaches (BM25 adds only +3.1pp). We release KohakuRAG as open-source software at https://github.com/KohakuBlueleaf/KohakuRAG.

Abstract

Adaptive RAG

Explore similar work

KohakuRAG: A simple RAG framework with hierarchical document indexing

Know Before You Fetch: Calibrated Retrieval-Budget Allocation for Retrieval-Augmented Generation

Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems