Organizations: Zhejiang University · State Key Lab of CAD&CG, Zhejiang University · Hong Kong University of Science and Technology · School of Software Engineering, South China University of Technology · School of Software Technology, Zhejiang University
Large Language Models (LLMs) demonstrate exceptional capabilities in factual question answering, yet they sometimes provide incorrect responses. To address this issue, knowledge editing techniques have emerged as effective methods for correcting factual information in LLMs. However, typical knowledge editing workflows struggle with identifying the optimal set of model layers for editing and rely on summary indicators that provide insufficient guidance. This lack of transparency hinders effective comparison and identification of optimal editing strategies. In this paper, we present KEditVis, a novel visual analytics system designed to assist users in gaining a deeper understanding of knowledge editing through interactive visualizations, improving editing outcomes, and discovering valuable insights for the future development of knowledge editing algorithms. With KEditVis, users can select appropriate layers as the editing target, explore the reasons behind ineffective edits, and perform more targeted and effective edits. Our evaluation, including usage scenarios, expert interviews, and a user study, validates the effectiveness and usability of the system.
Figures & tables
Figure 1: Example of knowledge editing.
Figure 2: The interface of our visual analytics system KEditVis. The system provides (A1) an LLM chat view for interactive dialogue, (A2) a knowledge graph, and (A3) management of editing facts and testing prompts. The edit view visualizes (B1 and B4) key internal data of the model layers both before and after editing, facilitating the selection of appropriate layers for editing and comparison of different model versions. The edit view leverages (B2) set visualization to show the relationship between different schemes, and combines (B3) both overview results and detailed results to compare and evaluate the editing outcomes across schemes. (C) The output comparison view employs text visualization to compare the model’s outputs before and after model editing. (D) The drift view visualizes the global impact of edits on the model through a scatter plot.
Figure 3: Scenario I: (A) The knowledge graph generated by the keyword “iPhone”; (B) the cosine similarity bar chart before editing; (C) the descending comparison results after selecting scheme 6-12 and pressing the “Recommend” button and the “Compare” button; (D) the results after adding scheme 6-7 to the comparison; (E) the cosine similarity chart and ranking chart after editing scheme 6-10; (F) the results after adding schemes 20-21, 8-14, and 6-8 on top of the previously edited weights for scheme 6-10; (G) the scatter plot for checking the global impact of editing.
Figure 4: Scenario II: (A) The knowledge graph generated by the keyword “Turing Award”; (B) the descending sorting of editing results from different layer selection schemes; (C) the ranking chart (for the last token) and the cosine similarity bars, before and after editing scheme 10-12.
Figure 5: (A) Diagram of the set visualization we designed; (B) results of the SUS questionnaire in the user study.