Large Language Models (LLMs) have shown a high capability in answering questions on a diverse range of topics. However, these models sometimes produce biased, ideologized or incorrect responses, limiting their applications if there is no clear understanding of which topics their answers can be trusted. In this research, we introduce a novel algorithm, named as GMRL-BD, designed to identify the untrustworthy boundaries (in terms of topics) of a given LLM, with black-box access to the LLM and under specific query constraints. Based on a general Knowledge Graph (KG) derived from Wikipedia, our algorithm incorporates with multiple reinforcement learning agents to efficiently identify topics (some nodes in KG) where the LLM is likely to generate biased answers. Our experiments demonstrated the efficiency of our algorithm, which can detect the untrustworthy boundary with just limited queries to the LLM. Additionally, we have released a new dataset containing popular LLMs including Llama2, Vicuna, Falcon, Qwen2, Gemma2 and Yi-1.5, along with labels indicating the topics on which each LLM is likely to be biased.
Figures & tables
Figure 1. GMRL-BD - Training Stage : Agent maximize rewards and enhance graph representations by interacting with nodes on the KG, target LLM outcomes, refining RL policies, and pinpointing biased nodes. Inference Stage : Multiple agents collaboratively navigate the KG to uncover biases in the black-box LLM. Agents coordinate to efficiently detect untrustworthy boundaries and dynamically adjust their exploration paths. For instance, nested bias topics, such as "multiculturalism," can be effectively identified by leveraging agents information exchange and the hierarchical structure of categories.
Figure 2. GMRL-BD Model Architecture. Agents interact with nodes on the KG to maximize diffusion navigation rewards and optimize graph representations through policy network. For inference stage, multiple agents explore the KG to detect biases in the black-box LLMs. Starting from different topic nodes, agents coordinate to efficiently identify untrustworthy boundaries, minimizing conflicts of action, with the AgentLinkHub facilitates communication and parameter sharing
Figure 3. Performance of the GMRL-BD Framework Across Six Tasks Using Llama-2, Vicuna, and Falcon Models. These graphs display the number of interactions for each LLM to detect untrustworthy boundaries across six distinct tasks. Each subplot depicts the convergence rate of the models, with lower interaction counts indicates a more efficient performance
Figure 4. Bias Transmission Rate.
Models
Immigration
Politics
Education
Economy
Llama-2
90.34
76.67
71.63
86.00
Vicuna
78.61
53.79
58.25
63.41
Falcon
90.69
70.41
65.50
83.88
Table 1. Bias Node Cover Rate(BNC) across Different LLMs and Topics. This table shows the recall performance of the GMRL-BD framework across different LLMs and topics.
Methods
Immigration-Task
Culture-Task
AI-Task
GMRL-BD
0.895
0.915
0.96
w/o sage layer
0.751
0.725
0.863
w/o att layer
0.802
0.778
0.9
w/o sage layer+att layer
0.618
0.319
0.75
Graph-enhanced Attention Driven Q-Network
0.710
0.770
0.845
Uniform Exploration and Exploitation
0.705
0.265
0.380
Table 2. Ablation Study Results for GMRL-BD Framework Across Different Tasks. This table presents the results of our ablation study, comparing the success rates of the GMRL-BD framework with and without key components (sage layer and attention layer) across three tasks: Immigration, Culture, and AI.
Figure 5. Impact of Increasing Number of Agents on Step Loss and Win Rate in Multi-Agent Collaborative Parameter Sharing
Large language model auditing tests models and deployed systems for bias, safety, privacy, reliability, and provenance using behavioral and internal evidence.
State Key Laboratory of Integrated Services Networks, School of Cyber Engineering, Xidian University · Hangzhou Institute of Technology, Xidian University · Department of Computer Science, Purdue University