Knowledge-Based VQA

Knowledge-based visual question answering studies questions that require combining what is visible in an image with facts from sources such as Wikipedia, knowledge graphs, documents, or a model’s internal knowledge. Common approaches retrieve images or text, identify entities, and use multimodal language models or search agents to select evidence and reason over it; other methods organize evidence as graphs or highlight relevant regions and passages. Training ranges from supervised retrieval and answer generation to reinforcement learning for multi-step search, while benchmarks test encyclopedic questions, multi-hop reasoning, and specialized settings such as medicine or navigation. A central challenge is ensuring that models use relevant, trustworthy evidence rather than noisy or conflicting retrievals, and that evaluations measure grounded reasoning rather than answer accuracy alone.