VLM Interpretability

Vision-language model interpretability studies how models encode visual and textual information, combine it, and use it to produce answers, predictions, or actions. Researchers use saliency and attribution maps, probing and sparse autoencoders, and mechanistic methods such as activation patching and circuit analysis to locate relevant features, layers, and attention heads; causal interventions test whether those components actually affect behavior. Evaluations span image-text retrieval and classification, visual question answering, grounding, counting, spatial reasoning, and hallucination benchmarks such as POPE and CHAIR. A central tension is that information may be decodable or highlighted by attention without guiding the model’s answer, so explanation faithfulness and actual use of visual evidence remain open challenges.