VLM Distillation

Vision-language model distillation studies how to transfer image-and-text understanding from a teacher model to a smaller or more efficient student, often for tasks such as visual question answering, retrieval, video reasoning, and robotics. Methods align teacher and student features, logits, attention, or relationships between visual and text representations; others train on teacher-generated rationales or student-generated responses, sometimes using multiple teachers or privileged visual cues. Work also targets practical constraints such as fewer visual tokens, quantization, and deployment on devices with limited memory or compute. A central challenge is deciding which teacher signals are reliable and grounded in evidence the student can access, since copying a teacher too closely can preserve its errors or encourage shortcuts rather than visual reasoning.