Spotlight: Identifying and Localizing Video Generation Errors Using VLMs

As Text-to-Video (T2V) models progress towards higher vi-sual realism, the artifacts and errors they produce are highly nuanced,fine-grained, and spatio-temporally localized. Vision Language Models(VLMs) are actively being adopted as automatic evaluators for videogeneration, driven by the promise of their perception and reasoning abili-ties. Yet, it remains unclear whether they can detect, localize, and explainfine-grained errors in modern high-fidelity video generations. We intro-duce Spotlight, a novel benchmark to rigorously assess whether cur-rent VLMs can precisely localize and explain nuanced video-generationerrors. Spotlight comprises 600 videos generated by state-of-the-artT2V models (Veo3, Seedance, LTX-2), annotated with over 1,600 fine-grained error localizations and explanations spanning physics, semanticsand anatomy. Our experiments reveal that current VLMs lag behindhumans significantly, with humans outperforming our best baselines bynearly 2× on the task. Our analysis highlights key areas of improvementfor utilizing VLMs as automated video evaluators, including the need formore robust perception and hallucination mitigation. Data and code isavailable at https://spotlight-video.github.io.