NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models

Out-of-Distribution (OOD) detection is essential for ensur-ing the robustness and reliability of object detection systems deployed insafety-critical applications. While prior research has mainly focused onuni-modal detectors or vision-language model (VLM) based classifiers,the potential of VLM-based object detectors in OOD scenarios remainsunderexplored. In this work, we take the first step toward building OODobject detection methods upon VLMs. We identify two key challenges:(i) learning discriminative features to separate in-distribution (ID) fromOOD instances, and (ii) designing scoring functions consistent with VLMprobabilistic outputs. Hence, we introduce Negative Label Guided Atten-tion and Scoring (NegAS). To address (i), we propose a negative labelguided attention module (NegA), where LLM-generated, visually-similarbut semantically-different negative labels are used to guide attention to-ward potential OOD background regions. To address (ii), we introducea novel sigmoid-based OOD scoring function (NegS) that leverages bothID and negative labels, producing strong responses for ID instances andsuppressed responses for OOD ones. Extensive experiments demonstratethat our approach improves OOD detection performance by a large mar-gin while maintaining competitive ID accuracy, e.g., reducing the FPR95by 11.4% on the COCO dataset and 25.5% on the OpenImages datasetcompared to the baseline model. While initially designed for dense VLMdetectors like YOLO-World, we successfully adapt NegAS to GroundingDINO, a query-based VLM transformer, achieving significant improve-ments, demonstrating the generalizability of our framework. The code isavailable at https://github.com/yjzzyj/NegAS.