Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization

Multimodal large language models (MLLMs) have made rapidprogress, yet they still exhibit object hallucination, generating plausiblebut incorrect descriptions that are inconsistent with the visual input. Di-rect Preference Optimization (DPO) mitigates this by training modelsto prefer non-hallucinated responses over hallucinated ones, and recentefforts further enrich the preference data with relevant context. How-ever, it remains unclear whether DPO actually leverages such context.To investigate this, we propose Contextual Preference Gain (CPG), asimple metric that measures how much a model’s preference strengthenswhen relevant context is provided. We find that higher CPG consistentlycorresponds to lower hallucination, yet standard DPO and its variantsexhibit only limited CPG, indicating that they underutilize contextualinformation and thus remain prone to hallucination. To address this, wepropose Context-Calibrated DPO (C2 -DPO), which directly maximizesCPG while preserving the original preference ordering. Across multiplebenchmarks, C2 -DPO substantially reduces hallucination without com-promising general reasoning, relatively reducing the Object HalBenchhallucination rate of Qwen2-VL-Instruct-2B by 36%. Code is availableat https://github.com/mlvlab/C2-DPO