Direct Preference Optimization for Perceptual Alignment via Vision-Language Consistency

Recent advances in Large Vision Language Models (LVLMs)have led to interest in alignment methods such as Direct PreferenceOptimization (DPO), which improves multi-modal instruction followingwithout costly supervision or explicit reward modeling. However, pre-vious attempts to improve DPO through data augmentation and pref-erence ranking fall short of bridging vision-language perceptual gaps,limiting consistent multi-modal alignment. In this paper, we proposea novel Vision-Language Consistency based Direct PreferenceOptimization (VLC-DPO), which automatically constructs prefer-ence data and leverages consistent information from visual inputs andcorresponding textual descriptions as alignment signals. The VLC-DPOconsists of three key components: (1) an initial response generation thatproduces image- and description-based responses to bridge the modal-ity gap in LVLMs by capturing complementary aspects of visual un-derstanding, (2) a preference data construction via vision-language con-sistency score that automatically selects high-quality preference pairsbased on response consistency, and (3) a VLC-DPO training that usesconsistency-based implicit rewards to optimize preferences, assigning dy-namic weights to less image-related or distant samples to improve DPOlearning. Extensive experiments on hallucination mitigation and zero-shot VQA demonstrate the e↵ectiveness of our method, which outper-forms existing approaches.