HIVE: Understanding Post Hallucination Reasoning in Vision Language Models

Hallucinations in vision–language models (VLMs) are com-monly treated as semantic errors, yet they often arise from partial or am-biguous visual evidence. Prior work mainly focuses on detecting or sup-pressing hallucinations at generation time, leaving the subsequent reason-ing stage largely unexplored. In this work, we study Post-HallucinationReasoning (PHR), the stage in which hallucinated semantics enter themodel’s inference context and influence downstream predictions. To sys-tematically investigate PHR, we introduce the HIVE (HallucinationInference and Verification Engine), an evaluation infrastructure that en-ables controlled comparisons between faithful and hallucinated captions.Across nine tasks and nine models, we observe structured modality-dependent patterns: hallucinated captions often improve accuracy onvision–language tasks, while text-only tasks exhibit limited or unstableeffects. Further analyses show that hallucinated cues broaden semanticcoverage and reshape reasoning dynamics while preserving stable infer-ence. These findings highlight that hallucinated semantics may influencedownstream reasoning once they enter the model’s inference context. Un-derstanding this post-hallucination stage is important for improving thereliability and interpretability of multimodal reasoning systems.