VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

The effectiveness of multimodal instruction tuning dependsnot only on dataset scale, but also critically on whether training samplesgenuinely require visual reasoning. However, existing instruction datasetsoften contain a substantial portion of visually redundant samples (solv-able from text alone), as well as multimodally misaligned supervisionthat can degrade learning. To address this, we propose VisNec (VisualNecessity Score), a principled data selection framework that measuresthe marginal contribution of visual input during instruction tuning. Bycomparing predictive loss with and without visual context, VisNec iden-tifies whether a training instance is vision-critical, redundant, or mis-aligned. To preserve task diversity, we combine VisNec with semanticclustering and select high-necessity samples within each cluster. Across10 downstream benchmarks, training on only 15% of the LLaVA-665Kdataset selected by VisNec achieves 100.2% of full-data performance. Onthe smaller Vision-Flan-186K dataset, our selection not only further re-duces data size but surpasses full-data training by 15.8%. These resultsdemonstrate that measuring and leveraging visual necessity provides aneffective solution for both efficient and robust multimodal instructiontuning. Project Page: https://dmk041218.github.io/VisNec/.