Fragmented Text Is Insufficient for Image Representation: Fine-Grained Correspondence in Multimodal Dataset Distillation

Multimodal dataset distillation (MDD) aims to synthesize acompact image-text dataset on which a visual-language model can betrained to achieve performance comparable to that trained on the orig-inal, full-scale dataset. While previous methods have achieved promis-ing results by using trajectory matching, they fail to consider the gapbetween single-semantic texts and multi-semantic images. Additionally,they usually use contrastive losses to pull together paired samples andpenalize non-paired samples, ignoring the textual information from un-paired samples that are semantically related to the corresponding images.To reduce the semantic gap, we introduce a multi-text fusion module thatstrengthens the cross-modal interaction between the image and text. Inaddition, we leverage uncertainty to adaptively guide the contribution ofnon-paired samples in the synthetic dataset, thereby improving the ef-fective utilization of information from these samples. We then provide aninformation theoretic analysis of the semantic gap and the limitations ofcontrastive supervision. Experiments on Flickr-30K and MS-COCO showthat our method consistently outperforms previous state-of-the-art MDDmethods, achieving significant improvements in retrieval performance(e.g., +6.5% in IR@10 and +4.6% in TR@10 in a 500-pair setting).Code is available at https://github.com/yiqiqiandefanhua/P2DE.