Dynamic Cluster Data Sampling for Efficient and Long-Tail-Aware Vision-Language Pre-training

The computational cost of training a vision-language model(VLM) can be reduced by sampling the training data. Previous work onefficient VLM pre-training has pointed to the importance of semanticdata balance, adjusting the distribution of topics in the data to improveVLM accuracy. However, existing efficient pre-training approaches maydisproportionately remove rare concepts from the training corpus. As aresult, long-tail concepts remain insufficiently represented in the trainingdata and are not effectively captured during training. In this work, weintroduce a dynamic cluster-based sampling approach (DynamiCS) thatdownsamples large clusters of data and upsamples small ones. We firstdemonstrate the advantage of our cluster-scaling approach, which main-tains the relative order of semantic clusters in the data and emphasizesthe long-tail. This approach contrasts with current work, which focusesonly on flattening the semantic distribution of the data. Then, we showthe importance of dynamic sampling, which applies sampling at eachepoch to improve cross-epoch data diversity and make upsampling prac-tical. Our experiments show that DynamiCS reduces the computationalcost of VLM training and provides a performance advantage for long-tail concepts. Code available at https://github.com/MingliangLiang3/DynamiCS.