Flash-DD: An Ultra Parameter-Efficient Approach to Dataset Distillation

Dataset distillation (DD) aims to create a smaller datasetthat encapsulates the essential knowledge of a larger dataset, therebyreducing storage demands and accelerating downstream training. Forlarge-scale dataset distillation, state-of-the-art methods achieve satisfac-tory performance by using soft labels generated by well-trained teachermodels during downstream training. However, it will cause some issues:(1) a substantial amount of additional storage is required to retain theteacher models, often significantly exceeding the storage needed for thesynthetic images; (2) generating labels through these teacher modelsslows down the downstream training process, counteracting the efficiencygoals of dataset distillation; and (3) downstream training guided by theseteacher models, according to our studies, yields suboptimal performance.Focusing on these drawbacks, in this paper, we propose plug-and-playparameter-efficient label generation techniques for dataset distillation,which maximizes the benefits of limited model parameters and can begeneralized to different DD methods, datasets, and settings. Specifically,we propose a DD-oriented model parameter reduction method that au-tomatically determines the optimal capacity of teacher models and elim-inates redundant parameters for dataset distillation tasks. Furthermore,for additional parameter space, we turn to model ensemble strategiesand propose guidelines to optimize the utilization efficiency of the ad-ditional space. Compared to the state-of-the-art methods, Flash-DD re-quires only 0.03% of the additional storage and significantly acceleratesdownstream label generation by 843.81× while maintaining compara-ble performance. Alternatively, with a mere 1.8% storage budget, it canboost accuracy by up to 13.4% over previous leading methods.