Distill on a Diet: Efficient Knowledge Distillation via Learnable Data Pruning

Knowledge Distillation (KD) is widely used to obtain com-pact models for efficient inference in resource-constrained environments.Yet the computational overhead of the distillation process itself is oftenoverlooked, raising the question of whether a better student model canbe obtained with less data and less compute via data pruning. However,existing data pruning methods are not designed for KD: some introducesubstantial overhead (e.g., obtaining training dynamics through retrain-ing), while others rely on heuristic selection rules that fail to capture whatKD actually requires, often resulting in suboptimal subsets. To addressthese issues, we propose IF-Beta, an efficient data pruning frameworkthat combines influence function and a learnable sampling policy. Em-pirically, we first demonstrate that influence functions can serve as aneffective and efficient estimator of sample impact in KD settings, whereonly a pretrained teacher is available. Building on this, our samplingpolicy is specifically parameterized by a Beta distribution, whose highlyflexible two-parameter family allows the policy to adapt to diverse prun-ing regimes rather than being tied to fixed heuristic forms. Next, weformulate KD pruning as optimizing this policy through a bilevel objec-tive, where the inner loop operates in the teacher’s feature space with aKD-aligned objective, enabling fast proxy training, while the outer loopupdates the policy parameters to maximize the distillation performance.This design ensures IF-Beta is both computationally efficient and inher-ently aligned with the goals of KD. Extensive experiments on CIFAR-10/100 and ImageNet show that IF-Beta consistently outperforms otherbaselines across a wide range of pruning ratios. Remarkably, IF-Betadoes enable students trained on less data and less compute to surpassthe performance of students distilled on the full dataset.