Aggregating Cross-Domain Knowledge via Learnable Tokens for Multi-Teacher Distillation

Multi-teacher distillation has emerged as a powerful paradigmfor consolidating diverse visual knowledge from multiple Vision Founda-tion Models (VFMs). However, existing methods typically rely on si-multaneous feature alignment, which forces the student to match mul-tiple teacher distributions directly. This rigid approach often inducesrepresentational dissonance and semantic conflicts, where teachers withdisparate architectures and divergent optimization goals lead to sub-optimal performance and restricted architectural flexibility. In this pa-per, we propose a novel framework termed ACTok that recasts multi-teacher distillation as a token-mediated aggregation process. By treat-ing visual features from both CNNs and Transformers as sets of tokens,we introduce learnable tokens that serve as flexible neural proxies tobridge diverse representational spaces. To avoid specialized modifica-tions on student backbones, we further design an alternating interactionmechanism that enables these learnable tokens to dynamically recon-cile complementary knowledge across teachers. Our ACTok is inherentlyarchitecture-agnostic, facilitating effective knowledge transfer to bothCNN and ViT students. Furthermore, by resolving teacher conflicts atthe token level, our method achieves superior performance using onlythe ImageNet-1K dataset, rendering it orders of magnitude more data-efficient than existing billion-scale distillation baselines. Extensive exper-iments on classification, segmentation, and depth estimation demonstratethat our token-based aggregation consistently outperforms state-of-the-art multi-teacher distillation methods, providing a robust and scalablesolution for VFM knowledge consolidation. Code is publicly availableon https://github.com/VISION-SJTU/ACTok.