Accurate Zero-shot Quantization via Hierarchical Teacher-Assistant Distillation

How can we bridge the numerical chasm between full-precisionCNN models and highly quantized targets when original training dataare unavailable? As the demand for deploying deep neural networkson resource-constrained edge hardware grows, achieving extreme modelcompression without compromising the privacy of training data has be-come a critical challenge. However, direct transitions to ultra-low-bitregimes introduce extreme optimization instability and activation col-lapse, where high quantization noise causes deep layers to lose their dis-criminative capacity and representational power.In this paper, we propose Zest (ZSQ Enhancement via Stepping-stoneTransition), an accurate zero-shot quantization framework for CNNs thatbridges the precision gap between full-precision models and ultra-low-bit targets through a unified two-stage distillation and reconstructionpipeline. Rather than attempting a direct and unstable jump from fullprecision to an ultra-low bit quantized model, Zest introduces a stableintermediate bit stage to tame the numerical distribution and regular-ize the model. Furthermore, we propose a two-phase distillation strategythat initializes high-fidelity semantic skeletons from the full-precisionmodel and subsequently recalibrates them through an intermediate bitstage to ensure calibration data are numerically aligned with quantized-domain constraints. Experimental results demonstrate that this progres-sive transition leads to better distillation, achieving 70.50% top-1 accu-racy on ResNet-18 at W 4A4 precision and showing higher performancecompared to existing SOTA methods in various models.