Activation Quantization of Vision Encoders Needs Prefixing Registers

Large pretrained vision encoders are central to multimodalintelligence, powering applications from on-device vision processing tovision-language models. Since these applications often demand real-timeprocessing of massive visual data, reducing the inference cost of visionencoders is critical. Quantization offers a practical path, but it remainschallenging even at 8-bit precision due to so-called outliers. In this work,we propose RegCache, a training-free algorithm that mitigates outliers inlarge-scale pretrained vision encoders and serves as a plug-in module thatcan be applied on top of other quantization methods. RegCache intro-duces outlier-prone yet semantically meaningless prefix tokens into thevision encoder, which prevent other tokens from having outliers. Notably,we observe that outliers in vision encoders behave differently from thosein language models, motivating two technical innovations: middle-layerprefixing and token deletion. Experimental results show that our methodconsistently improves quantized model performance across various visionencoders, particularly in extremely low-bit regimes (e.g., 4-bit).