Simple Filtering Improves Masked Autoencoders

Deep models are successfully applied to various fields, whiledemanding large amount of annotated data for high-performance recog-nition. To remedy the data-hunger issue, self-supervised learning, espe-cially masked autoencoder (MAE), is a promising approach to effectivelypre-train the deep models. The MAE leverages random masking to con-struct pretext tasks where masked image patches are reconstructed byusing unmasked (visible) ones. In vision domain, however, inherent imageproperties of high redundancy and correlation in local neighbors couldinterfere with the mask-based pretext tasks. In this study, we analyze twomain processes of masking and reconstruction in MAE through the lensof difficulty of the pretext task. The analysis inspires us to propose simpleyet effective approaches based on filtering to improve random maskingas well as raw-pixel reconstruction by properly controlling difficulty ofMAE task with a negligible extra computation cost. In the experimentson image classification, the proposed method renders favorable perfor-mance improvement to MAE using ViTs.