Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training

Learning efficient and expressive visual representation haslong been the pursuit of computer vision research. While Vision Trans-formers (ViTs) gradually replace traditional Convolutional Neural Net-works (CNNs) as more scalable vision learners, their applications areplagued by the quadratic complexity of the self-attention mechanism. Toaddress the challenge, we introduce a new linear-time sequence modelingmethod Test-Time Training (TTT) into vision and propose Vision-TTT,which treats visual sequences as datasets and compresses the visual to-ken sequences in a novel self-supervised learning manner. By incorporat-ing the dual-dataset strategy and Conv2d-based dataset preprocessing,Vision-TTT effectively extends vanilla TTT to model 2D visual cor-relations with global receptive fields. Extensive experiments show thatVittt-T/S/B achieve 77.7%, 81.8%, 82.7% Top-1 accuracy on ImageNetclassification and also greatly outperform their counterparts on down-stream tasks. At 1280 × 1280 resolution, Vittt-T reduces FLOPs by79.4% and runs 4.72× faster with 88.9% less memory than DeiT-T.These results demonstrate the expressiveness and efficiency of Vision-TTT as an alternative for the next-generation visual backbone. Codesare available at https://github.com/imKQv/Vision-TTT.