SWIFT: Spatial-Window Integrated Frequency-aware Token Pruning for Efficient MLLMs on Edge Devices

While Multimodal Large Language Models (MLLMs) ex-cel in visual understanding, the quadratic complexity imposed by densehigh-resolution visual tokens poses significant challenges for edge-devicedeployment. Current training-free pruning strategies predominantly de-pend on attention weights. However, viewed from the frequency domain,the self-attention mechanism functions as a low-pass filter, causing thediscard of essential high-frequency signals (e.g., textures and edges) andresulting in feature over-smoothing. In this paper, we present Spatial-Window Integrated Frequency-aware Token Pruning (SWIFT), a seam-less plug-and-play framework designed for efficient token compression.SWIFT leverages two novel components: a Frequency-Aware Indicator(FAI), which identifies fine-grained details by estimating high-frequencyresiduals through matrix factorization, and a Spatial-Window Integra-tion (SWI) module, which prevents spatial structure collapse and posi-tional bias via localized retention. Extensive evaluations on Qwen2.5VL-3B and LLaVA benchmarks show that under a practical 50% compressionratio, SWIFT achieves a 2.10× speedup with near-lossless performance.Under an extreme 75% compression stress test, SWIFT yields an averageperformance drop of 5.1% across 8 benchmarks, while delivering a 2.66×speedup and 69.07% KV cache reduction. SWIFT exhibits superior ro-bustness in detail-intensive tasks compared to SOTA methods, providinga potent lightweight solution for on-device MLLM applications. Our codeis available at https://github.com/damo-lgl/SWIFT.