MobileSAM2: Lightweight Segment Anything in Images and Videos via Hypergraphical Knowledge Distillation

The recent large video foundation model, SAM2, enablessegment anything in both images and videos, serving as a powerful basemodel for various applications. However, many of such use cases requireto operate on resource-constrained devices like mobile phones and laptops.In this work, we aim to make SAM2 more mobile-friendly by distillingthe heavyweight SAM2 into a lightweight model, facilitating segmentanything in both images and videos on mobile devices. To this end, wepropose Hypergraphical Knowledge Distill (HyperKD), which introducesthe idea of hypergraph into knowledge distillation, aiming to effectivelymodel and transfer SAM2’s generalizable and comprehensive knowledge.HyperKD consists of Temporal HyperKD and Granularity HyperKD thatconstruct hypergraphs to explicitly model and extract the generalizabletemporal knowledge and the comprehensive multi-granularity knowledgefrom SAM2 respectively, which are then distilled into the lightweightstudent model by aligning it with the constructed hypergraphs. Besides,we present MobileSAM2, a new family of lightweight SAM2 that balancesefficiency and effectiveness via searching the best model architectures withHyperKD during model size reduction. Extensive experiments validateMobileSAM2 across multiple benchmarks and show promising generaliza-tion performance on embodied AI tasks.