Towards Reconfigurable Visual Feature Compression

Recently, vision foundation models (VFMs) have demonstrated strong power for versatile downstream task analysis. This paper starts from the problem of how to efficiently deploy VFMs at the cloud to support various user requests with arbitrary multi-task combinations that end in any scalable fashion. To achieve feature data transmission between frontend-cloud, traditional feature coding employs the direct compression-reconstruction paradigm. However, this can result in potential redundancy: (1) task-irrelevant information, and (2) repeated coding of cross-task shared knowledge, leading to an inflexible and redundant scheme. To this end, we propose a novel feature coding paradigm, termed reconfigurable multi-task feature compression, which aims to efficiently and adaptively compress the intermediate features to support the requested targeted tasks. Correspondingly, we propose a unified Reconfigurable Feature Compression framework, RFC, by feature factorization and recomposition. Specifically, the original feature is first factorized into multiple task-specific features with light-weight adapters. Then to efficiently compress these separate features, we consolidate them in a task-conditional auto-regressive manner, leveraging previously encoded task features as hyperprior conditions to reduce the redundancy of shared information in the current task feature. At the cloud side, a task-attentive recomposition module is further developed to fulfill the multi-task inference power within a single forward pass. Finally, we construct a comprehensive benchmark of reconfigurable feature compression to verify the effectiveness of RFC. Our project page can be found at https://jhang2020.github.io/Projects/RFC/RFC.html.