COVERT: Privacy-Preserving Covariant Obfuscation for VLMaaS via Exact Reparameterization and Tailored Tuning

With the emergence of Vision-Language Models (VLMs),cloud-based VLM inference services have raised critical privacy concernsregarding the potential leakage of sensitive visual prompts. However, ex-isting defenses compromise either efficiency or model utility to achieveprivacy. Recently, the covariant obfuscation paradigm has shown greatpotential for private large language model inference, but it is incom-patible with cross-modal architectures and tasks. In this paper, we pro-pose COVERT, the first practical privacy-preserving VLM inferenceframework, which explicitly adapts covariant obfuscation for multimodalarchitectures. Specifically, it integrates exact architectural reparameter-ization to robustly obfuscate visual features against naive inversion at-tacks, alongside utility-aware parameter tuning tailored to neutralizehidden-state inversion attacks, thereby achieving a favorable privacy-utility-efficiency trade-off. Extensive evaluations show that COVERT ef-fectively shields sensitive visual attributes against hidden-state inversionattacks, while keeping the average accuracy degradation below 3% acrossmajor benchmarks and sustaining high inference throughput with a neg-ligible overhead of less than 6%.