Multi-modal Knowledge Preserving Adapter for Embedding Backward Compatibility

Upgrading embedding models typically requires expensivedatabase re-indexing, as new query embeddings are incompatible withexisting database embeddings. While Backward Compatible Training(BCT) mitigates this by enforcing compatibility during training, ex-isting approaches often require updating the backbone model. This isimpractical because of significant training cost, the risk of performanceregression, and limited access to proprietary model weights. We intro-duce Multi-modal Knowledge Preserving Adapter (MKP-Adapter),the first adapter-only BCT approach for Multi-Modal Large LanguageModels (MLLMs) that requires no backbone updates. We identified thatthe primary challenge in adapter-only BCT is preserving the knowledgeof the new embeddings while enforcing backward compatibility. Hence,we propose a multi-level preservation loss that maintains the geometricstructure of the embedding spaces throughout BCT. Furthermore, a focalre-weighting strategy is integrated to prioritize learning from challeng-ing samples. Experiments demonstrate that our method achieves strongbackward compatibility across diverse multi-modal benchmarks (image,text, visual document, and video retrieval tasks) and model types. No-tably, MKP-Adapter is trained solely on pre-extracted embeddings andrequires only negligible additional latency relative to the original back-bone forward pass, highlighting its efficiency.