BIP: Bi-level Information Transfer and Completion Prompting for Visual Recognition with Missing Modalities

Existing multimodal visual recognition methods addressing missing modalities have achieved great progress, mainly based on the feature reconstruction framework. However, feature reconstruction often introduces distribution inconsistency between synthesized and real features. Considering each modality as a different view of the same instance, each view is noisy and incomplete, but important representations, such as semantic details, tend to be shared between all modalities. Thus, we propose that AlignPrompt alleviates the distribution inconsistency problem by completing and aligning the semantic representation between synthesized and non-missing prompts. AlignPrompt is an instance-level prompt, implemented via a modal-shared-specific structure including a modalshared factor and a modal-specific factor. Specifically, for each modality pair, AlignPrompt aligns modality-specific factors of all modalities in a shared space by the modal-shared factor as an anchor, and completes the semantic representation by maximizing the mutual information between the synthesized factor and the non-missing factor. Besides, we also propose HyperPrompt to globally regulate the association of prompts of each layer based on different modalities missing scenarios: it transfers global prompts across layers by the inter-layer modules generated from an independent network. This collaborative prompts framework achieves state-of-the-art performance on multiple multimodal classification datasets while maintaining parameter efficiency.