Controlling Embedding Spaces with Text-Conditioned Transformations

Multimodal embedding spaces in models like CLIP enablepowerful capabilities such as semantic similarity retrieval and cross-modalzero-shot classification. These embeddings compress high-level seman-tics into a single vector, which comes at the cost of primarily express-ing a dominant semantics like main object while suppressing other im-portant attributes such as camera angle or color tone. We propose atext-conditioned transformation of visual embeddings that makes suchattributes explicitly accessible. Given a natural language description ofan attribute category (e.g., “color” or “art style”), a network generates anaffine transformation that emphasizes the specified attribute. Condition-ing on text enables it to learn many attributes simultaneously, accessingthem at inference time through an intuitive interface. The network istrained to align transformed embeddings with the frozen latent space,enabling retrieval using existing large-scale embeddings without any re-encoding. When applied to a full set, the same mechanism transforms thelatent space for attribute disentanglement tasks such as multi-clustering.By operating directly in latent space, our method provides a unifiedand efficient framework for controlling embedding spaces, demonstratingstate-of-the-art performance across both attribute-based retrieval andmulti-attribute organization tasks with near-zero inference cost.