IREU: Identity-Related Encoder-Only Unlearning for Customized Portrait Generation

Customized Portrait Generation (CPG) technologies havebeen widely used to generate high-fidelity person images given an inputimage indicating the identity and a text prompt indicating the requirededits. Yet these methods pose significant privacy risks by spreading fakevisual information. Against such risks, each public generator should beable to suppress its generation ability for a particular person when re-quested. Therefore, in this work we investigate the identity unlearningproblem for CPG. Since there are no previous methods in this field, wepropose a simple baseline that updates the image encoder by minimizingidentity similarity between generated and input images for target iden-tities to be unlearned, while maximizing it for identities to be retained.However, we find such a global perturbation in the feature space harmsthe fidelity of generated images for other identities to be retained. Tosolve this problem, we propose a novel method IREU, which first locatesidentity-related features in an offline manner and then only performsfeature perturbations on them. The experimental results show that ourproposed method IREU achieves better identity unlearning performancefor target identities to be unlearned, and also keeps high fidelity for otheridentities to be retained. In addition, our unlearned image encoder isgeneralizable across different generators with the same encoder withoutfine-tuning, which is friendly for deployment in practice.