ProtoFair: Fair Self-Supervised Contrastive Learning via Pseudo-Counterfactual Pairs

Self-supervised learning methods learn high-quality visualrepresentations, yet recent studies show that these representations of-ten capture demographic biases present in the training data. Existingfairness-aware methods address this by redesigning the self-supervisedobjective itself, limiting portability across the rapidly evolving land-scape of self-supervised learning (SSL) frameworks. We propose Proto-Fair, a fairness-aware contrastive loss designed to work alongside ex-isting SSL objectives without modifying them. ProtoFair leverages un-supervised prototype clustering to identify pseudo-counterfactual pairs:samples sharing the same cluster assignment but belonging to differentsensitive groups. By pulling these content-matched, cross-group samplestogether in the embedding space, ProtoFair encourages the encoder tolearn representations that are invariant to the sensitive attribute. Themethod requires only sensitive attribute annotations, no target labels,and integrates seamlessly with both SimCLR and SupCon. Experimentson CelebA and UTKFace demonstrate consistent fairness improvementswhile maintaining competitive accuracy.