FairSteer: Cross-Attention Steering Towards a Fairer Text-Guided Image Generation

Text-to-image diffusion models power everyday creative tasks,but they still reproduce the demographic biases in their training data. Oncommon prompts such as “a photo of a nurse,” “a photo of a CEO”, theyskew their outputs toward one gender, driven by the statistics of trainingdata rather than anything in the text. Existing debiasing methods showpromise in narrow settings but require retraining, batch-level control, orprompt-specific tuning, limiting their scalability. We propose EquiSteer,a training-free method that works per sample by steering cross-attention(CA) activations at inference time. For each target attribute, EquiSteerprecomputes steering vectors from contrastive prompts. Then at gener-ation time, a prompt-aware gate leaves attribute-specific prompts un-touched, while for neutral ones it clears existing attribute signals fromthe CA activations and injects a target attribute. Across SD-1.5, SD-2.1,SDXL, and SANA, EquiSteer reduces the average parity gap by up to87%, with minimal effect on image quality and text-image alignment.Code is available at https://github.com/Atmyre/EquiSteer.