The Map Is Not the Territory: Embedding-Coverage Blacklists for Safe Diffusion Steering

Ensuring safe content generation in text-to-image diffusionmodels remains a critical challenge. Existing safety mechanisms focuson model editing or trajectory steering, yet the design of the blacklist,the set of unsafe concepts these methods steer away from, is an equallypivotal but largely under-explored dimension. Through controlled exper-iments on blacklist-guided concept erasure, we uncover three findingsthat overturn common assumptions: longer keyword phrases do not nec-essarily improve safety, enlarging the blacklist yields diminishing returns,and mixing in irrelevant concepts degrades performance. Together theypoint to a single principle: blacklist effectiveness is governed by how wellits keywords cover unsafe regions of the embedding space, not by individ-ual keyword quality or list size. Building on this principle, we introduceMAPS, a framework for automated blacklist optimization. MAPS usesreinforcement learning to discover a prompt token that, when insertedinto an LLM prompt, elicits semantically diverse safety keywords, andthen assembles category-specific blacklists for distinct unsafe concepts.At inference, it selects the most relevant blacklist for each prompt viaembedding similarity, maximizing coverage of the threat while avoidinginterference from unrelated concepts. MAPS acts as a drop-in enhance-ment for existing trajectory-steering methods, reducing attack successrates by 25–52% (relative) across five red-teaming benchmarks on thenudity concept while preserving generation quality, all without modelediting or modifications to the steering procedure. Our code is availableat https://github.com/juyangbai/MAPS.