Rethinking Adversary in Semantic Segmentation: An Out-of-Distribution Perspective

Semantic segmentation models remain vulnerable to adversarial perturbations, which can degrade pixel-wise predictions despite being imperceptible. Existing adversarial training methods primarily rely on iterative prediction-driven attacks that push features across decision boundaries between in-distribution (ID) classes while remaining close to the ID manifold. This limits their ability to expose more severe representation failures. In this paper, we propose SegOOD, a novel adversarial attack framework that explicitly encourages out-of-distribution (OOD)- like deviations in latent feature space. SegOOD introduces a Feature Wasserstein Separation objective to increase distributional discrepancy between adversarial features and ID semantic prototypes, along with a Weighted K-Nearest-Neighbor Separation objective to enforce local feature-level deviation. These objectives are efficiently implemented using prototype aggregation and feature-aware superpixel clustering. Furthermore, we integrate SegOOD with conventional prediction-driven attacks during adversarial training, improving robustness against both existing ID-style and our OOD-like perturbations. Extensive experiments demonstrate state-of-the-art robustness while maintaining competitive clean performance and no additional inference cost.