Abstract: Adversarial robustness has been studied extensively in image classification, especially for the $\ell_\infty$-threat model, but significantly less so for related tasks such as object detection and semantic segmentation. Attacks on semantic segmentation models turn out to be harder than for image classification. We propose novel attacks and motivated by their complementary properties, we put them into an attack ensemble called SEA. We use SEA to show that existing attacks can severely overestimate the robustness of semantic segmentation models. Perhaps surprisingly, existing attempts of adversarial training for semantic segmentation turn out to yield only weakly robust models or are even completely non-robust. We investigate why previous adaptations of adversarial training to semantic segmentation failed and identify insufficient training time and number of attack steps as key elements. In turn we show how recently proposed robust ImageNet backbones can be used to obtain adversarially robust semantic segmentation models with up to six times less training time for Pascal VOC and the more challenging ADE20k.