HEM: a margin-based loss for visual categorisation tasks

Training deep neural networks (DNNs) on classi(cid:28)cation tasks can be performed with a number of di(cid:27)erent losses, but cross-entropy (CE) loss is the de-facto standard. Here, we propose an alternative loss, high error margin (HEM), which is a margin based loss modi(cid:28)ed to improve the training dynamics of neural networks. HEM loss is evaluated extensively using a wide range of DNN architectures and benchmark datasets with all experimental settings and training hyper-parameters taken from the literature, and hence, optimised for CE loss. HEM is found to be more e(cid:27)ective than CE loss across a range of image-based tasks: unknown class rejection, adversarial robustness, learning with imbalanced data, continual learning, and semantic segmentation (a pixel-wise classi- (cid:28)cation task). HEM is inferior to CE only in terms of clean and corrupt image classi(cid:28)cation with balanced training data, and this di(cid:27)erence is small. We also compare HEM to specialised losses that have previously been proposed to improve performance for speci(cid:28)c vision tasks. LogitNorm, a loss achieving state-of-the-art performance on unknown class rejection, produces similar performance to HEM for this task, but is much poorer for continual learning and semantic segmentation. Logit-adjusted loss, designed for imbalanced data, has superior results to HEM for that task, but performs worse on unknown class rejection and semantic segmentation. DICE, a popular loss for semantic segmentation, is inferior to HEM loss on all tasks, including semantic segmentation. Overall, HEM is competitive with the best alternative loss for all the tasks we have used and performs better than all other tested losses in terms of rejecting out-of-distribution examples, for continual learning, and by a substan- 1 tial margin for semantic segmentation . • • •