Operational Open-Set Recognition and PostMax Refinement

Open-Set Recognition (OSR) is a problem with mainly practical applications. However, recent evaluations have been dominated by small-scale data and ignore the real-world operational needs of parameter selection. Thus, we revisit the original OSR goals and showcase an improved large-scale evaluation. For modern large networks, we show how deep feature magnitudes of unknown samples are surprisingly larger than for known samples. Therefore, we introduce a novel PreMax algorithm that prenormalizes the logits with the deep feature magnitude and use an extreme-value-based generalized Pareto distribution to map the scores into proper probabilities. Additionally, our proposed Operational Open-Set Accuracy (OOSA) measure can be used to predict an operationally relevant threshold and compare different algorithms' real-world performances. For various pre-trained ImageNet classifiers, including both leading transformer and convolution-based architectures, our experiments demonstrate that PreMax advances the state of the art in open-set recognition with statistically significant improvements on traditional metrics such as AUROC, FPR95, and also on our novel OOSA metric.