Standard single-image super-resolution relies on creating paired training data from high-resolution images through static downsampling kernels. However, real-world super-resolution (RWSR) faces unknown degradations in the low-resolution inputs, while paired training data is lacking. Existing methods approach this problem by learning blind general models through complex synthetic augmentation on training inputs; they sacrifice the performance for broader generalization to many possible degradations. We address the unsupervised RWSR from a distillation perspective and introduce a novel pairwise distance distillation framework. Our framework adapts a model specialized in specific synthetic degradations to target real-world degradations by distilling intra- and inter-model distances across the specialized model and an auxiliary generalized model. Experiments on diverse datasets demonstrate that our method significantly enhances fidelity and perceptual quality, surpassing state-of-the-art approaches in RWSR.