We present a novel approach designed to address the complexities posed by challenging, out-of-distribution data in the single-image depth estimation task, including adverse weather conditions and non-Lambertian objects. Starting with images that facilitate depth prediction due to the absence of unfavorable factors, we systematically generate new, user-defined scenes with a comprehensive set of challenges and associated depth information. This is achieved by leveraging cutting-edge conditioned diffusion models, known for their ability to synthesize high-quality image content from textual prompts while preserving the coherence of the 3D structure between generated and source imagery. Subsequent fine-tuning of any monocular depth network, either supervised or self-supervised, is carried out through a self-distillation protocol that takes into account images generated using our strategy and its own depth predictions on simple, unchallenging scenes. Experimental results on benchmarks tailored for our purposes demonstrate the effectiveness and versatility of our proposal\footnote, showing its distinctive ability to simultaneously address adverse weather settings and non-Lambertian objects, and to deliver competitive results with respect to specialized state-of-the-art solutions designed exclusively for each individual challenge.