Traditional 3D scene understanding techniques rely on supervised learning from densely annotated 3D datasets. However, the collection and annotation of 3D data is expensive and tedious, which leads to the scarcity of labeled training data. In this paper, we investigate the use of diffusion models which are pre-trained on large-scale image-caption pairs for open-vocabulary 3D scene understanding. We propose a novel method, namely Diff2Scene, leverages frozen representations from text-image discriminative and generative models, along with salient-aware and geometric-aware masks, for open-vocabulary scene understanding. Diff2Scene gets rid of any labeled 3D data and effectively identifies objects, appearances, materials, locations and their compositions in 3D scenes using a single model. We show that it outperforms competitive baselines and achieves significant improvements over state-of-the-art methods in open-vocabulary 3D semantic segmentation tasks. In particular, Diff2Scene improves the state-of-the-art method on ScanNet200 by 12%.