The labor-intensive labeling for semantic segmentation has spurred the emergence of Unsupervised Semantic Segmentation. Recent studies utilize patch-wise contrastive learning based on features from image-level self-supervised pretrained models. However, relying solely on similarity-based supervision from image-level pretrained models often leads to unreliable guidance due to insufficient patch-level semantic representations. To address this, we propose Progressive Proxy Anchor Propagation (PPAP) strategy. This method gradually identifies more trustworthy positives of each anchor by relocating its proxy to densely populated regions of semantically similar samples. Specifically, we initially establish a tight boundary to gather a few reliable positive samples around each anchor. Then, considering the distribution of positive samples, we relocate the proxy anchor towards areas with a higher concentration of positives and adjust the positiveness boundary based on the propagation degree of the proxy anchor. In addition, there might exist ambiguous regions where positive and negative samples coexist near the positiveness boundary. Therefore, to further ensure the reliability of the negative set, we define an instance-wise ambiguous zone and exclude samples in such regions from the negative set. Our state-of-the-art performances on various datasets validate the effectiveness of the proposed method for Unsupervised Semantic Segmentation.