Generalized Category Discovery (GCD) utilizes labelled data from seen categories to cluster unlabelled samples from both seen and unseen categories. Previous methods have demonstrated that assigning pseudo-labels for representation learning is effective. However, these methods commonly predict pseudo-labels based on pairwise similarities, while the overall relationship among each instance's k-nearest neighbors (kNNs) is largely overlooked, leading to inaccurate pseudo-labeling. To address this issue, we introduce a Neighbor Graph Convolutional Network (NGCN) that learns to predict pairwise similarities between instances using only labelled data. NGCN explicitly leverages the relationships among each instance's \textit{k}NNs and is generalizable to samples of both seen and unseen classes. This helps produce more accurate positive samples by injecting the predicted similarities into subsequent clustering. Furthermore, we design a Cross-View Consistency Strategy (CVCS) to exclude samples with noisy pseudo-labels generated by clustering. This is achieved by comparing clusters from two different clustering algorithms. The filtered unlabelled data with pseudo-labels and the labelled data are then used to optimize the model through cluster- and instance-level contrastive objectives. The collaboration between NGCN and CVCS ensures the learning of a robust model, resulting in significant improvements in both seen and unseen class accuracies. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both generic and fine-grained GCD benchmarks.