Continual Test-Time Adaptation (CTTA) involves adapting a pre-trained source model to continually changing unsupervised target domains. In this paper, we systematically analyze the challenges of this task: online environment, unsupervised nature, and the risks of error accumulation and catastrophic forgetting under continual domain shifts. To address these challenges, we reshape the online data buffering and organizing mechanism for CTTA. By introducing an uncertainty-aware importance sampling approach, we dynamically evaluate and buffer the most significant samples with high certainty from the unsupervised, single-pass data stream. By efficiently organizing these samples into a class relation graph, we introduce a novel class relation preservation constraint to maintain the intrinsic class relations, efficiently overcoming catastrophic forgetting. Furthermore, a pseudo-target replay objective is incorporated to assign higher weights to these samples and mitigate error accumulation. Extensive experiments demonstrate the superiority of our method in both segmentation and classification CTTA tasks.