Remote photoplethysmography (rPPG) has gained increasing attention in recent years for its ability to extract physiological signals from facial videos. Existing rPPG measurement methods have shown satisfactory performance in the intra-dataset and cross-dataset scenarios. However, they often neglect the incremental learning scenario, in which training data is presented sequentially, resulting in the issue of catastrophic forgetting. Meanwhile, mainstream class incremental learning algorithms suffer performance degradation or even fail to effectively transfer to rPPG measurement. In this paper, we present a novel and practical method to tackle continual learning for rPPG measurement. We firstly employ adapter finetuning to adapt to new tasks efficiently while enhancing the model's stability. To alleviate catastrophic forgetting without storing previous samples, we design a prototype-based augmentation strategy to reproduce the domain factors of previous tasks. Additionally, drawing inspiration from humans' problem-solving manner, an inference simplification strategy is devised to convert the potentially forgotten tasks into familiar ones for the model. To evaluate our method and enable fair comparisons, we create the first continual learning protocol for rPPG measurement. Extensive experiments demonstrate that our approach significantly surpasses the state-of-the-art methods.