Remote photoplethysmography (rPPG) is an emerging technology that can detect the pulse rate remotely from face videos. However, it is easily influenced by the recording environment, as robustness to noise is still an open problem. This vulnerability can therefore be exploited to inject fake signals or impair predictions physically. In this study we propose the first dataset containing a wide set of physical domain attack scenarios divided in three categories (illumination, movement, concealment) that directly target the main weaknesses of rPPG. We propose the rPPG Physical Domain Attacks Database (RPDAD) as a benchmark for evaluation of robustness to physical attacks. We perform extensive experiments on conventional hand-crafted and deep learning (end-to-end, non-end-to-end, CNN, transformer, self-supervised) methods and study their susceptibility to the attacks. We conclude by discussing the most critical vulnerabilities discovered and stress the importance of designing more secure solutions.