Recent advancements have achieved impressive results in removing Multi-Path Interference (MPI) and shot noise. However, these methods only utilize a single frame of ToF data, neglecting the correlation between frames. The multi-frame ToF denoising is still underexplored. In this paper, we propose the first learning-based framework for multi-frame ToF denoising. Different from previous frameworks, ours leverages the correlation between inter frames to guide the ToF noise removal with a confidence map. Specifically, we introduce a Dual-Correlation Estimation Module, which exploits both intra- and inter-correlation. The intra-correlation explicitly establishes the relevance between the spatial positions of geometric objects within the scene, aiding in depth residual initialization. The inter-correlation discerns variations in ToF noise distribution across different frames, thereby locating the areas with strong noise. To further leverage dual-correlation, we introduce a Confidence-guided Residual Regression Module to predict a confidence map, which guides the residual regression to prioritize the regions with strong ToF noise. The experimental evaluations have consistently shown that our approach outperforms other ToF denoising methods, highlighting its superior performance in effectively reducing strong ToF noise.