Gait recognition, a biometric technology, aims to distinguish individuals by their walking patterns. However, we reveal that discriminative identity features often become entangled with non-identity clues, posing a challenge for extracting identity features effectively and efficiently in previous methods. To address this challenge, we propose CLTD, a causality-inspired discriminative feature learning module designed to effectively eliminate the influence of confounders in triple domains, \ie, the spatial, temporal, and spectral domains. Specifically, we utilize the Cross Pixel-wise Attention Generator (CPAG) to generate attention distributions for factual and counterfactual features in spatial and temporal domains. Then, we introduce the Fourier Projection Head (FPH) to project spatial features into the spectral space, preserving essential information while reducing computational costs. Furthermore, we employ an optimization method with contrastive learning to enforce semantic consistency constraints across sequences from the same subject. The significant performance improvements on challenging datasets demonstrate the effectiveness of our method. In addition, our method can seamlessly integrate into existing gait recognition methods.