Cross-view geo-localization aims at localizing a ground-level query image by matching it to its corresponding geo-referenced aerial view. In real-world scenarios, the task requires handling diverse ground images captured by users with varying orientations and field of views (FoVs). However, existing learning pipelines are orientation-specific or FoV-specific, requiring separate training for different settings. Such models heavily depend on the North-aligned spatial correspondence and specific FoVs in the training data, compromising the models' robustness in ground view variations. We propose ConGeo, a single- and cross-modal Contrastive method for Geo-localization: it enhances robustness and consistency in feature representations to improve a model's invariance to orientation and its resilience to FoV variations, by enforcing proximity between ground view variations of the same location. As a generic learning objective for cross-view geo-localization, when integrated into state-of-the-art pipelines, ConGeo significantly boosts the performance of three base models on four geo-localization benchmarks for diverse ground view variations and outperforms competing methods that train separate models for each ground view variation.