This paper tackles the simultaneous optimization of pose and Neural Radiance Fields (NeRF). Departing from the conventional practice of using explicit global representations for camera pose, we introduce the Invertible Neural Warp, a novel overparameterized representation for camera pose. Specifically, we establish that invertibility is a crucial property when using Multi-layer Perceptrons (MLPs) for ray transformation. This work proposes utilizing an Invertible Neural Network, coupled with a geometry-informed constraint to achieve enhanced optimization convergence. Experiments on the LLFF and DTU datasets demonstrate the effectiveness of our approach in terms of pose estimation and high-fidelity reconstruction compared to existing well-established approaches.