3D-structure based methods remain the top-performing solution for long-term visual localization tasks. However, the dimension of existing local descriptors is usually high and the map takes huge storage space, especially for large-scale scenes. We propose a novel asymmetric framework which learns to reduce the dimension of local descriptors and match them jointly. We can compress existing local descriptor to 1/128 of original size while maintaining high matching performance. Experiments on several public visual localization datasets show that our pipeline obtains better results than existing map compression methods and non-structure based alternatives.