UNet-Twice: A Simple Structured Reference-based Inpainting Framework

Reference-based inpainting utilizes the information in thereference image to inpaint the missing region in the target image, aimingto generate the fact object behind the mask rather than a reasonable one.The key challenge is how to accurately locate the region in the referenceimage and maximally utilize it. We proposed UNet-Twice, a simple yetnovel framework. UNet-Twice uses the same UNet two times, combinedwith a well-constructed data pair to build a close connection betweenthe target and reference image. The two passes fuse spatially comple-mentary features generated from the target and the reference image,which provides intuitively explicit guidance for image generation. Ourproposed framework is simple and easy to extend to a multi-referencesetting. Extensive experiments demonstrate that our proposed frame-work outperforms existing reference-based inpainting methods in bothsingle-reference and multi-reference settings.